{"owner":"pola-rs","repo":"polars","hasSkills":true,"totalSkillsCount":3,"totalTokensCount":4471,"categories":["anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":["docs/source/user-guide/plugins/io_plugins.md","docs/source/user-guide/plugins/index.md","docs/source/user-guide/plugins/expr_plugins.md"],"skills":{"docs/source/user-guide/plugins/io_plugins.md":"# IO Plugins\n\nBesides [expression plugins](./expr_plugins.md), we also support IO plugins. These allow you to\nregister different file formats as sources to the Polars engines. Because sources can move data zero\ncopy via Arrow FFI and sources can produce large chunks of data before returning, we've decided to\ninterface to IO plugins via Python for now, as we don't think the short time the GIL is needed\nshould lead to any contention.\n\nE.g. an IO source can read their dataframe's in rust and only at the rendez-vous move the data\nzero-copy having only a short time the GIL is needed.\n\n## Use case\n\nYou want IO plugins if you have a source file not supported by Polars and you want to benefit from\noptimizations like projection pushdown, predicate pushdown, early stopping and support of our\nstreaming engine.\n\n## Example\n\nSo let's write a simple, very bad, custom CSV source and register that as an IO plugin. I want to\nstress that this is a very bad example and is only given for learning purposes.\n\nFirst we define some imports we need:\n\n```python\n# Use python for csv parsing.\nimport csv\nimport polars as pl\n# Used to register a new generator on every instantiation.\nfrom polars.io.plugins import register_io_source\nfrom typing import Iterator\nimport io\n```\n\n### Parsing the schema\n\nEvery `scan` function in Polars has to be able to provide the schema of the data it reads. For this\nsimple csv parser we will always read the data as `pl.String`. The only thing that differs are the\nfield names and the number of fields.\n\n```python\ndef parse_schema(csv_str: str) -> pl.Schema:\n    first_line = csv_str.split(\"\\n\")[0]\n\n    return pl.Schema({k: pl.String for k in first_line.split(\",\")})\n```\n\nIf we run this with small csv file `\"a,b,c\\n1,2,3\"` we get the schema:\n`Schema([('a', String), ('b', String), ('c', String)])`.\n\n```python\n>>> print(parse_schema(\"a,b,c\\n1,2,3\"))\nSchema([('a', String), ('b', String), ('c', String)])\n```\n\n### Writing the source\n\nNext up is the actual source. For this we create an outer and an inner function. The outer function\n`my_scan_csv` is the user facing function. This function will accept the file name and other\npotential arguments you would need for reading the source. For csv files, these arguments could be\n\"delimiter\", \"quote_char\" and such.\n\nThis outer function calls `register_io_source` which accepts a `callable` and a `schema`. The schema\nis the Polars schema of the complete source file (independent of projection pushdown).\n\nThe callable is a function that will return a generator that produces `pl.DataFrame` objects.\n\nThe arguments of this function are predefined and this function must accept:\n\n- `with_columns`\n\n  Columns that are projected. The reader must project these columns if applied\n\n- `predicate`\n\n  Polars expression. The reader must filter their rows accordingly.\n\n- `n_rows`\n\n  Materialize only n rows from the source. The reader can stop when `n_rows` are read.\n\n- `batch_size`\n\n  A hint of the ideal batch size the reader's generator must produce.\n\nThe inner function is the actual implementation of the IO source and can also call into Rust/C++ or\nwherever the IO plugin is written. If you want to see an IO source implemented in Rust, take a look\nat our [plugins repository](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin).\n\n```python\ndef my_scan_csv(csv_str: str) -> pl.LazyFrame:\n    schema = parse_schema(csv_str)\n\n    def source_generator(\n        with_columns: list[str] | None,\n        predicate: pl.Expr | None,\n        n_rows: int | None,\n        batch_size: int | None,\n    ) -> Iterator[pl.DataFrame]:\n        \"\"\"\n        Generator function that creates the source.\n        This function will be registered as IO source.\n        \"\"\"\n        if batch_size is None:\n            batch_size = 100\n\n        # Initialize the reader.\n        reader = csv.reader(io.StringIO(csv_str), delimiter=',')\n        # Skip the header.\n        _ = next(reader)\n\n        # Ensure we don't read more rows than requested from the engine\n        while n_rows is None or n_rows > 0:\n            if n_rows is not None:\n                batch_size = min(batch_size, n_rows)\n\n            rows = []\n\n            for _ in range(batch_size):\n                try:\n                    row = next(reader)\n                except StopIteration:\n                    n_rows = 0\n                    break\n                rows.append(row)\n\n            df = pl.from_records(rows, schema=schema, orient=\"row\")\n            n_rows -= df.height\n\n            # If we would make a performant reader, we would not read these\n            # columns at all.\n            if with_columns is not None:\n                df = df.select(with_columns)\n\n            # If the source supports predicate pushdown, the expression can be parsed\n            # to skip rows/groups.\n            if predicate is not None:\n                df = df.filter(predicate)\n\n            yield df\n\n    return register_io_source(io_source=source_generator, schema=schema)\n```\n\n### Taking it for a (very slow) spin\n\nFinally we can test our source:\n\n```python\ncsv_str1 = \"\"\"a,b,c,d\n1,2,3,4\n9,10,11,2\n1,2,3,4\n1,122,3,4\"\"\"\n\nprint(my_scan_csv(csv_str1).collect())\n\n\ncsv_str2 = \"\"\"a,b\n1,2\n9,10\n1,2\n1,122\"\"\"\n\nprint(my_scan_csv(csv_str2).head(2).collect())\n```\n\nRunning the script above would print the following output to the console:\n\n```\nshape: (4, 4)\n┌─────┬─────┬─────┬─────┐\n│ a   ┆ b   ┆ c   ┆ d   │\n│ --- ┆ --- ┆ --- ┆ --- │\n│ str ┆ str ┆ str ┆ str │\n╞═════╪═════╪═════╪═════╡\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 9   ┆ 10  ┆ 11  ┆ 2   │\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 1   ┆ 122 ┆ 3   ┆ 4   │\n└─────┴─────┴─────┴─────┘\nshape: (2, 2)\n┌─────┬─────┐\n│ a   ┆ b   │\n│ --- ┆ --- │\n│ str ┆ str │\n╞═════╪═════╡\n│ 1   ┆ 2   │\n│ 9   ┆ 10  │\n└─────┴─────┘\n```\n\n## For engine authors: identifying a source\n\nThis section is for people writing a Polars execution engine. If you are writing an IO source, you\ncan skip it.\n\nPolars does not call your source directly. It calls a wrapper that deserializes the predicate and\nforwards the call. Engines may need to identify sources they know about, for example to provide\nexecution context or call additional methods.\n\nTo make that possible, the wrapper satisfies the `IOSourceScanFunction` protocol, which exposes the\nregistered source:\n\n```python\nfrom polars.io.plugins import IOSourceScanFunction\n\n# `scan_fn` is the scan function of the `PythonScan` node being executed.\nif isinstance(scan_fn, IOSourceScanFunction):\n    source = scan_fn.io_source  # exactly the object passed to register_io_source\n    if isinstance(source, MyEngineSource):\n        ...\n```\n\nThis is as unstable as the rest of the IO plugin API.\n\n## Further reading\n\n- [Rust example (distribution source)](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin)\n","docs/source/user-guide/plugins/index.md":"# Plugins\n\nPolars allows you to extend its functionality with either Expression plugins or IO plugins.\n\n- [Expression plugins](./expr_plugins.md)\n- [IO plugins](./io_plugins.md)\n\n## Community plugins\n\nHere is a curated (non-exhaustive) list of community-implemented plugins.\n\n### Various\n\n- [polars-xdt](https://github.com/pola-rs/polars-xdt) Polars plugin with extra datetime-related\n  functionality which isn't quite in-scope for the main library\n- [polars-hash](https://github.com/ion-elgreco/polars-hash) Stable non-cryptographic and\n  cryptographic hashing functions for Polars\n\n### Data science\n\n- [polars-distance](https://github.com/ion-elgreco/polars-distance) Polars plugin for pairwise\n  distance functions\n- [polars-ds](https://github.com/abstractqqq/polars_ds_extension) Polars extension aiming to\n  simplify common numerical/string data analysis procedures\n\n### Bioinformatics\n\n- [polars-bio](https://github.com/biodatageeks/polars-bio) Python library for genomics built on top\n  of Polars, Apache Arrow and Apache DataFusion\n\n### Geo\n\n- [polars-st](https://github.com/Oreilles/polars-st) Polars ST provides spatial operations on Polars\n  DataFrames, Series and Expressions. Just like Shapely and Geopandas.\n- [polars-reverse-geocode](https://github.com/MarcoGorelli/polars-reverse-geocode) Offline reverse\n  geocoder for finding the closest city to a given (latitude, longitude) pair.\n- [polars-h3](https://github.com/Filimoa/polars-h3) This is a Polars extension that adds support for\n  the H3 discrete global grid system, so you can index points and geometries to hexagons directly in\n  Polars.\n\n## Other material\n\n- [Ritchie Vink - Keynote on Polars Plugins](https://youtu.be/jKW-CBV7NUM)\n- [Polars plugins tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) Learn how to\n  write a plugin by going through some very simple and minimal examples\n- [cookiecutter-polars-plugin](https://github.com/MarcoGorelli/cookiecutter-polars-plugins) Project\n  template for Polars Plugins\n","docs/source/user-guide/plugins/expr_plugins.md":"# Expression Plugins\n\n<!-- (TODO) the steps outlined here aren't clear and may not work that well -->\n\nExpression plugins are the preferred way to create user defined functions. They allow you to compile\na Rust function and register that as an expression into the Polars library. The Polars engine will\ndynamically link your function at runtime and your expression will run almost as fast as native\nexpressions. Note that this works without any interference of Python and thus no GIL contention.\n\nThey will benefit from the same benefits default expressions have:\n\n- Optimization\n- Parallelism\n- Rust native performance\n\nTo get started we will see what is needed to create a custom expression.\n\n## Our first custom expression: Pig Latin\n\nFor our first expression we are going to create a pig latin converter. Pig latin is a silly language\nwhere in every word the first letter is removed, added to the back and finally \"ay\" is added. So the\nword \"pig\" would convert to \"igpay\".\n\nWe could of course already do that with expressions, e.g.\n`col(\"name\").str.slice(1) + col(\"name\").str.slice(0, 1) + \"ay\"`, but a specialized function for this\nwould perform better and allows us to learn about the plugins.\n\n### Setting up\n\nWe start with a new library as the following `Cargo.toml` file\n\n```toml\n[package]\nname = \"expression_lib\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[lib]\nname = \"expression_lib\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\npolars = { version = \"*\" }\npyo3 = { version = \"*\", features = [\"extension-module\", \"abi3-py310\"] }\npyo3-polars = { version = \"*\", features = [\"derive\"] }\nserde = { version = \"*\", features = [\"derive\"] }\n```\n\n### Writing the expression\n\nIn this library we create a helper function that converts a `&str` to pig-latin, and we create the\nfunction that we will expose as an expression. To expose a function we must add the\n`#[polars_expr(output_type=DataType)]` attribute and the function must always accept\n`inputs: &[Series]` as its first argument.\n\n```rust\n// src/expressions.rs\nuse polars::prelude::*;\nuse pyo3_polars::derive::polars_expr;\nuse std::fmt::Write;\n\nfn pig_latin_str(value: &str, output: &mut String) {\n    if let Some(first_char) = value.chars().next() {\n        write!(output, \"{}{}ay\", &value[1..], first_char).unwrap()\n    }\n}\n\n#[polars_expr(output_type=String)]\nfn pig_latinnify(inputs: &[Series]) -> PolarsResult<Series> {\n    let ca = inputs[0].str()?;\n    let out: StringChunked = ca.apply_into_string_amortized(pig_latin_str);\n    Ok(out.into_series())\n}\n```\n\nNote that we use `apply_into_string_amortized`, as opposed to `apply_values`, to avoid allocating a\nnew string for each row. If your plugin takes in multiple inputs, operates elementwise, and produces\na `String` output, then you may want to look at the `binary_elementwise_into_string_amortized`\nutility function in `polars::prelude::arity`.\n\nThis is all that is needed on the Rust side. On the Python side we must setup a folder with the same\nname as defined in the `Cargo.toml`, in this case \"expression_lib\". We will create a folder in the\nsame directory as our Rust `src` folder named `expression_lib` and we create an\n`expression_lib/__init__.py`. The resulting file structure should look something like this:\n\n```\n├── 📁 expression_lib/  # name must match \"lib.name\" in Cargo.toml\n|   └── __init__.py\n|\n├── 📁src/\n|   ├── lib.rs\n|   └── expressions.rs\n|\n├── Cargo.toml\n└── pyproject.toml\n```\n\nThen we create new expressions. The function name of our expression can be registered. Note that it\nis important that this name is correct, otherwise the main Polars package cannot resolve the\nfunction name. Furthermore we can set additional keyword arguments that explain to Polars how this\nexpression behaves. In this case we tell Polars that this function is elementwise. This allows\nPolars to run this expression in batches. Whereas for other operations this would not be allowed,\nthink for instance of a sort, or a slice.\n\n```python\n# expression_lib/__init__.py\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING\n\nimport polars as pl\nfrom polars.plugins import register_plugin_function\nfrom polars._typing import IntoExpr\n\nPLUGIN_PATH = Path(__file__).parent\n\ndef pig_latinnify(expr: IntoExpr) -> pl.Expr:\n    \"\"\"Pig-latinnify expression.\"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"pig_latinnify\",\n        args=expr,\n        is_elementwise=True,\n    )\n```\n\nWe can then compile this library in our environment by installing `maturin` and running\n`maturin develop --release`.\n\nAnd that's it. Our expression is ready to use!\n\n```python\nimport polars as pl\nfrom expression_lib import pig_latinnify\n\ndf = pl.DataFrame(\n    {\n        \"convert\": [\"pig\", \"latin\", \"is\", \"silly\"],\n    }\n)\nout = df.with_columns(pig_latin=pig_latinnify(\"convert\"))\n```\n\nAlternatively, you can\n[register a custom namespace](https://docs.pola.rs/api/python/stable/reference/api/polars.api.register_expr_namespace.html#polars.api.register_expr_namespace),\nwhich enables you to create a `Expr.language` namespace, allowing users to write:\n\n```python\nout = df.with_columns(\n    pig_latin=pl.col(\"convert\").language.pig_latinnify(),\n)\n```\n\n## Accepting kwargs\n\nIf you want to accept `kwargs` (keyword arguments) in a polars expression, all you have to do is\ndefine a Rust `struct` and make sure that it derives `serde::Deserialize`.\n\n```rust\n/// Provide your own kwargs struct with the proper schema and accept that type\n/// in your plugin expression.\n#[derive(Deserialize)]\npub struct MyKwargs {\n    float_arg: f64,\n    integer_arg: i64,\n    string_arg: String,\n    boolean_arg: bool,\n}\n\n/// If you want to accept `kwargs`. You define a `kwargs` argument\n/// on the second position in you plugin. You can provide any custom struct that is deserializable\n/// with the pickle protocol (on the Rust side).\n#[polars_expr(output_type=String)]\nfn append_kwargs(input: &[Series], kwargs: MyKwargs) -> PolarsResult<Series> {\n    let input = &input[0];\n    let input = input.cast(&DataType::String)?;\n    let ca = input.str().unwrap();\n\n    Ok(ca\n        .apply_into_string_amortized(|val, buf| {\n            write!(\n                buf,\n                \"{}-{}-{}-{}-{}\",\n                val, kwargs.float_arg, kwargs.integer_arg, kwargs.string_arg, kwargs.boolean_arg\n            )\n                .unwrap()\n        })\n        .into_series())\n}\n```\n\nOn the Python side the kwargs can be passed when we register the plugin.\n\n```python\ndef append_args(\n    expr: IntoExpr,\n    float_arg: float,\n    integer_arg: int,\n    string_arg: str,\n    boolean_arg: bool,\n) -> pl.Expr:\n    \"\"\"\n    This example shows how arguments other than `Series` can be used.\n    \"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"append_kwargs\",\n        args=expr,\n        kwargs={\n            \"float_arg\": float_arg,\n            \"integer_arg\": integer_arg,\n            \"string_arg\": string_arg,\n            \"boolean_arg\": boolean_arg,\n        },\n        is_elementwise=True,\n    )\n```\n\n## Output data types\n\nOutput data types of course don't have to be fixed. They often depend on the input types of an\nexpression. To accommodate this you can provide the `#[polars_expr()]` macro with an\n`output_type_func` argument that points to a function. This function can map input fields `&[Field]`\nto an output `Field` (name and data type).\n\nIn the snippet below is an example where we use the utility `FieldsMapper` to help with this\nmapping.\n\n```rust\nuse polars_plan::dsl::FieldsMapper;\n\nfn haversine_output(input_fields: &[Field]) -> PolarsResult<Field> {\n    FieldsMapper::new(input_fields).map_to_float_dtype()\n}\n\n#[polars_expr(output_type_func=haversine_output)]\nfn haversine(inputs: &[Series]) -> PolarsResult<Series> {\n    let out = match inputs[0].dtype() {\n        DataType::Float32 => {\n            let start_lat = inputs[0].f32().unwrap();\n            let start_long = inputs[1].f32().unwrap();\n            let end_lat = inputs[2].f32().unwrap();\n            let end_long = inputs[3].f32().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        DataType::Float64 => {\n            let start_lat = inputs[0].f64().unwrap();\n            let start_long = inputs[1].f64().unwrap();\n            let end_lat = inputs[2].f64().unwrap();\n            let end_long = inputs[3].f64().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        _ => polars_bail!(InvalidOperation: \"only supported for float types\"),\n    };\n    Ok(out)\n}\n```\n\nThat's all you need to know to get started. Take a look at\n[this repo](https://github.com/pola-rs/pyo3-polars/tree/main/example/derive_expression) to see how\nthis all fits together, and at\n[this tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) to gain a more thorough\nunderstanding.\n"},"files":{"docs/source/user-guide/plugins/io_plugins.md":"# IO Plugins\n\nBesides [expression plugins](./expr_plugins.md), we also support IO plugins. These allow you to\nregister different file formats as sources to the Polars engines. Because sources can move data zero\ncopy via Arrow FFI and sources can produce large chunks of data before returning, we've decided to\ninterface to IO plugins via Python for now, as we don't think the short time the GIL is needed\nshould lead to any contention.\n\nE.g. an IO source can read their dataframe's in rust and only at the rendez-vous move the data\nzero-copy having only a short time the GIL is needed.\n\n## Use case\n\nYou want IO plugins if you have a source file not supported by Polars and you want to benefit from\noptimizations like projection pushdown, predicate pushdown, early stopping and support of our\nstreaming engine.\n\n## Example\n\nSo let's write a simple, very bad, custom CSV source and register that as an IO plugin. I want to\nstress that this is a very bad example and is only given for learning purposes.\n\nFirst we define some imports we need:\n\n```python\n# Use python for csv parsing.\nimport csv\nimport polars as pl\n# Used to register a new generator on every instantiation.\nfrom polars.io.plugins import register_io_source\nfrom typing import Iterator\nimport io\n```\n\n### Parsing the schema\n\nEvery `scan` function in Polars has to be able to provide the schema of the data it reads. For this\nsimple csv parser we will always read the data as `pl.String`. The only thing that differs are the\nfield names and the number of fields.\n\n```python\ndef parse_schema(csv_str: str) -> pl.Schema:\n    first_line = csv_str.split(\"\\n\")[0]\n\n    return pl.Schema({k: pl.String for k in first_line.split(\",\")})\n```\n\nIf we run this with small csv file `\"a,b,c\\n1,2,3\"` we get the schema:\n`Schema([('a', String), ('b', String), ('c', String)])`.\n\n```python\n>>> print(parse_schema(\"a,b,c\\n1,2,3\"))\nSchema([('a', String), ('b', String), ('c', String)])\n```\n\n### Writing the source\n\nNext up is the actual source. For this we create an outer and an inner function. The outer function\n`my_scan_csv` is the user facing function. This function will accept the file name and other\npotential arguments you would need for reading the source. For csv files, these arguments could be\n\"delimiter\", \"quote_char\" and such.\n\nThis outer function calls `register_io_source` which accepts a `callable` and a `schema`. The schema\nis the Polars schema of the complete source file (independent of projection pushdown).\n\nThe callable is a function that will return a generator that produces `pl.DataFrame` objects.\n\nThe arguments of this function are predefined and this function must accept:\n\n- `with_columns`\n\n  Columns that are projected. The reader must project these columns if applied\n\n- `predicate`\n\n  Polars expression. The reader must filter their rows accordingly.\n\n- `n_rows`\n\n  Materialize only n rows from the source. The reader can stop when `n_rows` are read.\n\n- `batch_size`\n\n  A hint of the ideal batch size the reader's generator must produce.\n\nThe inner function is the actual implementation of the IO source and can also call into Rust/C++ or\nwherever the IO plugin is written. If you want to see an IO source implemented in Rust, take a look\nat our [plugins repository](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin).\n\n```python\ndef my_scan_csv(csv_str: str) -> pl.LazyFrame:\n    schema = parse_schema(csv_str)\n\n    def source_generator(\n        with_columns: list[str] | None,\n        predicate: pl.Expr | None,\n        n_rows: int | None,\n        batch_size: int | None,\n    ) -> Iterator[pl.DataFrame]:\n        \"\"\"\n        Generator function that creates the source.\n        This function will be registered as IO source.\n        \"\"\"\n        if batch_size is None:\n            batch_size = 100\n\n        # Initialize the reader.\n        reader = csv.reader(io.StringIO(csv_str), delimiter=',')\n        # Skip the header.\n        _ = next(reader)\n\n        # Ensure we don't read more rows than requested from the engine\n        while n_rows is None or n_rows > 0:\n            if n_rows is not None:\n                batch_size = min(batch_size, n_rows)\n\n            rows = []\n\n            for _ in range(batch_size):\n                try:\n                    row = next(reader)\n                except StopIteration:\n                    n_rows = 0\n                    break\n                rows.append(row)\n\n            df = pl.from_records(rows, schema=schema, orient=\"row\")\n            n_rows -= df.height\n\n            # If we would make a performant reader, we would not read these\n            # columns at all.\n            if with_columns is not None:\n                df = df.select(with_columns)\n\n            # If the source supports predicate pushdown, the expression can be parsed\n            # to skip rows/groups.\n            if predicate is not None:\n                df = df.filter(predicate)\n\n            yield df\n\n    return register_io_source(io_source=source_generator, schema=schema)\n```\n\n### Taking it for a (very slow) spin\n\nFinally we can test our source:\n\n```python\ncsv_str1 = \"\"\"a,b,c,d\n1,2,3,4\n9,10,11,2\n1,2,3,4\n1,122,3,4\"\"\"\n\nprint(my_scan_csv(csv_str1).collect())\n\n\ncsv_str2 = \"\"\"a,b\n1,2\n9,10\n1,2\n1,122\"\"\"\n\nprint(my_scan_csv(csv_str2).head(2).collect())\n```\n\nRunning the script above would print the following output to the console:\n\n```\nshape: (4, 4)\n┌─────┬─────┬─────┬─────┐\n│ a   ┆ b   ┆ c   ┆ d   │\n│ --- ┆ --- ┆ --- ┆ --- │\n│ str ┆ str ┆ str ┆ str │\n╞═════╪═════╪═════╪═════╡\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 9   ┆ 10  ┆ 11  ┆ 2   │\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 1   ┆ 122 ┆ 3   ┆ 4   │\n└─────┴─────┴─────┴─────┘\nshape: (2, 2)\n┌─────┬─────┐\n│ a   ┆ b   │\n│ --- ┆ --- │\n│ str ┆ str │\n╞═════╪═════╡\n│ 1   ┆ 2   │\n│ 9   ┆ 10  │\n└─────┴─────┘\n```\n\n## For engine authors: identifying a source\n\nThis section is for people writing a Polars execution engine. If you are writing an IO source, you\ncan skip it.\n\nPolars does not call your source directly. It calls a wrapper that deserializes the predicate and\nforwards the call. Engines may need to identify sources they know about, for example to provide\nexecution context or call additional methods.\n\nTo make that possible, the wrapper satisfies the `IOSourceScanFunction` protocol, which exposes the\nregistered source:\n\n```python\nfrom polars.io.plugins import IOSourceScanFunction\n\n# `scan_fn` is the scan function of the `PythonScan` node being executed.\nif isinstance(scan_fn, IOSourceScanFunction):\n    source = scan_fn.io_source  # exactly the object passed to register_io_source\n    if isinstance(source, MyEngineSource):\n        ...\n```\n\nThis is as unstable as the rest of the IO plugin API.\n\n## Further reading\n\n- [Rust example (distribution source)](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin)\n","docs/source/user-guide/plugins/index.md":"# Plugins\n\nPolars allows you to extend its functionality with either Expression plugins or IO plugins.\n\n- [Expression plugins](./expr_plugins.md)\n- [IO plugins](./io_plugins.md)\n\n## Community plugins\n\nHere is a curated (non-exhaustive) list of community-implemented plugins.\n\n### Various\n\n- [polars-xdt](https://github.com/pola-rs/polars-xdt) Polars plugin with extra datetime-related\n  functionality which isn't quite in-scope for the main library\n- [polars-hash](https://github.com/ion-elgreco/polars-hash) Stable non-cryptographic and\n  cryptographic hashing functions for Polars\n\n### Data science\n\n- [polars-distance](https://github.com/ion-elgreco/polars-distance) Polars plugin for pairwise\n  distance functions\n- [polars-ds](https://github.com/abstractqqq/polars_ds_extension) Polars extension aiming to\n  simplify common numerical/string data analysis procedures\n\n### Bioinformatics\n\n- [polars-bio](https://github.com/biodatageeks/polars-bio) Python library for genomics built on top\n  of Polars, Apache Arrow and Apache DataFusion\n\n### Geo\n\n- [polars-st](https://github.com/Oreilles/polars-st) Polars ST provides spatial operations on Polars\n  DataFrames, Series and Expressions. Just like Shapely and Geopandas.\n- [polars-reverse-geocode](https://github.com/MarcoGorelli/polars-reverse-geocode) Offline reverse\n  geocoder for finding the closest city to a given (latitude, longitude) pair.\n- [polars-h3](https://github.com/Filimoa/polars-h3) This is a Polars extension that adds support for\n  the H3 discrete global grid system, so you can index points and geometries to hexagons directly in\n  Polars.\n\n## Other material\n\n- [Ritchie Vink - Keynote on Polars Plugins](https://youtu.be/jKW-CBV7NUM)\n- [Polars plugins tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) Learn how to\n  write a plugin by going through some very simple and minimal examples\n- [cookiecutter-polars-plugin](https://github.com/MarcoGorelli/cookiecutter-polars-plugins) Project\n  template for Polars Plugins\n","docs/source/user-guide/plugins/expr_plugins.md":"# Expression Plugins\n\n<!-- (TODO) the steps outlined here aren't clear and may not work that well -->\n\nExpression plugins are the preferred way to create user defined functions. They allow you to compile\na Rust function and register that as an expression into the Polars library. The Polars engine will\ndynamically link your function at runtime and your expression will run almost as fast as native\nexpressions. Note that this works without any interference of Python and thus no GIL contention.\n\nThey will benefit from the same benefits default expressions have:\n\n- Optimization\n- Parallelism\n- Rust native performance\n\nTo get started we will see what is needed to create a custom expression.\n\n## Our first custom expression: Pig Latin\n\nFor our first expression we are going to create a pig latin converter. Pig latin is a silly language\nwhere in every word the first letter is removed, added to the back and finally \"ay\" is added. So the\nword \"pig\" would convert to \"igpay\".\n\nWe could of course already do that with expressions, e.g.\n`col(\"name\").str.slice(1) + col(\"name\").str.slice(0, 1) + \"ay\"`, but a specialized function for this\nwould perform better and allows us to learn about the plugins.\n\n### Setting up\n\nWe start with a new library as the following `Cargo.toml` file\n\n```toml\n[package]\nname = \"expression_lib\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[lib]\nname = \"expression_lib\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\npolars = { version = \"*\" }\npyo3 = { version = \"*\", features = [\"extension-module\", \"abi3-py310\"] }\npyo3-polars = { version = \"*\", features = [\"derive\"] }\nserde = { version = \"*\", features = [\"derive\"] }\n```\n\n### Writing the expression\n\nIn this library we create a helper function that converts a `&str` to pig-latin, and we create the\nfunction that we will expose as an expression. To expose a function we must add the\n`#[polars_expr(output_type=DataType)]` attribute and the function must always accept\n`inputs: &[Series]` as its first argument.\n\n```rust\n// src/expressions.rs\nuse polars::prelude::*;\nuse pyo3_polars::derive::polars_expr;\nuse std::fmt::Write;\n\nfn pig_latin_str(value: &str, output: &mut String) {\n    if let Some(first_char) = value.chars().next() {\n        write!(output, \"{}{}ay\", &value[1..], first_char).unwrap()\n    }\n}\n\n#[polars_expr(output_type=String)]\nfn pig_latinnify(inputs: &[Series]) -> PolarsResult<Series> {\n    let ca = inputs[0].str()?;\n    let out: StringChunked = ca.apply_into_string_amortized(pig_latin_str);\n    Ok(out.into_series())\n}\n```\n\nNote that we use `apply_into_string_amortized`, as opposed to `apply_values`, to avoid allocating a\nnew string for each row. If your plugin takes in multiple inputs, operates elementwise, and produces\na `String` output, then you may want to look at the `binary_elementwise_into_string_amortized`\nutility function in `polars::prelude::arity`.\n\nThis is all that is needed on the Rust side. On the Python side we must setup a folder with the same\nname as defined in the `Cargo.toml`, in this case \"expression_lib\". We will create a folder in the\nsame directory as our Rust `src` folder named `expression_lib` and we create an\n`expression_lib/__init__.py`. The resulting file structure should look something like this:\n\n```\n├── 📁 expression_lib/  # name must match \"lib.name\" in Cargo.toml\n|   └── __init__.py\n|\n├── 📁src/\n|   ├── lib.rs\n|   └── expressions.rs\n|\n├── Cargo.toml\n└── pyproject.toml\n```\n\nThen we create new expressions. The function name of our expression can be registered. Note that it\nis important that this name is correct, otherwise the main Polars package cannot resolve the\nfunction name. Furthermore we can set additional keyword arguments that explain to Polars how this\nexpression behaves. In this case we tell Polars that this function is elementwise. This allows\nPolars to run this expression in batches. Whereas for other operations this would not be allowed,\nthink for instance of a sort, or a slice.\n\n```python\n# expression_lib/__init__.py\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING\n\nimport polars as pl\nfrom polars.plugins import register_plugin_function\nfrom polars._typing import IntoExpr\n\nPLUGIN_PATH = Path(__file__).parent\n\ndef pig_latinnify(expr: IntoExpr) -> pl.Expr:\n    \"\"\"Pig-latinnify expression.\"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"pig_latinnify\",\n        args=expr,\n        is_elementwise=True,\n    )\n```\n\nWe can then compile this library in our environment by installing `maturin` and running\n`maturin develop --release`.\n\nAnd that's it. Our expression is ready to use!\n\n```python\nimport polars as pl\nfrom expression_lib import pig_latinnify\n\ndf = pl.DataFrame(\n    {\n        \"convert\": [\"pig\", \"latin\", \"is\", \"silly\"],\n    }\n)\nout = df.with_columns(pig_latin=pig_latinnify(\"convert\"))\n```\n\nAlternatively, you can\n[register a custom namespace](https://docs.pola.rs/api/python/stable/reference/api/polars.api.register_expr_namespace.html#polars.api.register_expr_namespace),\nwhich enables you to create a `Expr.language` namespace, allowing users to write:\n\n```python\nout = df.with_columns(\n    pig_latin=pl.col(\"convert\").language.pig_latinnify(),\n)\n```\n\n## Accepting kwargs\n\nIf you want to accept `kwargs` (keyword arguments) in a polars expression, all you have to do is\ndefine a Rust `struct` and make sure that it derives `serde::Deserialize`.\n\n```rust\n/// Provide your own kwargs struct with the proper schema and accept that type\n/// in your plugin expression.\n#[derive(Deserialize)]\npub struct MyKwargs {\n    float_arg: f64,\n    integer_arg: i64,\n    string_arg: String,\n    boolean_arg: bool,\n}\n\n/// If you want to accept `kwargs`. You define a `kwargs` argument\n/// on the second position in you plugin. You can provide any custom struct that is deserializable\n/// with the pickle protocol (on the Rust side).\n#[polars_expr(output_type=String)]\nfn append_kwargs(input: &[Series], kwargs: MyKwargs) -> PolarsResult<Series> {\n    let input = &input[0];\n    let input = input.cast(&DataType::String)?;\n    let ca = input.str().unwrap();\n\n    Ok(ca\n        .apply_into_string_amortized(|val, buf| {\n            write!(\n                buf,\n                \"{}-{}-{}-{}-{}\",\n                val, kwargs.float_arg, kwargs.integer_arg, kwargs.string_arg, kwargs.boolean_arg\n            )\n                .unwrap()\n        })\n        .into_series())\n}\n```\n\nOn the Python side the kwargs can be passed when we register the plugin.\n\n```python\ndef append_args(\n    expr: IntoExpr,\n    float_arg: float,\n    integer_arg: int,\n    string_arg: str,\n    boolean_arg: bool,\n) -> pl.Expr:\n    \"\"\"\n    This example shows how arguments other than `Series` can be used.\n    \"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"append_kwargs\",\n        args=expr,\n        kwargs={\n            \"float_arg\": float_arg,\n            \"integer_arg\": integer_arg,\n            \"string_arg\": string_arg,\n            \"boolean_arg\": boolean_arg,\n        },\n        is_elementwise=True,\n    )\n```\n\n## Output data types\n\nOutput data types of course don't have to be fixed. They often depend on the input types of an\nexpression. To accommodate this you can provide the `#[polars_expr()]` macro with an\n`output_type_func` argument that points to a function. This function can map input fields `&[Field]`\nto an output `Field` (name and data type).\n\nIn the snippet below is an example where we use the utility `FieldsMapper` to help with this\nmapping.\n\n```rust\nuse polars_plan::dsl::FieldsMapper;\n\nfn haversine_output(input_fields: &[Field]) -> PolarsResult<Field> {\n    FieldsMapper::new(input_fields).map_to_float_dtype()\n}\n\n#[polars_expr(output_type_func=haversine_output)]\nfn haversine(inputs: &[Series]) -> PolarsResult<Series> {\n    let out = match inputs[0].dtype() {\n        DataType::Float32 => {\n            let start_lat = inputs[0].f32().unwrap();\n            let start_long = inputs[1].f32().unwrap();\n            let end_lat = inputs[2].f32().unwrap();\n            let end_long = inputs[3].f32().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        DataType::Float64 => {\n            let start_lat = inputs[0].f64().unwrap();\n            let start_long = inputs[1].f64().unwrap();\n            let end_lat = inputs[2].f64().unwrap();\n            let end_long = inputs[3].f64().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        _ => polars_bail!(InvalidOperation: \"only supported for float types\"),\n    };\n    Ok(out)\n}\n```\n\nThat's all you need to know to get started. Take a look at\n[this repo](https://github.com/pola-rs/pyo3-polars/tree/main/example/derive_expression) to see how\nthis all fits together, and at\n[this tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) to gain a more thorough\nunderstanding.\n"},"items":[{"name":"expr_plugins.md","path":"docs/source/user-guide/plugins/expr_plugins.md","rawUrl":"https://raw.githubusercontent.com/pola-rs/polars/HEAD/docs/source/user-guide/plugins/expr_plugins.md","title":"Plugins Skill","category":"anthropic-skill","format":"markdown","content":"# Expression Plugins\n\n<!-- (TODO) the steps outlined here aren't clear and may not work that well -->\n\nExpression plugins are the preferred way to create user defined functions. They allow you to compile\na Rust function and register that as an expression into the Polars library. The Polars engine will\ndynamically link your function at runtime and your expression will run almost as fast as native\nexpressions. Note that this works without any interference of Python and thus no GIL contention.\n\nThey will benefit from the same benefits default expressions have:\n\n- Optimization\n- Parallelism\n- Rust native performance\n\nTo get started we will see what is needed to create a custom expression.\n\n## Our first custom expression: Pig Latin\n\nFor our first expression we are going to create a pig latin converter. Pig latin is a silly language\nwhere in every word the first letter is removed, added to the back and finally \"ay\" is added. So the\nword \"pig\" would convert to \"igpay\".\n\nWe could of course already do that with expressions, e.g.\n`col(\"name\").str.slice(1) + col(\"name\").str.slice(0, 1) + \"ay\"`, but a specialized function for this\nwould perform better and allows us to learn about the plugins.\n\n### Setting up\n\nWe start with a new library as the following `Cargo.toml` file\n\n```toml\n[package]\nname = \"expression_lib\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[lib]\nname = \"expression_lib\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\npolars = { version = \"*\" }\npyo3 = { version = \"*\", features = [\"extension-module\", \"abi3-py310\"] }\npyo3-polars = { version = \"*\", features = [\"derive\"] }\nserde = { version = \"*\", features = [\"derive\"] }\n```\n\n### Writing the expression\n\nIn this library we create a helper function that converts a `&str` to pig-latin, and we create the\nfunction that we will expose as an expression. To expose a function we must add the\n`#[polars_expr(output_type=DataType)]` attribute and the function must always accept\n`inputs: &[Series]` as its first argument.\n\n```rust\n// src/expressions.rs\nuse polars::prelude::*;\nuse pyo3_polars::derive::polars_expr;\nuse std::fmt::Write;\n\nfn pig_latin_str(value: &str, output: &mut String) {\n    if let Some(first_char) = value.chars().next() {\n        write!(output, \"{}{}ay\", &value[1..], first_char).unwrap()\n    }\n}\n\n#[polars_expr(output_type=String)]\nfn pig_latinnify(inputs: &[Series]) -> PolarsResult<Series> {\n    let ca = inputs[0].str()?;\n    let out: StringChunked = ca.apply_into_string_amortized(pig_latin_str);\n    Ok(out.into_series())\n}\n```\n\nNote that we use `apply_into_string_amortized`, as opposed to `apply_values`, to avoid allocating a\nnew string for each row. If your plugin takes in multiple inputs, operates elementwise, and produces\na `String` output, then you may want to look at the `binary_elementwise_into_string_amortized`\nutility function in `polars::prelude::arity`.\n\nThis is all that is needed on the Rust side. On the Python side we must setup a folder with the same\nname as defined in the `Cargo.toml`, in this case \"expression_lib\". We will create a folder in the\nsame directory as our Rust `src` folder named `expression_lib` and we create an\n`expression_lib/__init__.py`. The resulting file structure should look something like this:\n\n```\n├── 📁 expression_lib/  # name must match \"lib.name\" in Cargo.toml\n|   └── __init__.py\n|\n├── 📁src/\n|   ├── lib.rs\n|   └── expressions.rs\n|\n├── Cargo.toml\n└── pyproject.toml\n```\n\nThen we create new expressions. The function name of our expression can be registered. Note that it\nis important that this name is correct, otherwise the main Polars package cannot resolve the\nfunction name. Furthermore we can set additional keyword arguments that explain to Polars how this\nexpression behaves. In this case we tell Polars that this function is elementwise. This allows\nPolars to run this expression in batches. Whereas for other operations this would not be allowed,\nthink for instance of a sort, or a slice.\n\n```python\n# expression_lib/__init__.py\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING\n\nimport polars as pl\nfrom polars.plugins import register_plugin_function\nfrom polars._typing import IntoExpr\n\nPLUGIN_PATH = Path(__file__).parent\n\ndef pig_latinnify(expr: IntoExpr) -> pl.Expr:\n    \"\"\"Pig-latinnify expression.\"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"pig_latinnify\",\n        args=expr,\n        is_elementwise=True,\n    )\n```\n\nWe can then compile this library in our environment by installing `maturin` and running\n`maturin develop --release`.\n\nAnd that's it. Our expression is ready to use!\n\n```python\nimport polars as pl\nfrom expression_lib import pig_latinnify\n\ndf = pl.DataFrame(\n    {\n        \"convert\": [\"pig\", \"latin\", \"is\", \"silly\"],\n    }\n)\nout = df.with_columns(pig_latin=pig_latinnify(\"convert\"))\n```\n\nAlternatively, you can\n[register a custom namespace](https://docs.pola.rs/api/python/stable/reference/api/polars.api.register_expr_namespace.html#polars.api.register_expr_namespace),\nwhich enables you to create a `Expr.language` namespace, allowing users to write:\n\n```python\nout = df.with_columns(\n    pig_latin=pl.col(\"convert\").language.pig_latinnify(),\n)\n```\n\n## Accepting kwargs\n\nIf you want to accept `kwargs` (keyword arguments) in a polars expression, all you have to do is\ndefine a Rust `struct` and make sure that it derives `serde::Deserialize`.\n\n```rust\n/// Provide your own kwargs struct with the proper schema and accept that type\n/// in your plugin expression.\n#[derive(Deserialize)]\npub struct MyKwargs {\n    float_arg: f64,\n    integer_arg: i64,\n    string_arg: String,\n    boolean_arg: bool,\n}\n\n/// If you want to accept `kwargs`. You define a `kwargs` argument\n/// on the second position in you plugin. You can provide any custom struct that is deserializable\n/// with the pickle protocol (on the Rust side).\n#[polars_expr(output_type=String)]\nfn append_kwargs(input: &[Series], kwargs: MyKwargs) -> PolarsResult<Series> {\n    let input = &input[0];\n    let input = input.cast(&DataType::String)?;\n    let ca = input.str().unwrap();\n\n    Ok(ca\n        .apply_into_string_amortized(|val, buf| {\n            write!(\n                buf,\n                \"{}-{}-{}-{}-{}\",\n                val, kwargs.float_arg, kwargs.integer_arg, kwargs.string_arg, kwargs.boolean_arg\n            )\n                .unwrap()\n        })\n        .into_series())\n}\n```\n\nOn the Python side the kwargs can be passed when we register the plugin.\n\n```python\ndef append_args(\n    expr: IntoExpr,\n    float_arg: float,\n    integer_arg: int,\n    string_arg: str,\n    boolean_arg: bool,\n) -> pl.Expr:\n    \"\"\"\n    This example shows how arguments other than `Series` can be used.\n    \"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"append_kwargs\",\n        args=expr,\n        kwargs={\n            \"float_arg\": float_arg,\n            \"integer_arg\": integer_arg,\n            \"string_arg\": string_arg,\n            \"boolean_arg\": boolean_arg,\n        },\n        is_elementwise=True,\n    )\n```\n\n## Output data types\n\nOutput data types of course don't have to be fixed. They often depend on the input types of an\nexpression. To accommodate this you can provide the `#[polars_expr()]` macro with an\n`output_type_func` argument that points to a function. This function can map input fields `&[Field]`\nto an output `Field` (name and data type).\n\nIn the snippet below is an example where we use the utility `FieldsMapper` to help with this\nmapping.\n\n```rust\nuse polars_plan::dsl::FieldsMapper;\n\nfn haversine_output(input_fields: &[Field]) -> PolarsResult<Field> {\n    FieldsMapper::new(input_fields).map_to_float_dtype()\n}\n\n#[polars_expr(output_type_func=haversine_output)]\nfn haversine(inputs: &[Series]) -> PolarsResult<Series> {\n    let out = match inputs[0].dtype() {\n        DataType::Float32 => {\n            let start_lat = inputs[0].f32().unwrap();\n            let start_long = inputs[1].f32().unwrap();\n            let end_lat = inputs[2].f32().unwrap();\n            let end_long = inputs[3].f32().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        DataType::Float64 => {\n            let start_lat = inputs[0].f64().unwrap();\n            let start_long = inputs[1].f64().unwrap();\n            let end_lat = inputs[2].f64().unwrap();\n            let end_long = inputs[3].f64().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        _ => polars_bail!(InvalidOperation: \"only supported for float types\"),\n    };\n    Ok(out)\n}\n```\n\nThat's all you need to know to get started. Take a look at\n[this repo](https://github.com/pola-rs/pyo3-polars/tree/main/example/derive_expression) to see how\nthis all fits together, and at\n[this tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) to gain a more thorough\nunderstanding.\n","isInternal":false,"tokens":2199,"sizeBytes":9017},{"name":"index.md","path":"docs/source/user-guide/plugins/index.md","rawUrl":"https://raw.githubusercontent.com/pola-rs/polars/HEAD/docs/source/user-guide/plugins/index.md","title":"Plugins Skill","category":"anthropic-skill","format":"markdown","content":"# Plugins\n\nPolars allows you to extend its functionality with either Expression plugins or IO plugins.\n\n- [Expression plugins](./expr_plugins.md)\n- [IO plugins](./io_plugins.md)\n\n## Community plugins\n\nHere is a curated (non-exhaustive) list of community-implemented plugins.\n\n### Various\n\n- [polars-xdt](https://github.com/pola-rs/polars-xdt) Polars plugin with extra datetime-related\n  functionality which isn't quite in-scope for the main library\n- [polars-hash](https://github.com/ion-elgreco/polars-hash) Stable non-cryptographic and\n  cryptographic hashing functions for Polars\n\n### Data science\n\n- [polars-distance](https://github.com/ion-elgreco/polars-distance) Polars plugin for pairwise\n  distance functions\n- [polars-ds](https://github.com/abstractqqq/polars_ds_extension) Polars extension aiming to\n  simplify common numerical/string data analysis procedures\n\n### Bioinformatics\n\n- [polars-bio](https://github.com/biodatageeks/polars-bio) Python library for genomics built on top\n  of Polars, Apache Arrow and Apache DataFusion\n\n### Geo\n\n- [polars-st](https://github.com/Oreilles/polars-st) Polars ST provides spatial operations on Polars\n  DataFrames, Series and Expressions. Just like Shapely and Geopandas.\n- [polars-reverse-geocode](https://github.com/MarcoGorelli/polars-reverse-geocode) Offline reverse\n  geocoder for finding the closest city to a given (latitude, longitude) pair.\n- [polars-h3](https://github.com/Filimoa/polars-h3) This is a Polars extension that adds support for\n  the H3 discrete global grid system, so you can index points and geometries to hexagons directly in\n  Polars.\n\n## Other material\n\n- [Ritchie Vink - Keynote on Polars Plugins](https://youtu.be/jKW-CBV7NUM)\n- [Polars plugins tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) Learn how to\n  write a plugin by going through some very simple and minimal examples\n- [cookiecutter-polars-plugin](https://github.com/MarcoGorelli/cookiecutter-polars-plugins) Project\n  template for Polars Plugins\n","isInternal":false,"tokens":515,"sizeBytes":2007},{"name":"io_plugins.md","path":"docs/source/user-guide/plugins/io_plugins.md","rawUrl":"https://raw.githubusercontent.com/pola-rs/polars/HEAD/docs/source/user-guide/plugins/io_plugins.md","title":"Plugins Skill","category":"anthropic-skill","format":"markdown","content":"# IO Plugins\n\nBesides [expression plugins](./expr_plugins.md), we also support IO plugins. These allow you to\nregister different file formats as sources to the Polars engines. Because sources can move data zero\ncopy via Arrow FFI and sources can produce large chunks of data before returning, we've decided to\ninterface to IO plugins via Python for now, as we don't think the short time the GIL is needed\nshould lead to any contention.\n\nE.g. an IO source can read their dataframe's in rust and only at the rendez-vous move the data\nzero-copy having only a short time the GIL is needed.\n\n## Use case\n\nYou want IO plugins if you have a source file not supported by Polars and you want to benefit from\noptimizations like projection pushdown, predicate pushdown, early stopping and support of our\nstreaming engine.\n\n## Example\n\nSo let's write a simple, very bad, custom CSV source and register that as an IO plugin. I want to\nstress that this is a very bad example and is only given for learning purposes.\n\nFirst we define some imports we need:\n\n```python\n# Use python for csv parsing.\nimport csv\nimport polars as pl\n# Used to register a new generator on every instantiation.\nfrom polars.io.plugins import register_io_source\nfrom typing import Iterator\nimport io\n```\n\n### Parsing the schema\n\nEvery `scan` function in Polars has to be able to provide the schema of the data it reads. For this\nsimple csv parser we will always read the data as `pl.String`. The only thing that differs are the\nfield names and the number of fields.\n\n```python\ndef parse_schema(csv_str: str) -> pl.Schema:\n    first_line = csv_str.split(\"\\n\")[0]\n\n    return pl.Schema({k: pl.String for k in first_line.split(\",\")})\n```\n\nIf we run this with small csv file `\"a,b,c\\n1,2,3\"` we get the schema:\n`Schema([('a', String), ('b', String), ('c', String)])`.\n\n```python\n>>> print(parse_schema(\"a,b,c\\n1,2,3\"))\nSchema([('a', String), ('b', String), ('c', String)])\n```\n\n### Writing the source\n\nNext up is the actual source. For this we create an outer and an inner function. The outer function\n`my_scan_csv` is the user facing function. This function will accept the file name and other\npotential arguments you would need for reading the source. For csv files, these arguments could be\n\"delimiter\", \"quote_char\" and such.\n\nThis outer function calls `register_io_source` which accepts a `callable` and a `schema`. The schema\nis the Polars schema of the complete source file (independent of projection pushdown).\n\nThe callable is a function that will return a generator that produces `pl.DataFrame` objects.\n\nThe arguments of this function are predefined and this function must accept:\n\n- `with_columns`\n\n  Columns that are projected. The reader must project these columns if applied\n\n- `predicate`\n\n  Polars expression. The reader must filter their rows accordingly.\n\n- `n_rows`\n\n  Materialize only n rows from the source. The reader can stop when `n_rows` are read.\n\n- `batch_size`\n\n  A hint of the ideal batch size the reader's generator must produce.\n\nThe inner function is the actual implementation of the IO source and can also call into Rust/C++ or\nwherever the IO plugin is written. If you want to see an IO source implemented in Rust, take a look\nat our [plugins repository](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin).\n\n```python\ndef my_scan_csv(csv_str: str) -> pl.LazyFrame:\n    schema = parse_schema(csv_str)\n\n    def source_generator(\n        with_columns: list[str] | None,\n        predicate: pl.Expr | None,\n        n_rows: int | None,\n        batch_size: int | None,\n    ) -> Iterator[pl.DataFrame]:\n        \"\"\"\n        Generator function that creates the source.\n        This function will be registered as IO source.\n        \"\"\"\n        if batch_size is None:\n            batch_size = 100\n\n        # Initialize the reader.\n        reader = csv.reader(io.StringIO(csv_str), delimiter=',')\n        # Skip the header.\n        _ = next(reader)\n\n        # Ensure we don't read more rows than requested from the engine\n        while n_rows is None or n_rows > 0:\n            if n_rows is not None:\n                batch_size = min(batch_size, n_rows)\n\n            rows = []\n\n            for _ in range(batch_size):\n                try:\n                    row = next(reader)\n                except StopIteration:\n                    n_rows = 0\n                    break\n                rows.append(row)\n\n            df = pl.from_records(rows, schema=schema, orient=\"row\")\n            n_rows -= df.height\n\n            # If we would make a performant reader, we would not read these\n            # columns at all.\n            if with_columns is not None:\n                df = df.select(with_columns)\n\n            # If the source supports predicate pushdown, the expression can be parsed\n            # to skip rows/groups.\n            if predicate is not None:\n                df = df.filter(predicate)\n\n            yield df\n\n    return register_io_source(io_source=source_generator, schema=schema)\n```\n\n### Taking it for a (very slow) spin\n\nFinally we can test our source:\n\n```python\ncsv_str1 = \"\"\"a,b,c,d\n1,2,3,4\n9,10,11,2\n1,2,3,4\n1,122,3,4\"\"\"\n\nprint(my_scan_csv(csv_str1).collect())\n\n\ncsv_str2 = \"\"\"a,b\n1,2\n9,10\n1,2\n1,122\"\"\"\n\nprint(my_scan_csv(csv_str2).head(2).collect())\n```\n\nRunning the script above would print the following output to the console:\n\n```\nshape: (4, 4)\n┌─────┬─────┬─────┬─────┐\n│ a   ┆ b   ┆ c   ┆ d   │\n│ --- ┆ --- ┆ --- ┆ --- │\n│ str ┆ str ┆ str ┆ str │\n╞═════╪═════╪═════╪═════╡\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 9   ┆ 10  ┆ 11  ┆ 2   │\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 1   ┆ 122 ┆ 3   ┆ 4   │\n└─────┴─────┴─────┴─────┘\nshape: (2, 2)\n┌─────┬─────┐\n│ a   ┆ b   │\n│ --- ┆ --- │\n│ str ┆ str │\n╞═════╪═════╡\n│ 1   ┆ 2   │\n│ 9   ┆ 10  │\n└─────┴─────┘\n```\n\n## For engine authors: identifying a source\n\nThis section is for people writing a Polars execution engine. If you are writing an IO source, you\ncan skip it.\n\nPolars does not call your source directly. It calls a wrapper that deserializes the predicate and\nforwards the call. Engines may need to identify sources they know about, for example to provide\nexecution context or call additional methods.\n\nTo make that possible, the wrapper satisfies the `IOSourceScanFunction` protocol, which exposes the\nregistered source:\n\n```python\nfrom polars.io.plugins import IOSourceScanFunction\n\n# `scan_fn` is the scan function of the `PythonScan` node being executed.\nif isinstance(scan_fn, IOSourceScanFunction):\n    source = scan_fn.io_source  # exactly the object passed to register_io_source\n    if isinstance(source, MyEngineSource):\n        ...\n```\n\nThis is as unstable as the rest of the IO plugin API.\n\n## Further reading\n\n- [Rust example (distribution source)](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin)\n","isInternal":false,"tokens":1757,"sizeBytes":7112}],"systemPromptSnippet":"<yakaai_skills repo=\"pola-rs/polars\">\n<!-- File: docs/source/user-guide/plugins/expr_plugins.md (Tokens: ~2199 | Category: anthropic-skill) -->\n# Expression Plugins\n\n<!-- (TODO) the steps outlined here aren't clear and may not work that well -->\n\nExpression plugins are the preferred way to create user defined functions. They allow you to compile\na Rust function and register that as an expression into the Polars library. The Polars engine will\ndynamically link your function at runtime and your expression will run almost as fast as native\nexpressions. Note that this works without any interference of Python and thus no GIL contention.\n\nThey will benefit from the same benefits default expressions have:\n\n- Optimization\n- Parallelism\n- Rust native performance\n\nTo get started we will see what is needed to create a custom expression.\n\n## Our first custom expression: Pig Latin\n\nFor our first expression we are going to create a pig latin converter. Pig latin is a silly language\nwhere in every word the first letter is removed, added to the back and finally \"ay\" is added. So the\nword \"pig\" would convert to \"igpay\".\n\nWe could of course already do that with expressions, e.g.\n`col(\"name\").str.slice(1) + col(\"name\").str.slice(0, 1) + \"ay\"`, but a specialized function for this\nwould perform better and allows us to learn about the plugins.\n\n### Setting up\n\nWe start with a new library as the following `Cargo.toml` file\n\n```toml\n[package]\nname = \"expression_lib\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[lib]\nname = \"expression_lib\"\ncrate-type = [\"cdylib\"]\n\n[dependencies]\npolars = { version = \"*\" }\npyo3 = { version = \"*\", features = [\"extension-module\", \"abi3-py310\"] }\npyo3-polars = { version = \"*\", features = [\"derive\"] }\nserde = { version = \"*\", features = [\"derive\"] }\n```\n\n### Writing the expression\n\nIn this library we create a helper function that converts a `&str` to pig-latin, and we create the\nfunction that we will expose as an expression. To expose a function we must add the\n`#[polars_expr(output_type=DataType)]` attribute and the function must always accept\n`inputs: &[Series]` as its first argument.\n\n```rust\n// src/expressions.rs\nuse polars::prelude::*;\nuse pyo3_polars::derive::polars_expr;\nuse std::fmt::Write;\n\nfn pig_latin_str(value: &str, output: &mut String) {\n    if let Some(first_char) = value.chars().next() {\n        write!(output, \"{}{}ay\", &value[1..], first_char).unwrap()\n    }\n}\n\n#[polars_expr(output_type=String)]\nfn pig_latinnify(inputs: &[Series]) -> PolarsResult<Series> {\n    let ca = inputs[0].str()?;\n    let out: StringChunked = ca.apply_into_string_amortized(pig_latin_str);\n    Ok(out.into_series())\n}\n```\n\nNote that we use `apply_into_string_amortized`, as opposed to `apply_values`, to avoid allocating a\nnew string for each row. If your plugin takes in multiple inputs, operates elementwise, and produces\na `String` output, then you may want to look at the `binary_elementwise_into_string_amortized`\nutility function in `polars::prelude::arity`.\n\nThis is all that is needed on the Rust side. On the Python side we must setup a folder with the same\nname as defined in the `Cargo.toml`, in this case \"expression_lib\". We will create a folder in the\nsame directory as our Rust `src` folder named `expression_lib` and we create an\n`expression_lib/__init__.py`. The resulting file structure should look something like this:\n\n```\n├── 📁 expression_lib/  # name must match \"lib.name\" in Cargo.toml\n|   └── __init__.py\n|\n├── 📁src/\n|   ├── lib.rs\n|   └── expressions.rs\n|\n├── Cargo.toml\n└── pyproject.toml\n```\n\nThen we create new expressions. The function name of our expression can be registered. Note that it\nis important that this name is correct, otherwise the main Polars package cannot resolve the\nfunction name. Furthermore we can set additional keyword arguments that explain to Polars how this\nexpression behaves. In this case we tell Polars that this function is elementwise. This allows\nPolars to run this expression in batches. Whereas for other operations this would not be allowed,\nthink for instance of a sort, or a slice.\n\n```python\n# expression_lib/__init__.py\nfrom pathlib import Path\nfrom typing import TYPE_CHECKING\n\nimport polars as pl\nfrom polars.plugins import register_plugin_function\nfrom polars._typing import IntoExpr\n\nPLUGIN_PATH = Path(__file__).parent\n\ndef pig_latinnify(expr: IntoExpr) -> pl.Expr:\n    \"\"\"Pig-latinnify expression.\"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"pig_latinnify\",\n        args=expr,\n        is_elementwise=True,\n    )\n```\n\nWe can then compile this library in our environment by installing `maturin` and running\n`maturin develop --release`.\n\nAnd that's it. Our expression is ready to use!\n\n```python\nimport polars as pl\nfrom expression_lib import pig_latinnify\n\ndf = pl.DataFrame(\n    {\n        \"convert\": [\"pig\", \"latin\", \"is\", \"silly\"],\n    }\n)\nout = df.with_columns(pig_latin=pig_latinnify(\"convert\"))\n```\n\nAlternatively, you can\n[register a custom namespace](https://docs.pola.rs/api/python/stable/reference/api/polars.api.register_expr_namespace.html#polars.api.register_expr_namespace),\nwhich enables you to create a `Expr.language` namespace, allowing users to write:\n\n```python\nout = df.with_columns(\n    pig_latin=pl.col(\"convert\").language.pig_latinnify(),\n)\n```\n\n## Accepting kwargs\n\nIf you want to accept `kwargs` (keyword arguments) in a polars expression, all you have to do is\ndefine a Rust `struct` and make sure that it derives `serde::Deserialize`.\n\n```rust\n/// Provide your own kwargs struct with the proper schema and accept that type\n/// in your plugin expression.\n#[derive(Deserialize)]\npub struct MyKwargs {\n    float_arg: f64,\n    integer_arg: i64,\n    string_arg: String,\n    boolean_arg: bool,\n}\n\n/// If you want to accept `kwargs`. You define a `kwargs` argument\n/// on the second position in you plugin. You can provide any custom struct that is deserializable\n/// with the pickle protocol (on the Rust side).\n#[polars_expr(output_type=String)]\nfn append_kwargs(input: &[Series], kwargs: MyKwargs) -> PolarsResult<Series> {\n    let input = &input[0];\n    let input = input.cast(&DataType::String)?;\n    let ca = input.str().unwrap();\n\n    Ok(ca\n        .apply_into_string_amortized(|val, buf| {\n            write!(\n                buf,\n                \"{}-{}-{}-{}-{}\",\n                val, kwargs.float_arg, kwargs.integer_arg, kwargs.string_arg, kwargs.boolean_arg\n            )\n                .unwrap()\n        })\n        .into_series())\n}\n```\n\nOn the Python side the kwargs can be passed when we register the plugin.\n\n```python\ndef append_args(\n    expr: IntoExpr,\n    float_arg: float,\n    integer_arg: int,\n    string_arg: str,\n    boolean_arg: bool,\n) -> pl.Expr:\n    \"\"\"\n    This example shows how arguments other than `Series` can be used.\n    \"\"\"\n    return register_plugin_function(\n        plugin_path=PLUGIN_PATH,\n        function_name=\"append_kwargs\",\n        args=expr,\n        kwargs={\n            \"float_arg\": float_arg,\n            \"integer_arg\": integer_arg,\n            \"string_arg\": string_arg,\n            \"boolean_arg\": boolean_arg,\n        },\n        is_elementwise=True,\n    )\n```\n\n## Output data types\n\nOutput data types of course don't have to be fixed. They often depend on the input types of an\nexpression. To accommodate this you can provide the `#[polars_expr()]` macro with an\n`output_type_func` argument that points to a function. This function can map input fields `&[Field]`\nto an output `Field` (name and data type).\n\nIn the snippet below is an example where we use the utility `FieldsMapper` to help with this\nmapping.\n\n```rust\nuse polars_plan::dsl::FieldsMapper;\n\nfn haversine_output(input_fields: &[Field]) -> PolarsResult<Field> {\n    FieldsMapper::new(input_fields).map_to_float_dtype()\n}\n\n#[polars_expr(output_type_func=haversine_output)]\nfn haversine(inputs: &[Series]) -> PolarsResult<Series> {\n    let out = match inputs[0].dtype() {\n        DataType::Float32 => {\n            let start_lat = inputs[0].f32().unwrap();\n            let start_long = inputs[1].f32().unwrap();\n            let end_lat = inputs[2].f32().unwrap();\n            let end_long = inputs[3].f32().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        DataType::Float64 => {\n            let start_lat = inputs[0].f64().unwrap();\n            let start_long = inputs[1].f64().unwrap();\n            let end_lat = inputs[2].f64().unwrap();\n            let end_long = inputs[3].f64().unwrap();\n            crate::distances::naive_haversine(start_lat, start_long, end_lat, end_long)?\n                .into_series()\n        }\n        _ => polars_bail!(InvalidOperation: \"only supported for float types\"),\n    };\n    Ok(out)\n}\n```\n\nThat's all you need to know to get started. Take a look at\n[this repo](https://github.com/pola-rs/pyo3-polars/tree/main/example/derive_expression) to see how\nthis all fits together, and at\n[this tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) to gain a more thorough\nunderstanding.\n\n<!-- File: docs/source/user-guide/plugins/index.md (Tokens: ~515 | Category: anthropic-skill) -->\n# Plugins\n\nPolars allows you to extend its functionality with either Expression plugins or IO plugins.\n\n- [Expression plugins](./expr_plugins.md)\n- [IO plugins](./io_plugins.md)\n\n## Community plugins\n\nHere is a curated (non-exhaustive) list of community-implemented plugins.\n\n### Various\n\n- [polars-xdt](https://github.com/pola-rs/polars-xdt) Polars plugin with extra datetime-related\n  functionality which isn't quite in-scope for the main library\n- [polars-hash](https://github.com/ion-elgreco/polars-hash) Stable non-cryptographic and\n  cryptographic hashing functions for Polars\n\n### Data science\n\n- [polars-distance](https://github.com/ion-elgreco/polars-distance) Polars plugin for pairwise\n  distance functions\n- [polars-ds](https://github.com/abstractqqq/polars_ds_extension) Polars extension aiming to\n  simplify common numerical/string data analysis procedures\n\n### Bioinformatics\n\n- [polars-bio](https://github.com/biodatageeks/polars-bio) Python library for genomics built on top\n  of Polars, Apache Arrow and Apache DataFusion\n\n### Geo\n\n- [polars-st](https://github.com/Oreilles/polars-st) Polars ST provides spatial operations on Polars\n  DataFrames, Series and Expressions. Just like Shapely and Geopandas.\n- [polars-reverse-geocode](https://github.com/MarcoGorelli/polars-reverse-geocode) Offline reverse\n  geocoder for finding the closest city to a given (latitude, longitude) pair.\n- [polars-h3](https://github.com/Filimoa/polars-h3) This is a Polars extension that adds support for\n  the H3 discrete global grid system, so you can index points and geometries to hexagons directly in\n  Polars.\n\n## Other material\n\n- [Ritchie Vink - Keynote on Polars Plugins](https://youtu.be/jKW-CBV7NUM)\n- [Polars plugins tutorial](https://marcogorelli.github.io/polars-plugins-tutorial/) Learn how to\n  write a plugin by going through some very simple and minimal examples\n- [cookiecutter-polars-plugin](https://github.com/MarcoGorelli/cookiecutter-polars-plugins) Project\n  template for Polars Plugins\n\n<!-- File: docs/source/user-guide/plugins/io_plugins.md (Tokens: ~1757 | Category: anthropic-skill) -->\n# IO Plugins\n\nBesides [expression plugins](./expr_plugins.md), we also support IO plugins. These allow you to\nregister different file formats as sources to the Polars engines. Because sources can move data zero\ncopy via Arrow FFI and sources can produce large chunks of data before returning, we've decided to\ninterface to IO plugins via Python for now, as we don't think the short time the GIL is needed\nshould lead to any contention.\n\nE.g. an IO source can read their dataframe's in rust and only at the rendez-vous move the data\nzero-copy having only a short time the GIL is needed.\n\n## Use case\n\nYou want IO plugins if you have a source file not supported by Polars and you want to benefit from\noptimizations like projection pushdown, predicate pushdown, early stopping and support of our\nstreaming engine.\n\n## Example\n\nSo let's write a simple, very bad, custom CSV source and register that as an IO plugin. I want to\nstress that this is a very bad example and is only given for learning purposes.\n\nFirst we define some imports we need:\n\n```python\n# Use python for csv parsing.\nimport csv\nimport polars as pl\n# Used to register a new generator on every instantiation.\nfrom polars.io.plugins import register_io_source\nfrom typing import Iterator\nimport io\n```\n\n### Parsing the schema\n\nEvery `scan` function in Polars has to be able to provide the schema of the data it reads. For this\nsimple csv parser we will always read the data as `pl.String`. The only thing that differs are the\nfield names and the number of fields.\n\n```python\ndef parse_schema(csv_str: str) -> pl.Schema:\n    first_line = csv_str.split(\"\\n\")[0]\n\n    return pl.Schema({k: pl.String for k in first_line.split(\",\")})\n```\n\nIf we run this with small csv file `\"a,b,c\\n1,2,3\"` we get the schema:\n`Schema([('a', String), ('b', String), ('c', String)])`.\n\n```python\n>>> print(parse_schema(\"a,b,c\\n1,2,3\"))\nSchema([('a', String), ('b', String), ('c', String)])\n```\n\n### Writing the source\n\nNext up is the actual source. For this we create an outer and an inner function. The outer function\n`my_scan_csv` is the user facing function. This function will accept the file name and other\npotential arguments you would need for reading the source. For csv files, these arguments could be\n\"delimiter\", \"quote_char\" and such.\n\nThis outer function calls `register_io_source` which accepts a `callable` and a `schema`. The schema\nis the Polars schema of the complete source file (independent of projection pushdown).\n\nThe callable is a function that will return a generator that produces `pl.DataFrame` objects.\n\nThe arguments of this function are predefined and this function must accept:\n\n- `with_columns`\n\n  Columns that are projected. The reader must project these columns if applied\n\n- `predicate`\n\n  Polars expression. The reader must filter their rows accordingly.\n\n- `n_rows`\n\n  Materialize only n rows from the source. The reader can stop when `n_rows` are read.\n\n- `batch_size`\n\n  A hint of the ideal batch size the reader's generator must produce.\n\nThe inner function is the actual implementation of the IO source and can also call into Rust/C++ or\nwherever the IO plugin is written. If you want to see an IO source implemented in Rust, take a look\nat our [plugins repository](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin).\n\n```python\ndef my_scan_csv(csv_str: str) -> pl.LazyFrame:\n    schema = parse_schema(csv_str)\n\n    def source_generator(\n        with_columns: list[str] | None,\n        predicate: pl.Expr | None,\n        n_rows: int | None,\n        batch_size: int | None,\n    ) -> Iterator[pl.DataFrame]:\n        \"\"\"\n        Generator function that creates the source.\n        This function will be registered as IO source.\n        \"\"\"\n        if batch_size is None:\n            batch_size = 100\n\n        # Initialize the reader.\n        reader = csv.reader(io.StringIO(csv_str), delimiter=',')\n        # Skip the header.\n        _ = next(reader)\n\n        # Ensure we don't read more rows than requested from the engine\n        while n_rows is None or n_rows > 0:\n            if n_rows is not None:\n                batch_size = min(batch_size, n_rows)\n\n            rows = []\n\n            for _ in range(batch_size):\n                try:\n                    row = next(reader)\n                except StopIteration:\n                    n_rows = 0\n                    break\n                rows.append(row)\n\n            df = pl.from_records(rows, schema=schema, orient=\"row\")\n            n_rows -= df.height\n\n            # If we would make a performant reader, we would not read these\n            # columns at all.\n            if with_columns is not None:\n                df = df.select(with_columns)\n\n            # If the source supports predicate pushdown, the expression can be parsed\n            # to skip rows/groups.\n            if predicate is not None:\n                df = df.filter(predicate)\n\n            yield df\n\n    return register_io_source(io_source=source_generator, schema=schema)\n```\n\n### Taking it for a (very slow) spin\n\nFinally we can test our source:\n\n```python\ncsv_str1 = \"\"\"a,b,c,d\n1,2,3,4\n9,10,11,2\n1,2,3,4\n1,122,3,4\"\"\"\n\nprint(my_scan_csv(csv_str1).collect())\n\n\ncsv_str2 = \"\"\"a,b\n1,2\n9,10\n1,2\n1,122\"\"\"\n\nprint(my_scan_csv(csv_str2).head(2).collect())\n```\n\nRunning the script above would print the following output to the console:\n\n```\nshape: (4, 4)\n┌─────┬─────┬─────┬─────┐\n│ a   ┆ b   ┆ c   ┆ d   │\n│ --- ┆ --- ┆ --- ┆ --- │\n│ str ┆ str ┆ str ┆ str │\n╞═════╪═════╪═════╪═════╡\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 9   ┆ 10  ┆ 11  ┆ 2   │\n│ 1   ┆ 2   ┆ 3   ┆ 4   │\n│ 1   ┆ 122 ┆ 3   ┆ 4   │\n└─────┴─────┴─────┴─────┘\nshape: (2, 2)\n┌─────┬─────┐\n│ a   ┆ b   │\n│ --- ┆ --- │\n│ str ┆ str │\n╞═════╪═════╡\n│ 1   ┆ 2   │\n│ 9   ┆ 10  │\n└─────┴─────┘\n```\n\n## For engine authors: identifying a source\n\nThis section is for people writing a Polars execution engine. If you are writing an IO source, you\ncan skip it.\n\nPolars does not call your source directly. It calls a wrapper that deserializes the predicate and\nforwards the call. Engines may need to identify sources they know about, for example to provide\nexecution context or call additional methods.\n\nTo make that possible, the wrapper satisfies the `IOSourceScanFunction` protocol, which exposes the\nregistered source:\n\n```python\nfrom polars.io.plugins import IOSourceScanFunction\n\n# `scan_fn` is the scan function of the `PythonScan` node being executed.\nif isinstance(scan_fn, IOSourceScanFunction):\n    source = scan_fn.io_source  # exactly the object passed to register_io_source\n    if isinstance(source, MyEngineSource):\n        ...\n```\n\nThis is as unstable as the rest of the IO plugin API.\n\n## Further reading\n\n- [Rust example (distribution source)](https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin)\n</yakaai_skills>"}