SynapseML Documentation

1. Project Overview & Quickstart (microsoft/SynapseML) Copy Chapter

File: README.md Copy Chapter

Synapse Machine Learning

SynapseML (previously known as MMLSpark), is an open-source library that simplifies the creation of massively scalable machine learning (ML) pipelines. SynapseML provides simple, composable, and distributed APIs for a wide variety of different machine learning tasks such as text analytics, vision, anomaly detection, and many others. SynapseML is built on the Apache Spark distributed computing framework and shares the same API as the SparkML/MLLib library, allowing you to seamlessly embed SynapseML models into existing Apache Spark workflows.

With SynapseML, you can build scalable and intelligent systems to solve challenges in domains such as anomaly detection, computer vision, deep learning, text analytics, and others. SynapseML can train and evaluate models on single-node, multi-node, and elastically resizable clusters of computers. This lets you scale your work without wasting resources. SynapseML is usable across Python, R, Scala, Java, and .NET. Furthermore, its API abstracts over a wide variety of databases, file systems, and cloud data stores to simplify experiments no matter where data is located.

SynapseML publishes runtime-specific JVM artifacts: Spark 3.5 uses Scala 2.12,
while Spark 4.0 and 4.1 use Scala 2.13. See the
installation matrix before selecting a Maven
coordinate.

Table of Contents

• Synapse Machine Learning
  • Features
  • Documentation and Examples
  • Setup and installation
    • Microsoft Fabric
    • Synapse Analytics
    • Databricks
    • Python Standalone
    • Spark Submit
    • SBT
    • Apache Livy and HDInsight
    • Docker
    • R
    • Building from source
  • Papers
  • Learn More
  • Contributing & feedback
  • Other relevant projects

Features Copy Chapter

Documentation and Examples Copy Chapter

For quickstarts, documentation, demos, and examples please see our website.

Setup and installation Copy Chapter

SynapseML installation has two parts: the language wrapper and the JVM
artifacts loaded by Spark. Installing synapseml from PyPI does not add the
JVM artifacts. A Python wrapper can import successfully while its JVM class is
missing; using a _2.12 artifact with Spark 4 can produce errors such as
LightGBMClassifier does not exist in the JVM.

Choose one complete published build from the Spark runtime. master is the
canonical Spark 3.5 development line; Spark 4.0 and Spark 4.1 are maintained on
their corresponding branches.

Always configure
https://mmlspark.blob.core.windows.net/maven, where the Spark 4 artifacts are
published. See the full installation guide for platform-specific details.

Latest master snapshot Copy Section

The latest successful master build targets Spark 3.5 and Scala 2.12. This
copy-ready command reads the current snapshot version published by CI and starts
Spark with that exact JVM build:

MASTER_VERSION="$(
  curl -fsSL https://mmlspark.blob.core.windows.net/icons/badges/master_version3.svg |
    sed -n 's/.*aria-label="master version: \([^"]*\)".*/\1/p'
)"
test -n "$MASTER_VERSION"
spark-shell \
  --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.12:${MASTER_VERSION}"

The PyPI package contains released Python wrappers. If you need Python APIs
that are new on master, build the matching wheel from source.

First select the correct platform that you are installing SynapseML into:

• Synapse Machine Learning
  • Features
  • Documentation and Examples
  • Setup and installation
    • Latest master snapshot
    • Microsoft Fabric
    • Synapse Analytics
    • Databricks
    • Python Standalone
    • Spark Submit
    • SBT
    • Apache Livy and HDInsight
    • Docker
    • R
    • Building from source
  • Papers
  • Learn More
  • Contributing & feedback
  • Other relevant projects

Microsoft Fabric Copy Section

In Microsoft Fabric notebooks SynapseML is already installed. The following
copy-ready override targets a Spark 4.1 / Scala 2.13 runtime:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.13,org.scalactic:scalactic_2.13,org.scalatest:scalatest_2.13,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

Synapse Analytics Copy Section

Current Azure Synapse pools use Spark 3.5. Place the following in the first cell
of your notebook:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

To install at the pool level instead of the notebook level add the spark properties listed above to the pool configuration.

Databricks Copy Section

To install SynapseML on the Databricks
cloud, create a new library from Maven
coordinates
in your workspace.

Use one of these exact Maven coordinates:

• Spark 4.1 / Scala 2.13:
  com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1
• Spark 4.0 / Scala 2.13:
  com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0
• Spark 3.5 / Scala 2.12:
  com.microsoft.azure:synapseml_2.12:1.1.3

Add the resolver https://mmlspark.blob.core.windows.net/maven, attach the
library to the target cluster, and restart it before importing synapse.ml.

You can use SynapseML in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:

https://mmlspark.blob.core.windows.net/dbcs/SynapseMLExamplesv1.1.3.dbc

Python Standalone Copy Section

Choose exactly one complete runtime variant below, then start Spark with that
variant's JVM artifact.

Spark 4.1 / Python 3.13

python -m pip install "synapseml==1.1.3" "pyspark>=4.1,<4.2"

Spark 4.0 / Python 3.12

python -m pip install "synapseml==1.1.3" "pyspark>=4.0.1,<4.1"

Spark 3.5 / Python 3.11

python -m pip install "synapseml==1.1.3" "pyspark>=3.5,<3.6"

from pyspark.sql import SparkSession

# Spark 4.1. Select the coordinate matching the PySpark command used above.
synapseml_coordinate = "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"
# Spark 4.0:
# synapseml_coordinate = "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0"
# Spark 3.5:
# synapseml_coordinate = "com.microsoft.azure:synapseml_2.12:1.1.3"

spark = (
    SparkSession.builder.appName("MyApp")
    .config("spark.jars.packages", synapseml_coordinate)
    .config(
        "spark.jars.repositories",
        "https://mmlspark.blob.core.windows.net/maven",
    )
    .getOrCreate()
)
import synapse.ml

Spark Submit Copy Section

SynapseML can be conveniently installed on existing Spark clusters via the
--packages option. Each example below is independently copyable.

# Spark 4.1
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"

# Spark 4.0
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0"

# Spark 3.5
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.12:1.1.3"

SBT Copy Section

Choose the dependency matching your Spark runtime.

Spark 4.1

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "1.1.3-spark4.1"

Spark 4.0

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "1.1.3-spark4.0"

Spark 3.5

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.12" % "1.1.3"

Apache Livy and HDInsight Copy Section

To install SynapseML from within a Jupyter notebook served by Apache Livy, the
following Spark 3.5 / Scala 2.12 configure magic can be used. You will need to
start a new session after this configure cell is executed.

Excluding certain packages from the library may be necessary due to current issues with Livy 0.5.

%%configure -f
{
    "name": "synapseml",
    "conf": {
        "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
        "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
        "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind"
    }
}

Docker Copy Section

The easiest way to evaluate SynapseML is via our pre-built Docker container. To
do so, run the following command:

docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release jupyter notebook

Navigate to http://localhost:8888/ in your web browser to run the sample
notebooks. See the documentation for more on Docker use.

To read the EULA for using the docker image, run docker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release eula

R Copy Section

To try out SynapseML using the R autogenerated wrappers see our
instructions. Note: This feature is still under development
and some necessary custom wrappers may be missing.

Building from source Copy Section

SynapseML has recently transitioned to a new build infrastructure.
For detailed developer docs please see the Developer Readme

If you are an existing synapsemldeveloper, you will need to reconfigure your
development setup. We now support platform independent development and
better integrate with intellij and SBT.
If you encounter issues please reach out to our support email!

Papers Copy Chapter

• Large Scale Intelligent Microservices

• Conditional Image Retrieval

• MMLSpark: Unifying Machine Learning Ecosystems at Massive Scales

• Flexible and Scalable Deep Learning with SynapseML

• Large-Scale Automatic Audiobook Creation

Learn More Copy Chapter

• Visit our website.

• Watch our keynote demos at the Spark+AI Summit 2019, the Spark+AI European Summit 2018, the Spark+AI Summit 2018 and SynapseML at the Spark Summit.

• See how SynapseML is used to help endangered species.

• Explore generative adversarial artwork in our collaboration with The MET and MIT.

• Explore our collaboration with Apache Spark on image analysis.

Contributing & feedback Copy Chapter

This project has adopted the Microsoft Open Source Code of Conduct. For more
information see the Code of Conduct FAQ or contact
[email protected] with any additional
questions or comments.

See CONTRIBUTING.md for contribution guidelines.

To give feedback and/or report an issue, open a GitHub
Issue.

Other relevant projects Copy Chapter

• Vowpal Wabbit

• LightGBM

• DMTK: Microsoft Distributed Machine Learning Toolkit

• Recommenders

• JPMML-SparkML plugin for converting SynapseML LightGBM models to PMML

• Microsoft Cognitive Toolkit

Apache®, Apache Spark, and Spark® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

File: core/src/main/python/synapse/doc/index.rst Copy Chapter

.. MicrosoftMLSpark documentation master file

Microsoft Machine Learning for Apache Spark

Contents:

.. toctree::
:maxdepth: 2

modules
scala

Indices and tables

• :ref:genindex
• :ref:modindex
• :ref:search

File: core/src/main/python/synapse/doc/scala.rst Copy Chapter

Scala API Docs

See the Scala API documentation for SynapseML <../scala/index.html>_.

File: docs/Get Started/Install SynapseML.md Copy Chapter

title: Install SynapseML Copy Chapter

Choose the artifact that matches your Spark runtime Copy Chapter

SynapseML installation has two parts:

1. language wrappers such as the synapseml Python package; and
2. JVM artifacts loaded by Spark.

Installing the Python package does not add the JVM artifacts. A Python
wrapper can import successfully while its JVM class is missing. In particular,
using a _2.12 artifact with Spark 4 can cause errors such as
LightGBMClassifier does not exist in the JVM.

Choose one complete published build from the Spark runtime. master is the
canonical Spark 3.5 development line; Spark 4.0 and Spark 4.1 are maintained on
their corresponding branches.

Always add the SynapseML repository:

https://mmlspark.blob.core.windows.net/maven

Latest master snapshot Copy Chapter

The latest successful master build targets Spark 3.5 and Scala 2.12. This
copy-ready command reads the current snapshot version published by CI and starts
Spark with that exact JVM build:

MASTER_VERSION="$(
  curl -fsSL https://mmlspark.blob.core.windows.net/icons/badges/master_version3.svg |
    sed -n 's/.*aria-label="master version: \([^"]*\)".*/\1/p'
)"
test -n "$MASTER_VERSION"
spark-shell \
  --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.12:${MASTER_VERSION}"

The PyPI package contains released Python wrappers. If you need Python APIs
that are new on master, build the matching wheel from source.

Microsoft Fabric Copy Chapter

SynapseML is already installed in Microsoft Fabric notebooks. The following
copy-ready override targets a Spark 4.1 / Scala 2.13 runtime:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.13,org.scalactic:scalactic_2.13,org.scalatest:scalatest_2.13,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

Synapse Copy Chapter

Current Synapse Analytics pools use Spark 3.5. To override the preinstalled
version, place the following in the first cell of your notebook:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

Python Copy Chapter

To try out SynapseML on a Python (or Conda) installation, you can get Spark
installed via pip. Choose exactly one complete runtime variant below, then
start Spark with that variant's JVM artifact.

Spark 4.1 / Python 3.13

python -m pip install "synapseml==1.1.3" "pyspark>=4.1,<4.2"

Spark 4.0 / Python 3.12

python -m pip install "synapseml==1.1.3" "pyspark>=4.0.1,<4.1"

Spark 3.5 / Python 3.11

python -m pip install "synapseml==1.1.3" "pyspark>=3.5,<3.6"

from pyspark.sql import SparkSession

# Spark 4.1. Select the coordinate matching the PySpark command used above.
synapseml_coordinate = "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"
# Spark 4.0:
# synapseml_coordinate = "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0"
# Spark 3.5:
# synapseml_coordinate = "com.microsoft.azure:synapseml_2.12:1.1.3"

spark = (
    SparkSession.builder.appName("MyApp")
    .config("spark.jars.packages", synapseml_coordinate)
    .config(
        "spark.jars.repositories",
        "https://mmlspark.blob.core.windows.net/maven",
    )
    .getOrCreate()
)
import synapse.ml

SBT Copy Chapter

If you're building a Spark application in Scala, add the following lines to
your build.sbt. Choose the dependency matching your Spark runtime.

Spark 4.1

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "1.1.3-spark4.1"

Spark 4.0

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "1.1.3-spark4.0"

Spark 3.5

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.12" % "1.1.3"

Spark package Copy Chapter

SynapseML can be conveniently installed on existing Spark clusters via the
--packages option. Each example below is independently copyable.

# Spark 4.1
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"

# Spark 4.0
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0"

# Spark 3.5
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.12:1.1.3"

A similar technique can be used in other Spark contexts too. For example, you can use SynapseML
in AZTK by adding it to the
.aztk/spark-defaults.conf
file.

Databricks Copy Chapter

To install SynapseML on the Databricks
cloud, create a new library from Maven
coordinates
in your workspace.

Use one of these exact Maven coordinates:

• Spark 4.1 / Scala 2.13:
  com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1
• Spark 4.0 / Scala 2.13:
  com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0
• Spark 3.5 / Scala 2.12:
  com.microsoft.azure:synapseml_2.12:1.1.3

Add the resolver https://mmlspark.blob.core.windows.net/maven, attach the
library to the target cluster, and restart it before importing synapse.ml.

You can use SynapseML in both your Scala and PySpark notebooks. To get started with our example notebooks, import the following databricks archive:

https://mmlspark.blob.core.windows.net/dbcs/SynapseMLExamplesv1.1.3.dbc

Apache Livy and HDInsight Copy Chapter

To install SynapseML from within a Jupyter notebook served by Apache Livy, the
following Spark 3.5 / Scala 2.12 configure magic can be used. You'll need to
start a new session after this configure cell is executed.

Excluding certain packages from the library may be necessary due to current issues with Livy 0.5

%%configure -f
{
    "name": "synapseml",
    "conf": {
        "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
        "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
        "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind"
    }
}

In Azure Synapse, spark.yarn.user.classpath.first should be set to true to
override the existing SynapseML packages:

%%configure -f
{
    "name": "synapseml",
    "conf": {
        "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
        "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
        "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
        "spark.yarn.user.classpath.first": "true"
    }
}

Docker Copy Chapter

The easiest way to evaluate SynapseML is via our pre-built Docker container. To
do so, run the following command:

docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release

Navigate to http://localhost:8888/ in your web browser to run the sample
notebooks. See the [documentation](../../Reference/Docker Setup.md) for more on Docker use.

To read the EULA for using the docker image, run

docker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release eula

Building from source Copy Chapter

SynapseML has recently transitioned to a new build infrastructure.
For detailed developer docs, see the Developer Readme

If you're an existing SynapseML developer, you'll need to reconfigure your
development setup. We now support platform independent development and
better integrate with intellij and SBT.
If you encounter issues, reach out to our support email!

R (Beta) Copy Chapter

To try out SynapseML using the R autogenerated wrappers, see our
instructions. Note: This feature is still under development
and some necessary custom wrappers may be missing.

File: docs/Explore Algorithms/Vowpal Wabbit/Overview.md Copy Chapter

title: VW hide_title: true sidebar_label: About Copy Chapter

VowpalWabbit on Apache Spark

Overview Copy Section

VowpalWabbit (VW) is a machine learning system that
pushes the frontier of machine learning with techniques such as online, hashing, allreduce,
reductions, learning2search, active, and interactive learning.
VowpalWabbit is a popular choice in ad-tech due to its speed and cost efficacy.
Furthermore it includes many advances in the area of reinforcement learning (for instance, contextual bandits).

Advantages of VowpalWabbit Copy Section

• Composability: VowpalWabbit models can be incorporated into existing
  SparkML Pipelines, and used for batch, streaming, and serving workloads.
• Small footprint: VowpalWabbit memory consumption is rather small and can be controlled through '-b 18' or the setNumBits method.
  This option determines the size of the model (2^18 * some_constant, in this example).
• Feature Interactions: Feature interactions (quadratic, cubic,... terms, for instance) are created on-the-fly within the most inner
  learning loop in VW.
  Interactions can be specified by using the -q parameter and passing the first character of the namespaces that should be interacted.
  The VW namespace concept is mapped to Spark using columns. The column name is used as namespace name, thus one sparse or dense Spark ML vector corresponds to the features of a single namespace.
  To allow passing of multiple namespaces, the VW estimator (classifier or regression) exposes a property called additionalFeatures. Users can pass an array of column names.
• Simple deployment: all native dependencies are packaged into a single jars (including boost and zlib).
• VowpalWabbit command line arguments: users can pass VW command line arguments to control the learning process.
• VowpalWabbit binary models To start the training, users can supply an initial VowpalWabbit model, which can be produced outside of
  VW on Spark, by invoking setInitialModel and passing the model as a byte array. Similarly, users can access the binary model by invoking
  getModel on the trained model object.
• Java-based hashing VW's version of murmur-hash was reimplemented in Java (praise to JackDoe)
  providing a major performance improvement compared to passing input strings through JNI and hashing in C++.
• Cross language VowpalWabbit on Spark is available on Spark, PySpark, and SparklyR.

Limitations of VowpalWabbit on Spark Copy Section

• Linux and CentOS only The native binaries included with the published jar are built Linux and CentOS only.
  We're working on creating a more portable version by statically linking Boost and lib C++.
• Limited Parsing Features implemented in the native VW parser (ngrams, skips, ...) are not yet implemented in
  VowpalWabbitFeaturizer.

Usage Copy Section

In PySpark, you can run the VowpalWabbitClassifier via:

from synapse.ml.vw import VowpalWabbitClassifier
model = (VowpalWabbitClassifier(numPasses=5, args="--holdout_off --loss_function logistic")
            .fit(train))

Similarly, you can run the VowpalWabbitRegressor:

from synapse.ml.vw import VowpalWabbitRegressor
model = (VowpalWabbitRegressor(args="--holdout_off --loss_function quantile -q :: -l 0.1")
            .fit(train))

You can pass command line parameters to VW via the args parameter, as documented in the VW Wiki.

For an end to end application, check out the VowpalWabbit notebook
example.

Hyper-parameter tuning Copy Section

• Common parameters can also be set through methods enabling the use of SparkMLs ParamGridBuilder and CrossValidator (example). If
  the same parameters are passed through the args property (for instance, args="-l 0.2" and setLearningRate(0.5)) the args value will
  take precedence.
  parameter

• learningRate
• numPasses
• numBits
• l1
• l2
• powerT
• interactions
• ignoreNamespaces

Architecture Copy Section

VowpalWabbit on Spark uses an optimized JNI layer to efficiently support Spark.
Java bindings can be found in the VW GitHub repo.

VW's command line tool uses a two-thread architecture (1x parsing/hashing, 1x learning) for learning and inference.
To fluently embed VW into the Spark ML ecosystem, the following adaptions were made:

• VW classifier/regressor operates on Spark's dense/sparse vectors

  • Pro: best composability with existing Spark ML components.
  • Cons: due to type restrictions (for example, feature indices are Java integers), the maximum model size is limited to 30 bits. One could overcome this restriction by adding type support to the classifier/regressor to directly operate on input features (strings, int, double, ...).

• VW hashing is separated out into the VowpalWabbitFeaturizer transformer. It supports mapping Spark Dataframe schema into VW's namespaces and sparse
  features.

  • Pro: featurization can be scaled to many nodes, scale independent of distributed learning.
  • Pro: hashed features can be cached and efficiently reused when performing hyper-parameter sweeps.
  • Pro: featurization can be used for other Spark ML learning algorithms.
  • Cons: due to type restrictions (for instance, sparse indices are Java integers) the hash space is limited to 30 bits.

• VW multi-pass training can be enabled using '--passes 4' argument or setNumPasses method. Cache file is automatically named.

  • Pro: simplified usage.
  • Pro: certain algorithms (for example, l-bfgs) require a cache file when running in multi-pass node.
  • Cons: Since the cache file resides in the Java temp directory, a bottleneck may arise, depending on your node's I/O performance and the location of the temp directory.

• VW distributed training is transparently set up and can be controlled through the input dataframes number of partitions.
  Similar to LightGBM all training instances must be running at the same time, thus the maximum parallelism is restricted by the
  number of executors available in the cluster. Under the hood, VW's built-in spanning tree functionality is used to coordinate allreduce.
  Required parameters are automatically determined and supplied to VW. The spanning tree coordination process is run on the driver node.

  • Pro: seamless parallelization.
  • Cons: currently barrier execution mode isn't implemented and thus if one node crashes the complete job needs to be manually restarted.

File: docs/Explore Algorithms/Responsible AI/Data Balance Analysis.md Copy Chapter

title: Data Balance Analysis on Spark hide_title: true sidebar_label: Data Balance Analysis Copy Chapter

Data Balance Analysis on Spark

Context Copy Chapter

Data Balance Analysis is relevant for gaining an overall understanding of datasets, but it becomes essential when thinking about building AI systems in a responsible way, especially in terms of fairness.

AI systems can sometimes exhibit unwanted, unfair behaviors. These behaviors can cause fairness-related harms that affect various groups of people. They may amplify the marginalization of particular groups whose needs and contexts are often overlooked during AI development and deployment. Fairness-related harms can have varying severities, and the cumulative impact of even seemingly non-severe harms can be burdensome.

Fairness-related harms include:

• Allocation harms: When an AI system extends or withholds opportunities or resources in ways that negatively impact people’s lives.
• Quality of service harms: When an AI system does not work as well for one group of people as it does for another.
• Stereotyping harms: When an AI system makes unfair generalizations about groups of people and reinforces negative stereotypes.
• Demeaning harms: When an AI system is actively derogatory or offensive.
• Over/underrepresentation harms: When an AI system over/underrepresents some groups of people or may even erase some groups entirely.

Note: Because fairness in AI is fundamentally a sociotechnical challenge, it's often impossible to fully “de-bias” an AI system. Instead, teams tasked with developing and deploying AI systems must work to identify, measure, and mitigate fairness-related harms as much as possible. Data Balance Analysis is a tool to help do so, in combination with others.

Data Balance Analysis consists of a combination of three groups of measures: Feature Balance Measures, Distribution Balance Measures, and Aggregate Balance Measures.

In summary, Data Balance Analysis, when used as a step for building ML models, has the following benefits:

• It reduces the costs of building ML through the early identification of data representation gaps. Before proceeding to train their models, data scientists can seek mitigation steps such as collecting more data, following a specific sampling mechanism, creating synthetic data, and so on.
• It enables easy end-to-end debugging of ML systems in combination with the RAI Toolbox by providing a clear view of model-related issues versus data-related issues.

Examples Copy Chapter

• Quickstart - Data Balance Analysis

Usage Copy Chapter

Data Balance Analysis currently supports three transformers in the synapse.ml.exploratory namespace:

• FeatureBalanceMeasure - supervised (requires label column)
• DistributionBalanceMeasure - unsupervised (doesn't require label column)
• AggregateBalanceMeasure - unsupervised (doesn't require label column)

1. Import all three transformers.

   For example:

   python Copy

   from synapse.ml.exploratory import AggregateBalanceMeasure, DistributionBalanceMeasure, FeatureBalanceMeasure

2. Load your dataset, define features of interest, and ensure that the label column is binary. The FeatureBalanceMeasure transformer currently only supports binary labels, but support for numerical labels will be added soon.

   For example:

   python Copy

   import pyspark.sql.functions as F
   
   features = ["race", "sex"]
   label = "income"
   
   df = spark.read.parquet("wasbs://[email protected]/AdultCensusIncome.parquet")
   
   # Convert the "income" column from {<=50K, >50K} to {0, 1} to represent our binary classification label column
   df = df.withColumn(label, F.when(F.col(label).contains("<=50K"), F.lit(0)).otherwise(F.lit(1)))

3. Create a FeatureBalanceMeasure transformer and call setSensitiveCols to set the list of sensitive features and call setLabelCol to set the binary label column. Then, call the transform method with your dataset and visualize the resulting dataframe.

   For example:

   python Copy

   feature_balance_measures = (
       FeatureBalanceMeasure()
       .setSensitiveCols(features)
       .setLabelCol(label)
       .transform(df)
   )
   feature_balance_measures.show(truncate=False)

4. Create a DistributionBalanceMeasure transformer and call setSensitiveCols to set the list of sensitive features. Then, call the transform method with your dataset and visualize the resulting dataframe.

   For example:

   python Copy

   distribution_balance_measures = (
       DistributionBalanceMeasure()
       .setSensitiveCols(features)
       .transform(df)
   )
   distribution_balance_measures.show(truncate=False)

5. Create a AggregateBalanceMeasure transformer and call setSensitiveCols to set the list of sensitive features. Then, call the transform method with your dataset and visualize the resulting dataframe.

   For example:

   python Copy

   aggregate_balance_measures = (
       AggregateBalanceMeasure()
       .setSensitiveCols(features)
       .transform(df)
   )
   aggregate_balance_measures.show(truncate=False)

Note: If you're running this notebook in a Spark environment such as Azure Synapse or Databricks, then you can easily visualize the imbalance measures by calling the built-in plotting features display().

Measure Explanations Copy Chapter

Feature Balance Measures Copy Section

Feature Balance Measures allow us to see whether each combination of sensitive feature is receiving the positive outcome (true prediction) at balanced probability.

In this context, we define a feature balance measure, called the parity, for label y. It is the difference between the association metrics of two different sensitive classes $[x_A, x_B]$, with respect to the association metric $A(x_i, y)$. That is:

$parity(y \vert x_A, x_B, A(\cdot)) \coloneqq A(x_A, y) - A(x_B, y)$

Using the dataset, we can see if the various sexes and races are receiving >50k income at equal or unequal rates.

Note: Many of these metrics were influenced by this paper Measuring Model Biases in the Absence of Ground Truth.

Distribution Balance Measures Copy Section

Distribution Balance Measures allow us to compare our data with a reference distribution. They use a uniform reference distribution by default; an empty map also selects the uniform distribution. A custom distribution can be supplied with setReferenceDistribution, and its probabilities must be finite, non-negative, and sum to 1. The measures use the union of the observed and positive-probability reference categories: an observed-only category has reference probability 0, and a reference-only category has observed probability and count 0. Null observed values are treated as categories, but custom reference keys cannot be null. For integral sensitive columns, reference keys are strings parsed as the column's integral type and must resolve to distinct values. They are calculated per sensitive column and don't depend on the label column. Empty input datasets are rejected because they do not define an observed probability distribution.

For example, let's assume we have a dataset with nine rows and a Gender column, and we observe that:

• "Male" appears four times
• "Female" appears three times
• "Other" appears twice

Assuming the uniform distribution:

$$
ReferenceCount \coloneqq \frac{numRows}{numFeatureValues}
$$

$$
ReferenceProbability \coloneqq \frac{1}{numFeatureValues}
$$

We can use distance measures to find out how far our observed and reference distributions of these feature values are. Some of these distance measures include:

Aggregate Balance Measures Copy Section

Aggregate Balance Measures allow us to obtain a higher notion of inequality. They're calculated on the set of all sensitive columns and don't depend on the label column.

These measures look at distribution of records across all combinations of sensitive columns. For example, if Sex and Race are specified as sensitive features, it then tries to quantify imbalance across all combinations of the two specified features - (Male, Black), (Female, White), (Male, Asian-Pac-Islander), etc.

Mitigation Copy Chapter

It will not be a stretch to say that every real-world dataset has caveats, biases, and imbalances. Data collection is costly. Data Imbalance mitigation or de-biasing data is an area of research. There are many techniques available at various stages of ML lifecycle: during pre-processing, in-processing, and post processing. Here we outline a couple of pre-processing techniques -

Resampling Copy Section

Resampling involves under-sampling from majority class and over-sampling from minority class. A naïve way to over-sample would be to duplicate records. Similarly, to under-sample one could remove records at random.

• Caveats:

  1. Under-sampling may remove valuable information.
  2. Over-sampling may cause overfitting and poor generalization on test set.

There are smarter techniques to under-sample and over-sample in literature and implemented in Python’s imbalanced-learn package.

For example, we can cluster the records of the majority class, and do the under-sampling by removing records from each cluster, thus seeking to preserve information.

One technique of under-sampling is use of Tomek Links. Tomek links are pairs of instances that are very close but of opposite classes. Removing the instances of the majority class of each pair increases the space between the two classes, facilitating the classification process. A similar way to under-sample majority class is using Near-Miss. It first calculates the distance between all the points in the larger class with the points in the smaller class. When two points belonging to different classes are very close to each other in the distribution, this algorithm eliminates the datapoint of the larger class thereby trying to balance the distribution.

In over-sampling, instead of creating exact copies of the minority class records, we can introduce small variations into those copies, creating more diverse synthetic samples. This technique is called SMOTE (Synthetic Minority Oversampling Technique). It randomly picks a point from the minority class and computes the k-nearest neighbors for this point. The synthetic points are added between the chosen point and its neighbors.

Reweighting Copy Section

There is an expected and observed value in each table cell. The weight is the value of expected / observed. Reweighting is easy to extend to multiple features with more than two groups. The weights are then incorporated in loss function of model training.

File: docs/Explore Algorithms/Responsible AI/Interpreting Model Predictions.md Copy Chapter

title: Interpreting Model Predictions hide_title: true sidebar_label: Interpreting Model Predictions Copy Chapter

Model Interpretation on Spark

Interpretable Machine Learning Copy Chapter

Interpretable Machine Learning helps developers, data scientists and business stakeholders in the organization gain a comprehensive understanding of their machine learning models. It can also be used to debug models, explain predictions and enable auditing to meet compliance with regulatory requirements.

Why run model interpretation on Spark Copy Chapter

Model-agnostic interpretation methods can be computationally expensive due to the multiple evaluations needed to compute the explanations. Model interpretation on Spark enables users to interpret a black-box model at massive scales with the Apache Spark™ distributed computing ecosystem. Various components support local interpretation for tabular, vector, image and text classification models, with two popular model-agnostic interpretation methods: LIME and Kernel SHAP.

Usage Copy Chapter

Both LIME and Kernel SHAP are local interpretation methods. Local interpretation explains why does the model predict certain outcome for a given observation.

Both explainers extends from org.apache.spark.ml.Transformer. After setting up the explainer parameters, simply call the transform function on a DataFrame of observations to interpret the model behavior on these observations.

To see examples of model interpretability on Spark in action, take a look at these sample notebooks:

• Tabular Explainers
• Image Explainers
• Text Explainers

Common local explainer params Copy Section

All local explainers support the following params:

Common LIME explainer params Copy Section

All LIME based explainers (TabularLIME, VectorLIME, ImageLIME, TextLIME) support the following params:

Common SHAP explainer params Copy Section

All Kernel SHAP based explainers (TabularSHAP, VectorSHAP, ImageSHAP, TextSHAP) support the following params:

Tabular model explainer params Copy Section

All tabular model explainers (TabularLIME, TabularSHAP) support the following params:

Vector model explainer params Copy Section

All vector model explainers (VectorLIME, VectorSHAP) support the following params:

Image model explainer params Copy Section

All image model explainers (ImageLIME, ImageSHAP) support the following params:

Text model explainer params Copy Section

All text model explainers (TextLIME, TextSHAP) support the following params:

`TabularLIME` Copy Section

For categorical features, TabularLIME creates new samples by drawing samples based on the value distribution from the background dataset. For numerical features, it creates new samples by drawing from a normal distribution with mean taken from the target value to be explained, and standard deviation taken from the background dataset.

`TabularSHAP` Copy Section

No additional params are supported.

`VectorLIME` Copy Section

No additional params are supported.

VectorLIME assumes all features are numerical, and categorical features are not supported in VectorLIME.

`VectorSHAP` Copy Section

No additional params are supported.

`ImageLIME` Copy Section

ImageLIME creates new samples by randomly turning super-pixels on or off with probability of keeping on set to SamplingFraction.

`ImageSHAP` Copy Section

No additional params are supported.

`TextLIME` Copy Section

TextLIME creates new samples by randomly turning word tokens on or off with probability of keeping on set to SamplingFraction.

`TextSHAP` Copy Section

No additional params are supported.

Result interpretation Copy Chapter

LIME explainers Copy Section

LIME explainers return an array of vectors, and each vector maps to a class being explained. Each component of the vector is the coefficient for the corresponding feature, super-pixel, or word token from the local surrogate model.

• For categorical variables, super-pixels, or word tokens, the coefficient shows the average change in model outcome if this feature is unknown to the model, if the super-pixel is replaced with background color (black), or if the word token is replaced with empty string.
• For numeric variables, the coefficient shows the change in model outcome if the feature value is incremented by 1 unit.

SHAP explainers Copy Section

SHAP explainers return an array of vectors, and each vector maps to a class being explained. Each vector starts with the base value, and each following component of the vector is the Shapley value for each feature, super-pixel, or token.

The base value and Shapley values are additive, and they should add up to the model output for the target observation.

Base value

• For tabular and vector models, the base value represents the mean outcome of the model for the background dataset.
• For image models, the base value represents the model outcome for a background (all black) image.
• For text models, the base value represents the model outcome for an empty string.

File: docs/Explore Algorithms/Other Algorithms/Cyber ML.md Copy Chapter

title: CyberML hide_title: true sidebar_label: CyberML Copy Chapter

CyberML

access anomalies: [complement_access.py](https://github.com/microsoft/SynapseML/blob/master/core/src/main/python/synapse/ml/cyber/anomaly/complement_access.py) Copy Chapter

• Talk at European Spark Conference 2019
• (Internal Microsoft) Talk at MLADS November 2018
• (Internal Microsoft) Talk at MLADS June 2019

1. ComplementAccessTransformer
   is a SparkML Transformer.
   Given a dataframe, it returns a new dataframe comprised of access patterns sampled from
   the set of possible access patterns not present in the original dataframe.
   In other words, it returns a sample from the complement set.

feature engineering: [indexers.py](https://github.com/microsoft/SynapseML/blob/master/core/src/main/python/synapse/ml/cyber/feature/indexers.py) Copy Chapter

1. IdIndexer
   is a SparkML Estimator.
   Given a dataframe, it creates an IdIndexerModel (described next) for categorical features. The model
   maps each partition and column seen in the given dataframe to an ID,
   for each partition or one consecutive range for all partition and column values.
2. IdIndexerModel
   is a SparkML Transformer.
   Given a dataframe maps each partition and column field to a consecutive integer ID.
   Partitions or column values not encountered in the estimator are mapped to 0.
   The model can operate in two modes, either create consecutive integer ID independently
3. MultiIndexer
   is a SparkML Estimator.
   Uses multiple IdIndexers to generate a MultiIndexerModel (described next) for categorical features. The model
   contains multiple IdIndexers for multiple partitions and columns.
4. MultiIndexerModel
   is a SparkML Transformer.
   Given a dataframe maps each partition and column field to a consecutive integer ID.
   Partitions or column values not encountered in the estimator are mapped to 0.
   The model can operate in two modes, either create consecutive integer ID independently

feature engineering: [scalers.py](https://github.com/microsoft/SynapseML/blob/master/core/src/main/python/synapse/ml/cyber/feature/scalers.py) Copy Chapter

1. StandardScalarScaler
   is a SparkML Estimator.
   Given a dataframe it creates a StandardScalarScalerModel (described next) which normalizes
   any given dataframe according to the mean and standard deviation calculated on the
   dataframe given to the estimator.
2. StandardScalarScalerModel
   is a SparkML Transformer.
   Given a dataframe with a value column x, the transformer changes its value as follows:
   x'=(x-mean)/stddev. That is, if the transformer is given the same dataframe the estimator
   was given then the value column will have a mean of 0.0 and a standard deviation of 1.0.
3. LinearScalarScaler
   is a SparkML Estimator.
   Given a dataframe it creates a LinearScalarScalerModel (described next) which normalizes
   any given dataframe according to the minimum and maximum values calculated on the
   dataframe given to the estimator.
4. LinearScalarScalerModel
   is a SparkML Transformer.
   Given a dataframe with a value column x, the transformer changes its value such that
   if the transformer is given the same dataframe the estimator
   was given then the value column will be scaled linearly to the given ranges.

access anomalies: [collaborative_filtering.py](https://github.com/microsoft/SynapseML/blob/master/core/src/main/python/synapse/ml/cyber/anomaly/collaborative_filtering.py) Copy Chapter

1. AccessAnomaly
   is a SparkML Estimator.
   Given a dataframe, the estimator generates an AccessAnomalyModel (described next). The model
   can detect anomalous access of users to resources where the access
   is outside of the user's or resources's profile. For instance, a user from HR accessing
   a resource from Finance. This result is based solely on access patterns rather than explicit features.
   Internally, the code is based on Collaborative Filtering as implemented in Spark, using
   Matrix Factorization with Alternating Least Squares.
2. AccessAnomalyModel
   is a SparkML Transformer.
   Given a dataframe the transformer computes a value between (-inf, inf) where positive
   values indicate an anomaly score. Anomaly scores are computed to have a mean of 1.0
   and a standard deviation of 1.0 over the original dataframe given to the estimator.
3. ModelNormalizeTransformer
   is a SparkML Transformer.
   This transformer is used internally by AccessAnomaly to normalize a model to generate
   anomaly scores with mean 0.0 and standard deviation of 1.0.
4. AccessAnomalyConfig
   contains the default values for AccessAnomaly.

File: docs/Explore Algorithms/Other Algorithms/Smart Adaptive Recommendations.md Copy Chapter

title: Smart Adaptive Recommendations (SAR) Algorithm hide_title: true sidebar_label: SAR Algorithm Copy Chapter

Smart Adaptive Recommendations (SAR) Algorithm

The following document is a subset of the implemented logic. The original can be found here

SAR is a fast scalable adaptive algorithm for personalized recommendations based on user transactions history and items description. It produces easily explainable / interpretable recommendations.

The overall architecture of SAR is shown in the following diagram:

Input Copy Chapter

The input to SAR consists of:

• transaction (usage) data
• catalog data

Transaction data, also called usage data, contains information on interactions between users and items and has the following schema:

,,<Time>

Each row represents a single interaction between a user and an item, which we call a transaction. Here's an example of usage data (displayed as a table for readability, but note that it must be provided in comma-separated format):

Note that SAR doesn't require explicit user ratings, which are often noisy and unreliable. It also focuses on implicit events (transactions), which are more telling of user intent.

Collaborative Filtering Copy Chapter

SAR is based on a simple yet powerful collaborative filtering approach. In the heart of the algorithm is computation of two matrices:

• Item-to-Item similarity matrix
• User-to-Item affinity matrix

Item-to-Item similarity matrix contains for each pair of items a numerical value of similarity between these two items. A simple measure of item similarity is co-occurrence, which is the number of times two items appeared in a same transaction. Let's look at the following example:

Here, for example, cooccur(Item 1, Item 2) = 3, meaning that items 1 and 2 were together in a transaction three times. The Item-to-Item similarity matrix is symmetric. Diagonal elements, occ(Item i), represent the number of occurrences of each item.
The advantage of co-occurrence is that it's easy to update. However, it favors predictability, and the most popular items will be recommended most of the time. To alleviate that, two more similarity measures are used: lift and Jaccard. They can be thought of as normalized co-occurrences.

Lift measures how much the co-occurrence of two items is higher than it would be by chance, that is, what is the contribution of interaction of the two items. It is obtained as

lift(Item i, Item j) = cooccur(Item i, Item j) / (occ(Item i) * occ(Item j)) .

Lift favors serendipity / discoverability. For example, items 2 and 5 have the same co-occurrence with item 4, but item 5 in general occurs less frequently than item 2 and will be favored by lift.

Jaccard measure is defined as the number of transactions in which two items appear together divided by the number of transactions in which either of them appears:

Jaccard(Item 1, Item 2) = cooccur(Item1, Item 2) / (occ(Item 1) + occ(Item 2) - cooccur(Item 1, Item 2)) .

Jaccard measure is a tradeoff between co-occurrence and lift and is the default in SAR.

An item is cold if it has no transactions yet or the number of transactions is low, that is, below the SupportThreshold, which is configurable. If one or both items is cold, their item-to-item similarity cannot be estimated from the transactions data and item features must be used. A linear learner is trained using warm items, where the features of the model are (partial) matches on corresponding features of a pair of items. The target is the computed similarity, based on normalized co-occurrences of those items.
The model is then used to predict similarities between cold and cold/warm items.

User-to-Item affinity matrix contains for each user-item pair an affinity score of the user towards the item. Affinity score is computed as a weighted number of transactions in which the user and the item appear together, where newer transactions are weighted more than the older transactions. Also, weights are adjusted for the event type. For example, "Purchase" event may be valued four times more than the "Click" event. Therefore, affinity score takes into account frequency, recency and the event type of transactions in which the user appears with the item.

An example User-to-Item affinity matrix is shown in the following table:

Here, we can see that User 1 has the highest affinity to Item 1, followed by items 2 and 3. This user didn't have a transaction that included items 4 and 5. On the other hand, User 2 has the highest affinity to Item 3, followed by Item 2 and then items 1 and 4.

Making Recommendations Copy Chapter

SAR can produce two types of recommendations:

• User recommendations, which recommend items to individual users based on their transaction history, and

User Recommendations Copy Section

Personalized recommendations for a single user are obtained by multiplying the Item-to-Item similarity matrix with a user affinity vector. The user affinity vector is simply a transposed row of the affinity matrix corresponding to that user. For example, User 1 affinity vector is

By pre-multiplying this vector with the Item-to-Item similarity matrix, User 1 recommendation scores are obtained:

In this case, the recommendation score of an item is purely based on its similarity to Item 5. Assuming that a same item isn't
recommended again, items 1 and 4 have the highest score and would be recommended before items 2 and 3.

Now, if this user adds Item 2 to the shopping cart, affinity vector (assuming weight 2 for this transaction) will be

resulting in recommendation scores:

The recommendation score of an item is purely based on its similarity to Item 5 in this case.
Assuming that a same item isn't recommended again, items 1 and 4 have the highest score and would be recommended before items 2 and 3.
Now, if this user adds Item 2 to the shopping cart, affinity vector (assuming weight 2 for this transaction) will be

resulting in recommendation scores:

Now, Item 1 has the highest score, followed by Item 3 and then Item 4. Note that the advantage of SAR is that the updated scores are obtained without any retraining, that is, without having to recompute Item-to-Item similarities.

Interpretability

SAR provides explainable recommendations. For example, let's look at the reasoning behind the recommendation score for User 1 and Item 4:

rec(User 1, Item 4)
*= sim(Item 4, Item 1) * aff(User 1, Item 1)*
+ sim(Item 4, Item 2) * aff(User 1, Item 2)
+ sim(Item 4, Item 3) * aff(User 1, Item 3)
+ sim(Item 4, Item 4) * aff(User 1, Item 4)
+ sim(Item 4, Item 5) * aff(User 1, Item 5)
= 3 * 5 + 2 * 3 + 3 * 2.5 + 4 * 0 + 2 * 0
*= 15 + 6 + 7.5 + 0 + 0 = 28.5*

Clearly, the first term (highlighted) has the highest contribution to the score. We can say that "The algorithm recommends Item 4 to User 1 because it's similar to Item 1, to which User 1 has high affinity". A message like this can be displayed automatically for each recommendation.

File: docs/Explore Algorithms/LightGBM/Overview.md Copy Chapter

title: Overview hide_title: true sidebar_label: Overview Copy Chapter

LightGBM on Apache Spark

LightGBM Copy Section

LightGBM is an open-source,
distributed, high-performance gradient boosting (GBDT, GBRT, GBM, or
MART) framework. This framework specializes in creating high-quality and
GPU enabled decision tree algorithms for ranking, classification, and
many other machine learning tasks. LightGBM is part of Microsoft's
DMTK project.

Advantages of LightGBM through SynapseML Copy Section

• Composability: LightGBM models can be incorporated into existing
  SparkML Pipelines, and used for batch, streaming, and serving
  workloads.
• Performance: LightGBM on Spark is 10-30% faster than SparkML on
  the Higgs dataset, and achieves a 15% increase in AUC. Parallel
  experiments
  have verified that LightGBM can achieve a linear speed-up by using
  multiple machines for training in specific settings.
• Functionality: LightGBM offers a wide array of tunable
  parameters,
  that one can use to customize their decision tree system. LightGBM on
  Spark also supports new types of problems such as quantile regression.
• Cross platform LightGBM on Spark is available on Spark, PySpark, and SparklyR

Usage Copy Section

In PySpark, you can run the LightGBMClassifier via:

from synapse.ml.lightgbm import LightGBMClassifier
model = LightGBMClassifier(learningRate=0.3,
                           numIterations=100,
                           numLeaves=31).fit(train)

Similarly, you can run the LightGBMRegressor by setting the
application and alpha parameters:

from synapse.ml.lightgbm import LightGBMRegressor
model = LightGBMRegressor(application='quantile',
                          alpha=0.3,
                          learningRate=0.3,
                          numIterations=100,
                          numLeaves=31).fit(train)

For an end to end application, check out the LightGBM notebook
example.

Arguments/Parameters Copy Section

SynapseML exposes getters/setters for many common LightGBM parameters.
In python, you can use property-value pairs, or in Scala use
fluent setters. Examples of both are shown in this section.

import com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier
val classifier = new LightGBMClassifier()
                       .setLearningRate(0.2)
                       .setNumLeaves(50)

LightGBM has far more parameters than SynapseML exposes. For cases where you
need to set some parameters that SynapseML doesn't expose a setter for, use
passThroughArgs. This argument is just a free string that you can use to add extra parameters
to the command SynapseML sends to configure LightGBM.

In python:

from synapse.ml.lightgbm import LightGBMClassifier
model = LightGBMClassifier(passThroughArgs="force_row_wise=true min_sum_hessian_in_leaf=2e-3",
                           numIterations=100,
                           numLeaves=31).fit(train)

In Scala:

import com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier
val classifier = new LightGBMClassifier()
                      .setPassThroughArgs("force_row_wise=true min_sum_hessian_in_leaf=2e-3")
                      .setLearningRate(0.2)
                      .setNumLeaves(50)

For formatting options and specific argument documentation, see
LightGBM docs. SynapseML sets some
parameters specifically for the Spark distributed environment and
shouldn't be changed. Some parameters are for CLI mode only, and don't work within
Spark.

You can mix passThroughArgs and explicit args, as shown in the example. SynapseML
merges them to create one argument string to send to LightGBM. If you set a parameter in
both places, passThroughArgs takes precedence.

Reproducible training

deterministic=True applies to LightGBM after Spark has produced the training rows. For
CPU training, LightGBM also recommends selecting one histogram strategy explicitly:

model = LightGBMClassifier(
    deterministic=True,
    seed=777,
    passThroughArgs="force_col_wise=true",
).fit(train)

This does not make a nondeterministic Spark query deterministic. Operations such as
orderBy(rand()) without a seed can be evaluated again for each action, so a write and a
later fit() may consume different row order or even a different sample. Materialize one
training snapshot before comparing fits: write and reload it, or persist it and complete a
materializing action before training. A Parquet reload can therefore expose an input-lineage
difference without changing vector values, feature metadata, labels, weights, or initial
scores. Compare the exact rows and metadata before interpreting a higher training AUC as a
serialization defect; stable validation AUC can instead indicate training-set overfit.

When deterministic=True, SynapseML logs a warning if Spark marks the training query as
nondeterministic. For effective CPU training, it also warns if neither force_col_wise=true
nor force_row_wise=true is enabled.

GPU training with a custom OpenCL native library

SynapseML's published lightgbmlib artifact contains CPU-only native libraries. Only
deviceType="gpu" selects an accelerator: it selects LightGBM's OpenCL learner and
requires a compatible custom native library. All cuda requests are rejected before
native training because LightGBM 3.3.510 CUDA is incompatible with SynapseML streaming
Datasets.

Accelerator training is intended for users who provide their own compatible LightGBM
native build. Put both lib_lightgbm and lib_lightgbm_swig on java.library.path for
the Spark driver and every executor before LightGBM is initialized. NativeLoader checks
that path first and falls back to the CPU-only libraries packaged in the SynapseML JAR.
For example, a Spark deployment can set both spark.driver.extraLibraryPath and
spark.executor.extraLibraryPath to the directory containing the custom libraries.
The custom SWIG library must be ABI-compatible with the Java classes shipped by the
SynapseML version in use; supplying only one library can accidentally mix incompatible
custom and bundled binaries.

SynapseML does not support deviceType="cuda" with lightgbmlib 3.3.510. Its CUDA
objective expects CUDA metadata that is not created by the serialized streaming Dataset
path and can segfault the Spark executor during booster creation. SynapseML rejects CUDA
before native training. Use deviceType="gpu" with an OpenCL-enabled native build; this
path supports classifier, regressor, and ranker training on NVIDIA GPUs such as T4.

deviceType exposes only the accelerator backends implemented by LightGBM:

Apple Metal Performance Shaders (mps) and Habana HPU are not LightGBM tree-learning
backends and are therefore not accepted values. Apple Silicon can only be evaluated
through LightGBM's OpenCL learner and a custom macOS ARM64 native build; this is not MPS
support, Apple has deprecated OpenCL, and LightGBM documents a macOS Boost.Compute cache
workaround. Do not claim Apple Silicon GPU support without testing that exact native build,
Spark/JVM architecture, dataset correctness, and performance on supported macOS hardware.

After installing the custom native libraries, select the learner explicitly:

model = LightGBMClassifier(deviceType="gpu").fit(train)

The default is cpu and does not add a device_type native parameter. If
passThroughArgs contains device_type, that canonical value takes precedence over both
the device alias and deviceType, regardless of argument order. If only device is
present, it takes precedence over deviceType. A native-effective value of cuda is
always rejected. If a requested OpenCL accelerator is unavailable, SynapseML reports that
the bundled native is CPU-only and identifies the custom-library configuration required.

Architecture Copy Section

LightGBM on Spark uses the Simple Wrapper and Interface Generator (SWIG)
to add Java support for LightGBM. These Java Binding use the Java Native
Interface call into the distributed C++
API.

We initialize LightGBM by calling
LGBM_NetworkInit
with the Spark executors within a MapPartitions call. We then pass each
workers partitions into LightGBM to create the in-memory distributed
dataset for LightGBM. We can then train LightGBM to produce a model
that can then be used for inference.

The LightGBMClassifier and LightGBMRegressor use the SparkML API,
inherit from the same base classes, integrate with SparkML pipelines,
and can be tuned with SparkML's cross
validators.

Models built can be saved as SparkML pipeline with native LightGBM model
using saveNativeModel(). Additionally, they're fully compatible with PMML and
can be converted to PMML format through the
JPMML-SparkML-LightGBM plugin.

Dynamic Allocation Limitations

The native LightGBM library has a distributed mode that allows the algorithm to work over multiple machines. SynapseML
uses this mode to call LightGBM from Spark. SynapseML first gathers all the Spark executor networking information, passes that to LightGBM, and then
waits for LightGBM to complete its work. However, the native LightGBM algorithm implementation assumes all networking is constant over the time period of a single
training or scoring session. The native LightGBM distributed mode was designed this way and isn't a limitation of SynapseML by itself.

Dynamic compute changes can cause LightGBM problems if the Spark executors change during data processing. Spark can naturally
take advantage of cluster autoscaling and can also dynamically replace any failed executor with another, but LightGBM can't
handle these networking changes. Large datasets are affected in particular since they're more likely to cause executor scaling
or have a single executor fail during a single processing pass.

If you're experiencing problems with LightGBM as exposed through SynapseML due to executor changes (for example, occasional Task failures or networking hangs),
there are several options.

1. In the Spark platform, turn off any autoscaling on the cluster you have provisioned.
2. Set numTasks manually to be smaller so that fewer executors are used (reducing probability of single executor failure).
3. Turn off dynamic executor scaling with configuration in a notebook cell. In Synapse and Fabric, you can use:

%%configure
   {
       "conf":
       {
           "spark.dynamicAllocation.enabled": "false"
       }
   }

Note: setting any custom configuration can affect cluster startup time if your compute platform takes advantage of "live pools"
to improve notebook performance.

If you still have problems, you can consider splitting your data into smaller segments using numBatches. Splitting into multiple
batches increases total processing time, but can potentially be used to increase reliability.

Data Transfer Mode Copy Section

SynapseML must pass data from Spark partitions to LightGBM native Datasets before turning over control to
the actual LightGBM execution code for training and inference. SynapseML has two modes
that control how this data is transferred: streaming and bulk.
This mode doesn't affect training but can affect memory usage and overall fit/transform time.
By default, SynapseML uses "streaming" mode.

Bulk Execution mode

The "Bulk" mode is older and requires accumulating all data in executor memory before creating Datasets. This mode can cause
OOM errors for large data, especially since the data must be accumulated in its original uncompressed double-format size.

For bulk mode, native LightGBM Datasets can either be created per partition (useSingleDatasetMode=false), or
per executor (useSingleDatasetMode=true). Generally, one Dataset per executor is more efficient since it reduces LightGBM network size and complexity during training or fitting. It also avoids using slow network protocols on partitions
that are actually on the same executor node.

Streaming Execution Mode

The "streaming" execution mode uses new native LightGBM APIs created just for SynapseML that don't require loading extra copies of the data into memory. In particular, data is passed directly
from partitions to Datasets in small "micro-batches", similar to Spark streaming. The microBatchSize parameter controls the size of these micro-batches.
Smaller micro-batch sizes reduce memory overhead, but larger sizes avoid overhead from repeatedly transferring data to the native layer. The default
100, uses far less memory than bulk mode since only 100 rows of data will be loaded at a time. If your dataset has
few columns, you can increase the batch size. Alternatively, if
your dataset has a large number of columns you can decrease the micro-batch size to avoid OOM issues.

These new streaming APIs in LightGBM are thread-safe, and allow all partitions in the same executor
to push data into a shared Dataset in parallel. Because of this, streaming mode always uses the more efficient
"useSingleDatasetMode=true", creating only one Dataset per executor.

You can explicitly specify Execution Mode and MicroBatch size as parameters.

val lgbm = new LightGBMClassifier()
    .setExecutionMode("streaming")
    .setMicroBatchSize(100)
    .setLabelCol(labelColumn)
    .setObjective("binary")
...

For streaming mode, only one Dataset is created per partition, so useSingleDataMode has no effect. It's effectively always true.

Data Sampling Copy Section

In order for LightGBM algorithm to work, it must first create a set of bin boundaries for optimization. It does this calculation by
first sampling the data before any training or inferencing starts. (LightGBM docs). The number of
samples to use is set using binSampleCount, which must be a minimal percent of the data or LightGBM rejects it.

For bulk mode, this sampling is automatically done over the entire data, and each executor uses its own partitions to calculate samples for only
a subset of the features. This distributed sampling can have subtle effects since partitioning can affect the calculated bins.
Also, all data is sampled no matter what.

For streaming mode, there are more explicit user controls for this sampling, and it's all done from the driver.
The samplingMode property controls the behavior. The efficiency of these methods increases from first to last.

• global - Like bulk mode, the random sample is calculated by iterating over entire data (hence data is traversed twice)
• subset - (default) Samples only from the first samplingSubsetSize elements. Assumes this subset is representative.
• fixed - There's no random sample. The first binSampleSize rows are used. Assumes randomized data.
  For large row counts, subset and fixed modes can save a first iteration over the entire data.

Reference Dataset

The sampling of the data to calculate bin boundaries happens every fit call.
If repeating a fit many times (for example, hyperparameter tuning), this calculation is duplicated effort.

For streaming mode, there's an optimization that a client can set to use the previously calculated bin boundaries. The
sampling calculation results in a reference dataset, which can be reused. After a fit, there will be a referenceDataset property
on the estimator that was calculated and used for that fit. If that is set on the next estimator (or you reuse the same one),
it will use that instead of resampling the data.

from synapse.ml.lightgbm import LightGBMClassifier
classifier = LightGBMClassifier(learningRate=0.3,
                                numIterations=100,
                                numLeaves=31)
model1 = classifier.fit(train)

classifier.learningRate = 0.4
model2 = classifier.fit(train)

The 'model2' call to 'fit' doesn't resample the data and uses the same bin boundaries as 'model1'.

Caution: Some parameters actually affect the bin boundary calculation and require the use of a new reference dataset every time.
These parameters include isEnableSparse, useMissing, and zeroAsMissing that you can set from SynapseML. If you manually set
some parameters with passThroughArgs, you should look at LightGBM docs to see if they affect bin boundaries. If you're setting
any parameter that affects bin boundaries and reusing the same estimator, you should set referenceDataset to an empty array between calls.

Barrier Execution Mode Copy Section

By default LightGBM uses the regular spark paradigm for launching tasks and communicates with the driver to coordinate task execution.
The driver thread aggregates all task host:port information and then communicates the full list back to the workers in order for NetworkInit to be called.
This procedure requires the driver to know how many tasks there are, and a mismatch between the expected number of tasks and the actual number causes
the initialization to deadlock.

If you're experiencing network issues, you can try using Spark's barrier execution mode. SynapseML provides a UseBarrierExecutionMode flag,
to use Apache Spark's barrier() stage to ensure all tasks execute at the same time.
Barrier execution mode changes the logic to aggregate host:port information across all tasks in a synchronized way.
To use it in scala, you can call setUseBarrierExecutionMode(true), for example:

val lgbm = new LightGBMClassifier()
    .setLabelCol(labelColumn)
    .setObjective(binaryObjective)
    .setUseBarrierExecutionMode(true)
...

Note: barrier execution mode can also cause complicated issues, so use it only if needed.

Barrier execution mode is also the only mode that can recover from a task failure that happens after the
network topology has been negotiated. A LightGBM network is negotiated once and then fixed, so an individual
Spark task retry can never rejoin it. Spark restarts a barrier stage in its entirety, and the driver serves a
fresh topology round for each stage attempt, so training can survive a failure that would otherwise abort the
job. Failures that happen before a task joins the network are still retried normally in either mode.

Diagnosing "Connection refused" during training Copy Section

Distributed training first exchanges host:port information with the driver, which serves that exchange
once per stage attempt. If a task fails after that exchange, the regular (non-barrier) retry of that task
reconnects to a driver endpoint that is no longer accepting connections. Because Spark only reports the
most recent attempt, this can hide the failure that actually caused the retry.

When this happens, the reported error explains that it's a retry that could not rejoin the network, and
names the partition to investigate. Look for the first failed attempt of that partition in the executor
logs — that attempt holds the real cause.

IPv6 clusters Copy Section

Distributed training works on clusters whose executors only have IPv6 addresses.

The native LightGBM library is IPv4-only in every released version: it splits each machine list entry on
: and keeps the entry only when that yields exactly two parts, and it builds every socket with AF_INET
and inet_pton(AF_INET, ...). An IPv6 endpoint is therefore dropped or misread by the native parser, and
could not be dialed or accepted even if it survived parsing.

SynapseML handles this itself. The topology exchange publishes IPv6 endpoints in the unambiguous
[address]:port form, and each task then bridges the transport for the native library:

• the port a task advertised to its peers is owned by SynapseML, which accepts peer connections over
  either address family and forwards them to the native listener over IPv4 loopback;
• each IPv6 peer gets an IPv4 loopback relay that forwards what the native library sends to that peer's
  real IPv6 address;
• the machine list handed to the native library has the same entries in the same order, with every bridged
  endpoint rewritten to 127.0.0.1:port and an explicit rank, so LightGBM ranks are unchanged.

An IPv4 machine list is passed to the native library exactly as before, with no relay and no rewriting, so
IPv4 clusters see no behavior or performance change. On an IPv6 cluster, peer traffic takes one extra
loopback hop on each side, and IPv4 loopback must be available on the executors. Measured on a 16 core
developer machine over loopback, a bridged link sustains roughly a third of the throughput of a direct one
(about 0.6 GB/s per direction) for about four times the CPU per transferred byte, and adds roughly 200
microseconds to a small message round trip per hop. Links faster than a few Gb/s can therefore be limited
by the bridge rather than by the network. The relays never buffer in the JVM heap: a reader that stops
reading stops the sender, with only the socket buffers in between.

The advertised port is the one address peers know, so SynapseML binds it on every interface, exactly as the
native listener did before this change, and performs the LightGBM link handshake there itself. A machine
opens a LightGBM link by sending its rank, so a connection has to produce a valid, unused, lower rank within
a timeout before any of its bytes reach the native library; connections that stall, repeat a rank, or claim
a rank the topology does not have are closed by the bridge. That matters because the native accept loop has
no timeout and treats the first four bytes of any connection as a rank.

The native listener itself is the one thing this library cannot rebind: TcpSocket::Bind hardcodes
0.0.0.0, so while it is open it is reachable on every IPv4 interface. The bridge therefore claims each of
that listener's link slots itself, over IPv4 loopback, as soon as the port is bound — one slot per lower
rank, which is exactly how many the native library accepts before closing the listener. In practice the
listener is open for milliseconds on an unadvertised ephemeral port rather than for the whole handshake
phase, and every byte the native library reads from it comes from the bridge. Closing that window entirely
requires an upstream change to LightGBM: TcpSocket::Bind would have to take a bind address so that
Linkers::TryBind can pass a loopback one. Until then, a LightGBM training port must only be reachable
from the cluster's own executors, which was already true before this change.

The per peer relays are bound to IPv4 loopback and are not reachable from outside the machine, the number of
links a bridge will relay is capped at twice the machine count, and each lower rank may link only once. One
event loop thread serves every listener and every link, so a bridge runs exactly one thread whatever the
machine count is, with two fixed size buffers per link.

A connection the topology cannot need — an unsolicited one, one past the cap, or one that fails its
handshake — is refused without consuming any of that budget, so it cannot starve the links the native
library itself opens. Transport failures are retried while they can be: a dial that fails, immediately or
later, backs off and retries until its deadline, and a failure while handling one connection never closes
the listener that accepted it. A failure that cannot be retried away is recorded and fails the Spark task
with that cause, both before and after the native initialization call, rather than leaving the task waiting
on a transport that will not recover.

Link-local addresses (fe80::/10) are supported only when a peer advertises a zone identifier
(fe80::1%eth0) that names an interface on every other machine, since a link-local address is scoped to a
single interface. A task normalizes a numeric scope (an interface index, which only means something on the
machine that produced it) to the interface name before publishing its endpoint, and a peer that still
advertises a numeric scope is rejected. A link-local peer without a zone, or with a zone this machine does
not have, fails immediately with an error naming the address instead of hanging. Use a globally routable or
unique-local IPv6 address for distributed training.

2. Official Technical Reference & Guides (microsoft/microsoft.github.io) Copy Chapter

[microsoft.github.io](https://microsoft.github.io)

Many open source projects released by Microsoft products, services and teams also have
deployments powered by GitHub Pages on this domain.

To discover Microsoft-released open source projects, learn about Microsoft's open source
approach and other information, check us out online at opensource.microsoft.com.

If we can help answer questions about the open source program at Microsoft, you can
contact opensource at microsoft com - although we are not able to provide support
for projects through this mechanism.

Microsoft GitHub resources Copy Chapter

• Default repo templates including SECURITY.MD and code of conduct files: microsoft/repo-templates
• Code of Conduct (FAQ)

Microsoft open source resources Copy Chapter

• Open Source Blog
• The Open Source Show on Channel 9
• https://opensource.microsoft.com/

Resources for Microsoft employees Copy Chapter

• docs.opensource.microsoft.com
• Link your GitHub Account and join GitHub organizations automatically

How to identify an official Microsoft organization on GitHub Copy Chapter

• Official Microsoft GitHub orgs have a professional avatar and include a web address at Microsoft.com or an associated official domain
• Official Microsoft orgs are Verified on GitHub - look for the domain verification indicator on GitHub