### Doc/About

About us
========

.. include:: ../AUTHORS.rst

.. _citing-imbalanced-learn:

Citing imbalanced-learn
-----------------------

If you use imbalanced-learn in a scientific publication, we would appreciate
citations to the following paper::

  @article{JMLR:v18:16-365,
  author  = {Guillaume  Lema{{\^i}}tre and Fernando Nogueira and Christos K. Aridas},
  title   = {Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning},
  journal = {Journal of Machine Learning Research},
  year    = {2017},
  volume  = {18},
  number  = {17},
  pages   = {1-5},
  url     = {http://jmlr.org/papers/v18/16-365.html}
  }

---

### Doc/Combine

.. _combine:

=======================================
Combination of over- and under-sampling
=======================================

.. currentmodule:: imblearn.over_sampling

We previously presented :class:`SMOTE` and showed that this method can generate
noisy samples by interpolating new points between marginal outliers and
inliers. This issue can be solved by cleaning the space resulting
from over-sampling.

.. currentmodule:: imblearn.combine

In this regard, Tomek's link and edited nearest-neighbours are the two cleaning
methods that have been added to the pipeline after applying SMOTE over-sampling
to obtain a cleaner space. The two ready-to use classes imbalanced-learn
implements for combining over- and undersampling methods are: (i)
:class:`SMOTETomek` :cite:`batista2004study` and (ii) :class:`SMOTEENN`
:cite:`batista2003balancing`.

Those two classes can be used like any other sampler with parameters identical
to their former samplers::

  >>> from collections import Counter
  >>> from sklearn.datasets import make_classification
  >>> X, y = make_classification(n_samples=5000, n_features=2, n_informative=2,
  ...                            n_redundant=0, n_repeated=0, n_classes=3,
  ...                            n_clusters_per_class=1,
  ...                            weights=[0.01, 0.05, 0.94],
  ...                            class_sep=0.8, random_state=0)
  >>> print(sorted(Counter(y).items()))
  [(0, 64), (1, 262), (2, 4674)]
  >>> from imblearn.combine import SMOTEENN
  >>> smote_enn = SMOTEENN(random_state=0)
  >>> X_resampled, y_resampled = smote_enn.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 4060), (1, 4381), (2, 3502)]
  >>> from imblearn.combine import SMOTETomek
  >>> smote_tomek = SMOTETomek(random_state=0)
  >>> X_resampled, y_resampled = smote_tomek.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 4499), (1, 4566), (2, 4413)]

We can also see in the example below that :class:`SMOTEENN` tends to clean more
noisy samples than :class:`SMOTETomek`.

.. image:: ./auto_examples/combine/images/sphx_glr_plot_comparison_combine_001.png
   :target: ./auto_examples/combine/plot_comparison_combine.html
   :scale: 60
   :align: center

.. topic:: Examples

  * :ref:`sphx_glr_auto_examples_combine_plot_comparison_combine.py`

---

### Doc/Common Pitfalls

.. _common_pitfalls:

=========================================
Common pitfalls and recommended practices
=========================================

This section is a complement to the documentation given
`[here] <https://scikit-learn.org/dev/common_pitfalls.html>`_ in scikit-learn.
Indeed, we will highlight the issue of misusing resampling, leading to a
**data leakage**. Due to this leakage, the performance of a model reported
will be over-optimistic.

Data leakage
============

As mentioned in the scikit-learn documentation, data leakage occurs when
information that would not be available at prediction time is used when
building the model.

In the resampling setting, there is a common pitfall that corresponds to
resample the **entire** dataset before splitting it into a train and a test
partitions. Note that it would be equivalent to resample the train and test
partitions as well.

Such of a processing leads to two issues:

* the model will not be tested on a dataset with class distribution similar
  to the real use-case. Indeed, by resampling the entire dataset, both the
  training and testing set will be potentially balanced while the model should
  be tested on the natural imbalanced dataset to evaluate the potential bias
  of the model;
* the resampling procedure might use information about samples in the dataset
  to either generate or select some of the samples. Therefore, we might use
  information of samples which will be later used as testing samples which
  is the typical data leakage issue.

We will demonstrate the wrong and right ways to do some sampling and emphasize
the tools that one should use, avoiding to fall in the trap.

We will use the adult census dataset. For the sake of simplicity, we will only
use the numerical features. Also, we will make the dataset more imbalanced to
increase the effect of the wrongdoings::

  >>> from sklearn.datasets import fetch_openml
  >>> from imblearn.datasets import make_imbalance
  >>> X, y = fetch_openml(
  ...     data_id=1119, as_frame=True, return_X_y=True
  ... )
  >>> X = X.select_dtypes(include="number")
  >>> X, y = make_imbalance(
  ...     X, y, sampling_strategy={">50K": 300}, random_state=1
  ... )

Let's first check the balancing ratio on this dataset::

  >>> from collections import Counter
  >>> {key: value / len(y) for key, value in Counter(y).items()}
  {'<=50K': 0.988..., '>50K': 0.011...}

To later highlight some of the issue, we will keep aside a left-out set that we
will not use for the evaluation of the model::

  >>> from sklearn.model_selection import train_test_split
  >>> X, X_left_out, y, y_left_out = train_test_split(
  ...     X, y, stratify=y, random_state=0
  ... )

We will use a :class:`sklearn.ensemble.HistGradientBoostingClassifier` as a
baseline classifier. First, we will train and check the performance of this
classifier, without any preprocessing to alleviate the bias toward the majority
class. We evaluate the generalization performance of the classifier via
cross-validation::

  >>> from sklearn.ensemble import HistGradientBoostingClassifier
  >>> from sklearn.model_selection import cross_validate
  >>> model = HistGradientBoostingClassifier(random_state=0)
  >>> cv_results = cross_validate(
  ...     model, X, y, scoring="balanced_accuracy",
  ...     return_train_score=True, return_estimator=True,
  ...     n_jobs=-1
  ... )
  >>> print(
  ...     f"Balanced accuracy mean +/- std. dev.: "
  ...     f"{cv_results['test_score'].mean():.3f} +/- "
  ...     f"{cv_results['test_score'].std():.3f}"
  ... )
  Balanced accuracy mean +/- std. dev.: 0.609 +/- 0.024

We see that the classifier does not give good performance in terms of balanced
accuracy mainly due to the class imbalance issue.

In the cross-validation, we stored the different classifiers of all folds. We
will show that evaluating these classifiers on the left-out data will give
close statistical performance::

  >>> import numpy as np
  >>> from sklearn.metrics import balanced_accuracy_score
  >>> scores = []
  >>> for fold_id, cv_model in enumerate(cv_results["estimator"]):
  ...     scores.append(
  ...         balanced_accuracy_score(
  ...             y_left_out, cv_model.predict(X_left_out)
  ...         )
  ...     )
  >>> print(
  ...     f"Balanced accuracy mean +/- std. dev.: "
  ...     f"{np.mean(scores):.3f} +/- {np.std(scores):.3f}"
  ... )
  Balanced accuracy mean +/- std. dev.: 0.628 +/- 0.009

Let's now show the **wrong** pattern to apply when it comes to resampling to
alleviate the class imbalance issue. We will use a sampler to balance the
**entire** dataset and check the statistical performance of our classifier via
cross-validation::

  >>> from imblearn.under_sampling import RandomUnderSampler
  >>> sampler = RandomUnderSampler(random_state=0)
  >>> X_resampled, y_resampled = sampler.fit_resample(X, y)
  >>> model = HistGradientBoostingClassifier(random_state=0)
  >>> cv_results = cross_validate(
  ...     model, X_resampled, y_resampled, scoring="balanced_accuracy",
  ...     return_train_score=True, return_estimator=True,
  ...     n_jobs=-1
  ... )
  >>> print(
  ...     f"Balanced accuracy mean +/- std. dev.: "
  ...     f"{cv_results['test_score'].mean():.3f} +/- "
  ...     f"{cv_results['test_score'].std():.3f}"
  ... )
  Balanced accuracy mean +/- std. dev.: 0.724 +/- 0.042

The cross-validation performance looks good, but evaluating the classifiers
on the left-out data shows a different picture::

  >>> scores = []
  >>> for fold_id, cv_model in enumerate(cv_results["estimator"]):
  ...     scores.append(
  ...         balanced_accuracy_score(
  ...             y_left_out, cv_model.predict(X_left_out)
  ...        )
  ...     )
  >>> print(
  ...     f"Balanced accuracy mean +/- std. dev.: "
  ...     f"{np.mean(scores):.3f} +/- {np.std(scores):.3f}"
  ... )
  Balanced accuracy mean +/- std. dev.: 0.698 +/- 0.014

We see that the performance is now worse than the cross-validated performance.
Indeed, the data leakage gave us too optimistic results due to the reason
stated earlier in this section.

We will now illustrate the correct pattern to use. Indeed, as in scikit-learn,
using a :class:`~imblearn.pipeline.Pipeline` avoids to make any data leakage
because the resampling will be delegated to imbalanced-learn and does not
require any manual steps::

  >>> from imblearn.pipeline import make_pipeline
  >>> model = make_pipeline(
  ...     RandomUnderSampler(random_state=0),
  ...     HistGradientBoostingClassifier(random_state=0)
  ... )
  >>> cv_results = cross_validate(
  ...     model, X, y, scoring="balanced_accuracy",
  ...     return_train_score=True, return_estimator=True,
  ...     n_jobs=-1
  ... )
  >>> print(
  ...     f"Balanced accuracy mean +/- std. dev.: "
  ...     f"{cv_results['test_score'].mean():.3f} +/- "
  ...     f"{cv_results['test_score'].std():.3f}"
  ... )
  Balanced accuracy mean +/- std. dev.: 0.732 +/- 0.019

We observe that we get good statistical performance as well. However, now we
can check the performance of the model from each cross-validation fold to
ensure that we have similar performance::

  >>> scores = []
  >>> for fold_id, cv_model in enumerate(cv_results["estimator"]):
  ...     scores.append(
  ...         balanced_accuracy_score(
  ...             y_left_out, cv_model.predict(X_left_out)
  ...        )
  ...     )
  >>> print(
  ...     f"Balanced accuracy mean +/- std. dev.: "
  ...     f"{np.mean(scores):.3f} +/- {np.std(scores):.3f}"
  ... )
  Balanced accuracy mean +/- std. dev.: 0.727 +/- 0.008

We see that the statistical performance are very close to the cross-validation
study that we perform, without any sign of over-optimistic results.

---

### Doc/Developers Utils

.. _developers-utils:

===================
Developer guideline
===================

Developer utilities
-------------------

Imbalanced-learn contains a number of utilities to help with development. These are
located in :mod:`imblearn.utils`, and include tools in a number of categories.
All the following functions and classes are in the module :mod:`imblearn.utils`.

.. warning ::

   These utilities are meant to be used internally within the imbalanced-learn
   package. They are not guaranteed to be stable between versions of
   imbalanced-learn. Backports, in particular, will be removed as the
   imbalanced-learn dependencies evolve.


Validation Tools
~~~~~~~~~~~~~~~~

.. currentmodule:: imblearn.utils

These are tools used to check and validate input. When you write a function
which accepts arrays, matrices, or sparse matrices as arguments, the following
should be used when applicable.

- :func:`check_neighbors_object`: Check the objects is consistent to be a NN.
- :func:`check_target_type`: Check the target types to be conform to the current
  samplers.
- :func:`check_sampling_strategy`: Checks that sampling target is consistent with
  the type and return a dictionary containing each targeted class with its
  corresponding number of pixel.


Deprecation
~~~~~~~~~~~

.. currentmodule:: imblearn.utils.deprecation

.. warning ::
   Apart from :func:`deprecate_parameter` the rest of this section is taken from
   scikit-learn. Please refer to their original documentation.

If any publicly accessible method, function, attribute or parameter
is renamed, we still support the old one for two releases and issue
a deprecation warning when it is called/passed/accessed.
E.g., if the function ``zero_one`` is renamed to ``zero_one_loss``,
we add the decorator ``deprecated`` (from ``sklearn.utils``)
to ``zero_one`` and call ``zero_one_loss`` from that function::

    from ..utils import deprecated

    def zero_one_loss(y_true, y_pred, normalize=True):
        # actual implementation
        pass

    @deprecated("Function 'zero_one' was renamed to 'zero_one_loss' "
                "in version 0.13 and will be removed in release 0.15. "
                "Default behavior is changed from 'normalize=False' to "
                "'normalize=True'")
    def zero_one(y_true, y_pred, normalize=False):
        return zero_one_loss(y_true, y_pred, normalize)

If an attribute is to be deprecated,
use the decorator ``deprecated`` on a property.
E.g., renaming an attribute ``labels_`` to ``classes_`` can be done as::

    @property
    @deprecated("Attribute labels_ was deprecated in version 0.13 and "
                "will be removed in 0.15. Use 'classes_' instead")
    def labels_(self):
        return self.classes_

If a parameter has to be deprecated, use ``FutureWarning`` appropriately.
In the following example, k is deprecated and renamed to n_clusters::

    import warnings

    def example_function(n_clusters=8, k=None):
        if k is not None:
            warnings.warn("'k' was renamed to n_clusters in version 0.13 and "
                          "will be removed in 0.15.", DeprecationWarning)
            n_clusters = k

As in these examples, the warning message should always give both the
version in which the deprecation happened and the version in which the
old behavior will be removed. If the deprecation happened in version
0.x-dev, the message should say deprecation occurred in version 0.x and
the removal will be in 0.(x+2). For example, if the deprecation happened
in version 0.18-dev, the message should say it happened in version 0.18
and the old behavior will be removed in version 0.20.

In addition, a deprecation note should be added in the docstring, recalling the
same information as the deprecation warning as explained above. Use the
``.. deprecated::`` directive::

  .. deprecated:: 0.13
     ``k`` was renamed to ``n_clusters`` in version 0.13 and will be removed
     in 0.15.

On the top of all the functionality provided by scikit-learn. imbalanced-learn
provides :func:`deprecate_parameter`: which is used to deprecate a sampler's
parameter (attribute) by another one.

Making a release
----------------
This section document the different steps that are necessary to make a new
imbalanced-learn release.

Major release
~~~~~~~~~~~~~

* Update the release note `whats_new/v0.<version number>.rst` by giving a date
  and removing the status "Under development" from the title.
* Run `bumpversion release`. It will remove the `dev0` tag.
* Commit the change `git commit -am "bumpversion 0.<version number>.0"`
  (e.g., `git commit -am "bumpversion 0.5.0"`).
* Create a branch for this version
  (e.g., `git checkout -b 0.<version number>.X`).
* Push the new branch into the upstream remote imbalanced-learn repository.
* Change the `symlink` in the
  `imbalanced-learn website repository <https://github.com/imbalanced-learn/imbalanced-learn.github.io>`_
  such that stable points to the latest release version,
  i.e, `0.<version number>`. To do this, clone the repository,
  `run unlink stable`, followed by `ln -s 0.<version number> stable`. To check
  that this was performed correctly, ensure that stable has the new version
  number using `ls -l`.
* Return to your imbalanced-learn repository, in the branch
  `0.<version number>.X`.
* Create the source distribution and wheel: `python setup.py sdist` and
  `python setup.py bdist_wheel`.
* Upload these file to PyPI using `twine upload dist/*`
* Switch to the `master` branch and run `bumpversion minor`, commit and push on
  upstream. We are officially at `0.<version number + 1>.0.dev0`.
* Create a GitHub release by clicking on "Draft a new release" here.
  "Tag version" should be the latest version number (e.g., `0.<version>.0`),
  "Target" should be the branch for that the release
  (e.g., `0.<version number>.X`) and "Release title" should be
  "Version <version number>". Add the notes from the release notes there.
* Add a new `v0.<version number + 1>.rst` file in `doc/whats_new/` and
  `.. include::` this new file in `doc/whats_new.rst`. Mark the version as the
  version under development.
* Finally, go to the `conda-forge feedstock <https://github.com/conda-forge/imbalanced-learn-feedstock>`_
  and a new PR will be created when the feedstock will synchronizing with the
  PyPI repository. Merge this PR such that we have the binary for `conda`
  available.

Bug fix release
~~~~~~~~~~~~~~~

* Find the commit(s) hash of the bug fix commit you wish to back port using
  `git log`.
* Checkout the branch for the lastest release, e.g.,
  `git checkout 0.<version number>.X`.
* Append the bug fix commit(s) to the branch using `git cherry-pick <hash>`.
  Alternatively, you can use interactive rebasing from the `master` branch.
* Bump the version number with bumpversion patch. This will bump the patch
  version, for example from `0.X.0` to `0.X.* dev0`.
* Mark the current version as a release version (as opposed to `dev` version)
  with `bumpversion release --allow-dirty`. It will bump the version, for
  example from `0.X.* dev0` to `0.X.1`.
* Commit the changes with `git commit -am 'bumpversion <new version>'`.
* Push the changes to the release branch in upstream, e.g.
  `git push <upstream remote> <release branch>`.
* Use the same process as in a major release to upload on PyPI and conda-forge.

---

### Doc/Ensemble

.. _ensemble:

====================
Ensemble of samplers
====================

.. currentmodule:: imblearn.ensemble

.. _ensemble_meta_estimators:

Classifier including inner balancing samplers
=============================================

.. _bagging:

Bagging classifier
------------------

In ensemble classifiers, bagging methods build several estimators on different
randomly selected subset of data. In scikit-learn, this classifier is named
:class:`~sklearn.ensemble.BaggingClassifier`. However, this classifier does not
allow each subset of data to be balanced. Therefore, when training on an imbalanced
data set, this classifier will favor the majority classes::

  >>> from sklearn.datasets import make_classification
  >>> X, y = make_classification(n_samples=10000, n_features=2, n_informative=2,
  ...                            n_redundant=0, n_repeated=0, n_classes=3,
  ...                            n_clusters_per_class=1,
  ...                            weights=[0.01, 0.05, 0.94], class_sep=0.8,
  ...                            random_state=0)
  >>> from sklearn.model_selection import train_test_split
  >>> from sklearn.metrics import balanced_accuracy_score
  >>> from sklearn.ensemble import BaggingClassifier
  >>> from sklearn.tree import DecisionTreeClassifier
  >>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
  >>> bc = BaggingClassifier(DecisionTreeClassifier(), random_state=0)
  >>> bc.fit(X_train, y_train) #doctest:
  BaggingClassifier(...)
  >>> y_pred = bc.predict(X_test)
  >>> balanced_accuracy_score(y_test, y_pred)
  0.77...

In :class:`BalancedBaggingClassifier`, each bootstrap sample will be further
resampled to achieve the `sampling_strategy` desired. Therefore,
:class:`BalancedBaggingClassifier` takes the same parameters as the
scikit-learn :class:`~sklearn.ensemble.BaggingClassifier`. In addition, the
sampling is controlled by the parameter `sampler` or the two parameters
`sampling_strategy` and `replacement`, if one wants to use the
:class:`~imblearn.under_sampling.RandomUnderSampler`::

  >>> from imblearn.ensemble import BalancedBaggingClassifier
  >>> bbc = BalancedBaggingClassifier(DecisionTreeClassifier(),
  ...                                 sampling_strategy='auto',
  ...                                 replacement=False,
  ...                                 random_state=0)
  >>> bbc.fit(X_train, y_train)
  BalancedBaggingClassifier(...)
  >>> y_pred = bbc.predict(X_test)
  >>> balanced_accuracy_score(y_test, y_pred)
  0.8...

Changing the `sampler` will give rise to different known implementations
:cite:`maclin1997empirical`, :cite:`hido2009roughly`,
:cite:`wang2009diversity`. You can refer to the following example which shows these
different methods in practice:
:ref:`sphx_glr_auto_examples_ensemble_plot_bagging_classifier.py`

.. _forest:

Forest of randomized trees
--------------------------

:class:`BalancedRandomForestClassifier` is another ensemble method in which
each tree of the forest will be provided a balanced bootstrap sample
:cite:`chen2004using`. This class provides all functionality of the
:class:`~sklearn.ensemble.RandomForestClassifier`::

  >>> from imblearn.ensemble import BalancedRandomForestClassifier
  >>> brf = BalancedRandomForestClassifier(
  ...     n_estimators=100, random_state=0, sampling_strategy="all", replacement=True,
  ...     bootstrap=False,
  ... )
  >>> brf.fit(X_train, y_train)
  BalancedRandomForestClassifier(...)
  >>> y_pred = brf.predict(X_test)
  >>> balanced_accuracy_score(y_test, y_pred)
  0.8...

.. _boosting:

Boosting
--------

Several methods taking advantage of boosting have been designed.

:class:`RUSBoostClassifier` randomly under-samples the dataset before performing
a boosting iteration :cite:`seiffert2009rusboost`::

  >>> from imblearn.ensemble import RUSBoostClassifier
  >>> rusboost = RUSBoostClassifier(n_estimators=200, random_state=0)
  >>> rusboost.fit(X_train, y_train)
  RUSBoostClassifier(...)
  >>> y_pred = rusboost.predict(X_test)
  >>> balanced_accuracy_score(y_test, y_pred)
  0...

A specific method which uses :class:`~sklearn.ensemble.AdaBoostClassifier` as
learners in the bagging classifier is called "EasyEnsemble". The
:class:`EasyEnsembleClassifier` allows bagging AdaBoost learners which are
trained on balanced bootstrap samples :cite:`liu2008exploratory`. Similarly to
the :class:`BalancedBaggingClassifier` API, one can construct the ensemble as::

  >>> from imblearn.ensemble import EasyEnsembleClassifier
  >>> eec = EasyEnsembleClassifier(random_state=0)
  >>> eec.fit(X_train, y_train)
  EasyEnsembleClassifier(...)
  >>> y_pred = eec.predict(X_test)
  >>> balanced_accuracy_score(y_test, y_pred)
  0.6...

.. topic:: Examples

  * :ref:`sphx_glr_auto_examples_ensemble_plot_comparison_ensemble_classifier.py`

---

### Doc/Index

.. project-template documentation master file, created by
   sphinx-quickstart on Mon Jan 18 14:44:12 2016.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

:notoc:

##############################
imbalanced-learn documentation
##############################

**Date**: |today| **Version**: |version|

**Useful links**:
`Binary Installers <https://pypi.org/project/imbalanced-learn>`__ |
`Source Repository <https://github.com/scikit-learn-contrib/imbalanced-learn>`__ |
`Issues & Ideas <https://github.com/scikit-learn-contrib/imbalanced-learn/issues>`__ |
`Q&A Support <https://gitter.im/scikit-learn-contrib/imbalanced-learn>`__

Imbalanced-learn (imported as :mod:`imblearn`) is an open source, MIT-licensed
library relying on scikit-learn (imported as :mod:`sklearn`) and provides tools
when dealing with classification with imbalanced classes.

.. grid:: 1 2 2 2
    :gutter: 4
    :padding: 2 2 0 0
    :class-container: sd-text-center

    .. grid-item-card:: Getting started
        :img-top: _static/index_getting_started.svg
        :class-card: intro-card
        :shadow: md

        Check out the getting started guides to install `imbalanced-learn`.
        Some extra information to get started with a new contribution is also provided.

        +++

        .. button-ref:: getting_started
            :ref-type: ref
            :click-parent:
            :color: secondary
            :expand:

            To the installation guideline

    .. grid-item-card::  User guide
        :img-top: _static/index_user_guide.svg
        :class-card: intro-card
        :shadow: md

        The user guide provides in-depth information on the key concepts of
        `imbalanced-learn` with useful background information and explanation.

        +++

        .. button-ref:: user_guide
            :ref-type: ref
            :click-parent:
            :color: secondary
            :expand:

            To the user guide

    .. grid-item-card::  API reference
        :img-top: _static/index_api.svg
        :class-card: intro-card
        :shadow: md

        The reference guide contains a detailed description of
        the `imbalanced-learn` API. To known more about methods parameters.

        +++

        .. button-ref:: api
            :ref-type: ref
            :click-parent:
            :color: secondary
            :expand:

            To the reference guide

    .. grid-item-card::  Examples
        :img-top: _static/index_examples.svg
        :class-card: intro-card
        :shadow: md

        The gallery of examples is a good place to see `imbalanced-learn` in action.
        Select an example and dive in.

        +++

        .. button-ref:: general_examples
            :ref-type: ref
            :click-parent:
            :color: secondary
            :expand:

            To the gallery of examples


.. toctree::
    :maxdepth: 3
    :hidden:
    :titlesonly:

    install
    user_guide
    references/index
    auto_examples/index
    whats_new
    about

---

### Doc/Install

.. _getting_started:

###############
Getting Started
###############

Prerequisites
=============

.. |PythonMinVersion| replace:: 3.10
.. |NumPyMinVersion| replace:: 1.25.2
.. |SciPyMinVersion| replace:: 1.11.4
.. |ScikitLearnMinVersion| replace:: 1.4.2
.. |MatplotlibMinVersion| replace:: 3.7.3
.. |PandasMinVersion| replace:: 2.0.3
.. |TensorflowMinVersion| replace:: 2.16.1
.. |KerasMinVersion| replace:: 3.3.3
.. |SeabornMinVersion| replace:: 0.12.2
.. |PytestMinVersion| replace:: 7.2.2

`imbalanced-learn` requires the following dependencies:

- Python (>= |PythonMinVersion|)
- NumPy (>= |NumPyMinVersion|)
- SciPy (>= |SciPyMinVersion|)
- Scikit-learn (>= |ScikitLearnMinVersion|)
- Pytest (>= |PytestMinVersion|)

Additionally, `imbalanced-learn` requires the following optional dependencies:

- Pandas (>= |PandasMinVersion|) for dealing with dataframes
- Tensorflow (>= |TensorflowMinVersion|) for dealing with TensorFlow models
- Keras (>= |KerasMinVersion|) for dealing with Keras models

The examples will requires the following additional dependencies:

- Matplotlib (>= |MatplotlibMinVersion|)
- Seaborn (>= |SeabornMinVersion|)

Install
=======

From PyPi or conda-forge repositories
-------------------------------------

imbalanced-learn is currently available on the PyPi's repositories and you can
install it via `pip`::

  pip install imbalanced-learn

The package is released also on the conda-forge repositories and you can install
it with `conda` (or `mamba`)::

  conda install -c conda-forge imbalanced-learn

Intel optimizations via scikit-learn-intelex
--------------------------------------------

Imbalanced-learn relies entirely on scikit-learn algorithms. Intel provides an
optimized version of scikit-learn for Intel hardwares, called scikit-learn-intelex.
Installing scikit-learn-intelex and patching scikit-learn will activate the
Intel optimizations.

You can refer to the following
`blog post <https://medium.com/intel-analytics-software/why-pay-more-for-machine-learning-893683bd78e4>`_
for some benchmarks.

Refer to the following documentation for instructions:

- `Installation guide <https://intel.github.io/scikit-learn-intelex/installation.html>`_.
- `Patching guide <https://intel.github.io/scikit-learn-intelex/what-is-patching.html>`_.

From source available on GitHub
-------------------------------

If you prefer, you can clone it and run the setup.py file. Use the following
commands to get a copy from Github and install all dependencies::

  git clone https://github.com/scikit-learn-contrib/imbalanced-learn.git
  cd imbalanced-learn
  pip install .

Be aware that you can install in developer mode with::

  pip install --no-build-isolation --editable .

If you wish to make pull-requests on GitHub, we advise you to install
pre-commit::

  pip install pre-commit
  pre-commit install

Test and coverage
=================

You want to test the code before to install::

  $ make test

You wish to test the coverage of your version::

  $ make coverage

You can also use `pytest`::

  $ pytest imblearn -v

Contribute
==========

You can contribute to this code through Pull Request on GitHub_. Please, make
sure that your code is coming with unit tests to ensure full coverage and
continuous integration in the API.

.. _GitHub: https://github.com/scikit-learn-contrib/imbalanced-learn/pulls

---

### Doc/Introduction

.. _introduction:

============
Introduction
============

.. _api_imblearn:

API's of imbalanced-learn samplers
----------------------------------

The available samplers follow the
`scikit-learn API <https://scikit-learn.org/stable/getting_started.html#fitting-and-predicting-estimator-basics>`_
using the base estimator
and incorporating a sampling functionality via the ``sample`` method:

:Estimator:

    The base object, implements a ``fit`` method to learn from data::

      estimator = obj.fit(data, targets)

:Resampler:

    To resample a data sets, each sampler implements a ``fit_resample`` method::

      data_resampled, targets_resampled = obj.fit_resample(data, targets)

Imbalanced-learn samplers accept the same inputs as scikit-learn estimators:

* `data`, 2-dimensional array-like structures, such as:
   * Python's list of lists :class:`list`,
   * Numpy arrays :class:`numpy.ndarray`,
   * Panda dataframes :class:`pandas.DataFrame`,
   * Scipy sparse matrices :class:`scipy.sparse.csr_matrix` or :class:`scipy.sparse.csc_matrix`;

* `targets`, 1-dimensional array-like structures, such as:
   * Numpy arrays :class:`numpy.ndarray`,
   * Pandas series :class:`pandas.Series`.

The output will be of the following type:

* `data_resampled`, 2-dimensional aray-like structures, such as:
   * Numpy arrays :class:`numpy.ndarray`,
   * Pandas dataframes :class:`pandas.DataFrame`,
   * Scipy sparse matrices :class:`scipy.sparse.csr_matrix` or :class:`scipy.sparse.csc_matrix`;

* `targets_resampled`, 1-dimensional array-like structures, such as:
   * Numpy arrays :class:`numpy.ndarray`,
   * Pandas series :class:`pandas.Series`.

.. topic:: Pandas in/out

   Unlike scikit-learn, imbalanced-learn provides support for pandas in/out.
   Therefore providing a dataframe, will output as well a dataframe.

.. topic:: Sparse input

   For sparse input the data is **converted to the Compressed Sparse Rows
   representation** (see ``scipy.sparse.csr_matrix``) before being fed to the
   sampler. To avoid unnecessary memory copies, it is recommended to choose the
   CSR representation upstream.

.. _problem_statement:

Problem statement regarding imbalanced data sets
------------------------------------------------

The learning and prediction phrases of machine learning algorithms
can be impacted by the issue of **imbalanced datasets**. This imbalance
refers to the difference in the number of samples across different classes.
We demonstrate the effect of training a `Logistic Regression classifier
<https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html>`_
with varying levels of class balancing by adjusting their weights.

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_001.png
   :target: ./auto_examples/over-sampling/plot_comparison_over_sampling.html
   :scale: 60
   :align: center

As expected, the decision function of the Logistic Regression classifier varies significantly
depending on how imbalanced the data is. With a greater imbalance ratio, the decision function
tends to favour the class with the larger number of samples, usually referred to as the
**majority class**.

---

### Doc/Metrics

.. _metrics:

=======
Metrics
=======

.. currentmodule:: imblearn.metrics

Classification metrics
----------------------

Currently, scikit-learn only offers the
``sklearn.metrics.balanced_accuracy_score`` (in 0.20) as metric to deal with
imbalanced datasets. The module :mod:`imblearn.metrics` offers a couple of
other metrics which are used in the literature to evaluate the quality of
classifiers.

.. _sensitivity_specificity:

Sensitivity and specificity metrics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sensitivity and specificity are metrics which are well known in medical
imaging. Sensitivity (also called true positive rate or recall) is the
proportion of the positive samples which is well classified while specificity
(also called true negative rate) is the proportion of the negative samples
which are well classified. Therefore, depending of the field of application,
either the sensitivity/specificity or the precision/recall pair of metrics are
used.

Currently, only the `precision and recall metrics
<http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html>`_
are implemented in scikit-learn. :func:`sensitivity_specificity_support`,
:func:`sensitivity_score`, and :func:`specificity_score` add the possibility to
use those metrics.

.. _imbalanced_metrics:

Additional metrics specific to imbalanced datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The :func:`geometric_mean_score`
:cite:`barandela2003strategies,kubat1997addressing` is the root of the product
of class-wise sensitivity. This measure tries to maximize the accuracy on each
of the classes while keeping these accuracies balanced.

The :func:`make_index_balanced_accuracy` :cite:`garcia2012effectiveness` can
wrap any metric and give more importance to a specific class using the
parameter ``alpha``.

.. _macro_averaged_mean_absolute_error:

Macro-Averaged Mean Absolute Error (MA-MAE)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Ordinal classification is used when there is a rank among classes, for example
levels of functionality or movie ratings.

The :func:`macro_averaged_mean_absolute_error` :cite:`esuli2009ordinal` is used
for imbalanced ordinal classification. The mean absolute error is computed for
each class and averaged over classes, giving an equal weight to each class.

.. _classification_report:

Summary of important metrics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The :func:`classification_report_imbalanced` will compute a set of metrics per
class and summarize it in a table. The parameter `output_dict` allows to get a
string or a Python dictionary. This dictionary can be reused to create a Pandas
dataframe for instance.

The bottom row (i.e "avg/total") contains the weighted average by the support
(i.e column "sup") of each column.

Note that the weighted average of the class recalls is also known as the
classification accuracy.

.. _pairwise_metrics:

Pairwise metrics
----------------

The :mod:`imblearn.metrics.pairwise` submodule implements pairwise distances
that are available in scikit-learn while used in some of the methods in
imbalanced-learn.

.. _vdm:

Value Difference Metric
~~~~~~~~~~~~~~~~~~~~~~~

The class :class:`~imblearn.metrics.pairwise.ValueDifferenceMetric` is
implementing the Value Difference Metric proposed in
:cite:`stanfill1986toward`. This measure is used to compute the proximity
of two samples composed of only categorical values.

Given a single feature, categories with similar correlation with the target
vector will be considered closer. Let's give an example to illustrate this
behaviour as given in :cite:`wilson1997improved`. `X` will be represented by a
single feature which will be some color and the target will be if a sample is
whether or not an apple::

    >>> import numpy as np
    >>> X = np.array(["green"] * 10 + ["red"] * 10 + ["blue"] * 10).reshape(-1, 1)
    >>> y = ["apple"] * 8 + ["not apple"] * 5 + ["apple"] * 7 + ["not apple"] * 9 + ["apple"]

In this dataset, the categories "red" and "green" are more correlated to the
target `y` and should have a smaller distance than with the category "blue".
We should this behaviour. Be aware that we need to encode the `X` to work with
numerical values::

    >>> from sklearn.preprocessing import OrdinalEncoder
    >>> encoder = OrdinalEncoder(dtype=np.int32)
    >>> X_encoded = encoder.fit_transform(X)

Now, we can compute the distance between three different samples representing
the different categories::

    >>> from imblearn.metrics.pairwise import ValueDifferenceMetric
    >>> vdm = ValueDifferenceMetric().fit(X_encoded, y)
    >>> X_test = np.array(["green", "red", "blue"]).reshape(-1, 1)
    >>> X_test_encoded = encoder.transform(X_test)
    >>> vdm.pairwise(X_test_encoded)
    array([[0.  ,  0.04,  1.96],
           [0.04,  0.  ,  1.44],
           [1.96,  1.44,  0.  ]])

We see that the minimum distance happen when the categories "red" and "green"
are compared. Whenever comparing with "blue", the distance is much larger.

**Mathematical formulation**

The distance between feature values of two samples is defined as:

.. math::
    \delta(x, y) = \sum_{c=1}^{C} |p(c|x_{f}) - p(c|y_{f})|^{k} \ ,

where :math:`x` and :math:`y` are two samples and :math:`f` a given
feature, :math:`C` is the number of classes, :math:`p(c|x_{f})` is the
conditional probability that the output class is :math:`c` given that
the feature value :math:`f` has the value :math:`x` and :math:`k` an
exponent usually defined to 1 or 2.

The distance for the feature vectors :math:`X` and :math:`Y` is
subsequently defined as:

.. math::
    \Delta(X, Y) = \sum_{f=1}^{F} \delta(X_{f}, Y_{f})^{r} \ ,

where :math:`F` is the number of feature and :math:`r` an exponent usually
defined equal to 1 or 2.

---

### Doc/Miscellaneous

.. _miscellaneous:

======================
Miscellaneous samplers
======================

.. currentmodule:: imblearn

.. _function_sampler:

Custom samplers
---------------

A fully customized sampler, :class:`FunctionSampler`, is available in
imbalanced-learn such that you can fast prototype your own sampler by defining
a single function. Additional parameters can be added using the attribute
``kw_args`` which accepts a dictionary. The following example illustrates how
to retain the 10 first elements of the array ``X`` and ``y``::

  >>> import numpy as np
  >>> from imblearn import FunctionSampler
  >>> from sklearn.datasets import make_classification
  >>> X, y = make_classification(n_samples=5000, n_features=2, n_informative=2,
  ...                            n_redundant=0, n_repeated=0, n_classes=3,
  ...                            n_clusters_per_class=1,
  ...                            weights=[0.01, 0.05, 0.94],
  ...                            class_sep=0.8, random_state=0)
  >>> def func(X, y):
  ...   return X[:10], y[:10]
  >>> sampler = FunctionSampler(func=func)
  >>> X_res, y_res = sampler.fit_resample(X, y)
  >>> np.all(X_res == X[:10])
  True
  >>> np.all(y_res == y[:10])
  True

In addition, the parameter ``validate`` controls input checking. For instance,
turning ``validate=False`` allows to pass any type of target ``y`` and do some
sampling for regression targets::

  >>> from sklearn.datasets import make_regression
  >>> X_reg, y_reg = make_regression(n_samples=100, random_state=42)
  >>> rng = np.random.RandomState(42)
  >>> def dummy_sampler(X, y):
  ...     indices = rng.choice(np.arange(X.shape[0]), size=10)
  ...     return X[indices], y[indices]
  >>> sampler = FunctionSampler(func=dummy_sampler, validate=False)
  >>> X_res, y_res = sampler.fit_resample(X_reg, y_reg)
  >>> y_res
  array([  41.49112498, -142.78526195,   85.55095317,  141.43321419,
           75.46571114,  -67.49177372,  159.72700509, -169.80498923,
          211.95889757,  211.95889757])

We illustrated the use of such sampler to implement an outlier rejection
estimator which can be easily used within a
:class:`~imblearn.pipeline.Pipeline`:
:ref:`sphx_glr_auto_examples_applications_plot_outlier_rejections.py`

.. _generators:

Custom generators
-----------------

Imbalanced-learn provides specific generators for TensorFlow and Keras which
will generate balanced mini-batches.

.. _tensorflow_generator:

TensorFlow generator
~~~~~~~~~~~~~~~~~~~~

The :func:`~imblearn.tensorflow.balanced_batch_generator` allows to generate
balanced mini-batches using an imbalanced-learn sampler which returns indices.

Let's first generate some data::

  >>> n_features, n_classes = 10, 2
  >>> X, y = make_classification(
  ...     n_samples=10_000, n_features=n_features, n_informative=2,
  ...     n_redundant=0, n_repeated=0, n_classes=n_classes,
  ...     n_clusters_per_class=1, weights=[0.1, 0.9],
  ...     class_sep=0.8, random_state=0
  ... )
  >>> X = X.astype(np.float32)

Then, we can create the generator that will yield mini-batches that will be
balanced::

  >>> from imblearn.under_sampling import RandomUnderSampler
  >>> from imblearn.tensorflow import balanced_batch_generator
  >>> training_generator, steps_per_epoch = balanced_batch_generator(
  ...     X,
  ...     y,
  ...     sample_weight=None,
  ...     sampler=RandomUnderSampler(),
  ...     batch_size=32,
  ...     random_state=42,
  ... )

The ``generator`` and ``steps_per_epoch`` are used during the training of a
Tensorflow model. We will illustrate how to use this generator. First, we can
define a logistic regression model which will be optimized by a gradient
descent::

  >>> import tensorflow as tf
  >>> # initialize the weights and intercept
  >>> normal_initializer = tf.random_normal_initializer(mean=0, stddev=0.01)
  >>> coef = tf.Variable(normal_initializer(
  ...     shape=[n_features, n_classes]), dtype="float32"
  ... )
  >>> intercept = tf.Variable(
  ...     normal_initializer(shape=[n_classes]), dtype="float32"
  ... )
  >>> # define the model
  >>> def logistic_regression(X):
  ...     return tf.nn.softmax(tf.matmul(X, coef) + intercept)
  >>> # define the loss function
  >>> def cross_entropy(y_true, y_pred):
  ...     y_true = tf.one_hot(y_true, depth=n_classes)
  ...     y_pred = tf.clip_by_value(y_pred, 1e-9, 1.)
  ...     return tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred)))
  >>> # define our metric
  >>> def balanced_accuracy(y_true, y_pred):
  ...     cm = tf.math.confusion_matrix(tf.cast(y_true, tf.int64), tf.argmax(y_pred, 1))
  ...     per_class = np.diag(cm) / tf.math.reduce_sum(cm, axis=1)
  ...     return np.mean(per_class)
  >>> # define the optimizer
  >>> optimizer = tf.optimizers.SGD(learning_rate=0.01)
  >>> # define the optimization step
  >>> def run_optimization(X, y):
  ...     with tf.GradientTape() as g:
  ...         y_pred = logistic_regression(X)
  ...         loss = cross_entropy(y, y_pred)
  ...     gradients = g.gradient(loss, [coef, intercept])
  ...     optimizer.apply_gradients(zip(gradients, [coef, intercept]))

Once initialized, the model is trained by iterating on balanced mini-batches of
data and minimizing the loss previously defined::

  >>> epochs = 10
  >>> for e in range(epochs):
  ...     y_pred = logistic_regression(X)
  ...     loss = cross_entropy(y, y_pred)
  ...     bal_acc = balanced_accuracy(y, y_pred)
  ...     print(f"epoch: {e}, loss: {loss:.3f}, accuracy: {bal_acc}")
  ...     for i in range(steps_per_epoch):
  ...         X_batch, y_batch = next(training_generator)
  ...         run_optimization(X_batch, y_batch)
  epoch: 0, ...

.. _keras_generator:

Keras generator
~~~~~~~~~~~~~~~

Keras provides an higher level API in which a model can be defined and train by
calling ``fit_generator`` method to train the model. To illustrate, we will
define a logistic regression model::

  >>> from tensorflow import keras
  >>> y = keras.utils.to_categorical(y, 3)
  >>> model = keras.Sequential()
  >>> model.add(
  ...     keras.layers.Dense(
  ...         y.shape[1], input_dim=X.shape[1], activation='softmax'
  ...     )
  ... )
  >>> model.compile(
  ...     optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy']
  ... )

:func:`~imblearn.keras.balanced_batch_generator` creates a balanced
mini-batches generator with the associated number of mini-batches which will be
generated::

  >>> from imblearn.keras import balanced_batch_generator
  >>> training_generator, steps_per_epoch = balanced_batch_generator(
  ...     X, y, sampler=RandomUnderSampler(), batch_size=10, random_state=42
  ... )

Then, ``fit`` can be called passing the generator and the step::

  >>> callback_history = model.fit(
  ...     training_generator,
  ...     steps_per_epoch=steps_per_epoch,
  ...     epochs=10,
  ...     verbose=1,
  ... )
  Epoch 1/10 ...

The second possibility is to use
:class:`~imblearn.keras.BalancedBatchGenerator`. Only an instance of this class
will be passed to ``fit``::

  >>> from imblearn.keras import BalancedBatchGenerator
  >>> training_generator = BalancedBatchGenerator(
  ...     X, y, sampler=RandomUnderSampler(), batch_size=10, random_state=42
  ... )
  >>> callback_history = model.fit(
  ...     training_generator,
  ...     steps_per_epoch=steps_per_epoch,
  ...     epochs=10,
  ...     verbose=1,
  ... )
  Epoch 1/10 ...

.. topic:: References

  * :ref:`sphx_glr_auto_examples_applications_porto_seguro_keras_under_sampling.py`

---

### Doc/Model Selection

.. _cross_validation:

================
Cross validation
================

.. currentmodule:: imblearn.model_selection


.. _instance_hardness_threshold_cv:

The term instance hardness is used in literature to express the difficulty to correctly
classify an instance. An instance for which the predicted probability of the true class
is low, has large instance hardness. The way these hard-to-classify instances are
distributed over train and test sets in cross validation, has significant effect on the
test set performance metrics. The :class:`~imblearn.model_selection.InstanceHardnessCV`
splitter distributes samples with large instance hardness equally over the folds,
resulting in more robust cross validation.

We will discuss instance hardness in this document and explain how to use the
:class:`~imblearn.model_selection.InstanceHardnessCV` splitter.

Instance hardness and average precision
=======================================

Instance hardness is defined as 1 minus the probability of the most probable class:

.. math::

   H(x) = 1 - P(\hat{y}|x)

In this equation :math:`H(x)` is the instance hardness for a sample with features
:math:`x` and :math:`P(\hat{y}|x)` the probability of predicted label :math:`\hat{y}`
given the features. If the model predicts label 0 and gives a `predict_proba` output
of [0.9, 0.1], the probability of the most probable class (0) is 0.9 and the
instance hardness is `1-0.9=0.1`.

Samples with large instance hardness have significant effect on the area under
precision-recall curve, or average precision. Especially samples with label 0
with large instance hardness (so the model predicts label 1) reduce the average
precision a lot as these points affect the precision-recall curve in the left
where the area is largest; the precision is lowered in the range of low recall
and high thresholds. When doing cross validation, e.g. in case of hyperparameter
tuning or recursive feature elimination, random gathering of these points in
some folds introduce variance in CV results that deteriorates robustness of the
cross validation task. The :class:`~imblearn.model_selection.InstanceHardnessCV`
splitter aims to distribute the samples with large instance hardness over the
folds in order to reduce undesired variance. Note that one should use this
splitter to make model *selection* tasks robust like hyperparameter tuning and
feature selection but not for model *performance estimation* for which you also
want to know the variance of performance to be expected in production.


Create imbalanced dataset with samples with large instance hardness
===================================================================

Let's start by creating a dataset to work with. We create a dataset with 5% class
imbalance using scikit-learn's :func:`~sklearn.datasets.make_blobs` function.

  >>> import numpy as np
  >>> from matplotlib import pyplot as plt
  >>> from sklearn.datasets import make_blobs
  >>> from imblearn.datasets import make_imbalance
  >>> random_state = 10
  >>> X, y = make_blobs(n_samples=[950, 50], centers=((-3, 0), (3, 0)),
  ...                   random_state=random_state)
  >>> plt.scatter(X[:, 0], X[:, 1], c=y)
  >>> plt.show()

.. image:: ./auto_examples/model_selection/images/sphx_glr_plot_instance_hardness_cv_001.png
   :target: ./auto_examples/model_selection/plot_instance_hardness_cv.html
   :align: center

Now we add some samples with large instance hardness

  >>> X_hard, y_hard = make_blobs(n_samples=10, centers=((3, 0), (-3, 0)),
  ...                             cluster_std=1,
  ...                             random_state=random_state)
  >>> X = np.vstack((X, X_hard))
  >>> y = np.hstack((y, y_hard))
  >>> plt.scatter(X[:, 0], X[:, 1], c=y)
  >>> plt.show()

.. image:: ./auto_examples/model_selection/images/sphx_glr_plot_instance_hardness_cv_002.png
   :target: ./auto_examples/model_selection/plot_instance_hardness_cv.html
   :align: center

Assess cross validation performance variance using `InstanceHardnessCV` splitter
================================================================================

Then we take a :class:`~sklearn.linear_model.LogisticRegression` and assess the
cross validation performance using a :class:`~sklearn.model_selection.StratifiedKFold`
cv splitter and the :func:`~sklearn.model_selection.cross_validate` function.

  >>> from sklearn.ensemble import LogisticRegressionClassifier
  >>> clf = LogisticRegressionClassifier(random_state=random_state)
  >>> skf_cv = StratifiedKFold(n_splits=5, shuffle=True,
  ...                           random_state=random_state)
  >>> skf_result = cross_validate(clf, X, y, cv=skf_cv, scoring="average_precision")

Now, we do the same using an :class:`~imblearn.model_selection.InstanceHardnessCV`
splitter. We use provide our classifier to the splitter to calculate instance hardness
and distribute samples with large instance hardness equally over the folds.

  >>> ih_cv = InstanceHardnessCV(estimator=clf, n_splits=5,
  ...                               random_state=random_state)
  >>> ih_result = cross_validate(clf, X, y, cv=ih_cv, scoring="average_precision")

When we plot the test scores for both cv splitters, we see that the variance using the
:class:`~imblearn.model_selection.InstanceHardnessCV` splitter is lower than for the
:class:`~sklearn.model_selection.StratifiedKFold` splitter.

  >>> plt.boxplot([skf_result['test_score'], ih_result['test_score']],
  ...               tick_labels=["StratifiedKFold", "InstanceHardnessCV"],
  ...               vert=False)
  >>> plt.xlabel('Average precision')
  >>> plt.tight_layout()

.. image:: ./auto_examples/model_selection/images/sphx_glr_plot_instance_hardness_cv_003.png
   :target: ./auto_examples/model_selection/plot_instance_hardness_cv.html
   :align: center

Be aware that the most important part of cross-validation splitters is to simulate the
conditions that one will encounter in production. Therefore, if it is likely to get
difficult samples in production, one should use a cross-validation splitter that
emulates this situation. In our case, the
:class:`~sklearn.model_selection.StratifiedKFold` splitter did not allow to distribute
the difficult samples over the folds and thus it was likely a problem for our use case.

---

### Doc/Over Sampling

.. _over-sampling:

=============
Over-sampling
=============

.. currentmodule:: imblearn.over_sampling

A practical guide
=================

You can refer to
:ref:`sphx_glr_auto_examples_over-sampling_plot_comparison_over_sampling.py`.

.. _random_over_sampler:

Naive random over-sampling
--------------------------

One way to fight this issue is to generate new samples in the classes which are
under-represented. The most naive strategy is to generate new samples by
randomly sampling with replacement the current available samples. The
:class:`RandomOverSampler` offers such scheme::

   >>> from sklearn.datasets import make_classification
   >>> X, y = make_classification(n_samples=5000, n_features=2, n_informative=2,
   ...                            n_redundant=0, n_repeated=0, n_classes=3,
   ...                            n_clusters_per_class=1,
   ...                            weights=[0.01, 0.05, 0.94],
   ...                            class_sep=0.8, random_state=0)
   >>> from imblearn.over_sampling import RandomOverSampler
   >>> ros = RandomOverSampler(random_state=0)
   >>> X_resampled, y_resampled = ros.fit_resample(X, y)
   >>> from collections import Counter
   >>> print(sorted(Counter(y_resampled).items()))
   [(0, 4674), (1, 4674), (2, 4674)]

The augmented data set should be used instead of the original data set to train
a classifier::

  >>> from sklearn.linear_model import LogisticRegression
  >>> clf = LogisticRegression()
  >>> clf.fit(X_resampled, y_resampled)
  LogisticRegression(...)

In the figure below, we compare the decision functions of a classifier trained
using the over-sampled data set and the original data set.

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_002.png
   :target: ./auto_examples/over-sampling/plot_comparison_over_sampling.html
   :scale: 60
   :align: center

As a result, the majority class does not take over the other classes during the
training process. Consequently, all classes are represented by the decision
function.

In addition, :class:`RandomOverSampler` allows to sample heterogeneous data
(e.g. containing some strings)::

  >>> import numpy as np
  >>> X_hetero = np.array([['xxx', 1, 1.0], ['yyy', 2, 2.0], ['zzz', 3, 3.0]],
  ...                     dtype=object)
  >>> y_hetero = np.array([0, 0, 1])
  >>> X_resampled, y_resampled = ros.fit_resample(X_hetero, y_hetero)
  >>> print(X_resampled)
  [['xxx' 1 1.0]
   ['yyy' 2 2.0]
   ['zzz' 3 3.0]
   ['zzz' 3 3.0]]
  >>> print(y_resampled)
  [0 0 1 1]

It would also work with pandas dataframe::

  >>> from sklearn.datasets import fetch_openml
  >>> df_adult, y_adult = fetch_openml(
  ...     'adult', version=2, as_frame=True, return_X_y=True)
  >>> df_adult.head()  # doctest: +SKIP
  >>> df_resampled, y_resampled = ros.fit_resample(df_adult, y_adult)
  >>> df_resampled.head()  # doctest: +SKIP

If repeating samples is an issue, the parameter `shrinkage` allows to create a
smoothed bootstrap. However, the original data needs to be numerical. The
`shrinkage` parameter controls the dispersion of the new generated samples. We
show an example illustrate that the new samples are not overlapping anymore
once using a smoothed bootstrap. This ways of generating smoothed bootstrap is
also known a Random Over-Sampling Examples
(ROSE) :cite:`torelli2014rose`.

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_003.png
   :target: ./auto_examples/over-sampling/plot_comparison_over_sampling.html
   :scale: 60
   :align: center

.. _smote_adasyn:

From random over-sampling to SMOTE and ADASYN
---------------------------------------------

Apart from the random sampling with replacement, there are two popular methods
to over-sample minority classes: (i) the Synthetic Minority Oversampling
Technique (SMOTE) :cite:`chawla2002smote` and (ii) the Adaptive Synthetic
(ADASYN) :cite:`he2008adasyn` sampling method. These algorithms can be used in
the same manner::

  >>> from imblearn.over_sampling import SMOTE, ADASYN
  >>> X_resampled, y_resampled = SMOTE().fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 4674), (1, 4674), (2, 4674)]
  >>> clf_smote = LogisticRegression().fit(X_resampled, y_resampled)
  >>> X_resampled, y_resampled = ADASYN().fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 4673), (1, 4662), (2, 4674)]
  >>> clf_adasyn = LogisticRegression().fit(X_resampled, y_resampled)

The figure below illustrates the major difference of the different
over-sampling methods.

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_004.png
   :target: ./auto_examples/over-sampling/plot_comparison_over_sampling.html
   :scale: 60
   :align: center

Ill-posed examples
------------------

While the :class:`RandomOverSampler` is over-sampling by duplicating some of
the original samples of the minority class, :class:`SMOTE` and :class:`ADASYN`
generate new samples in by interpolation. However, the samples used to
interpolate/generate new synthetic samples differ. In fact, :class:`ADASYN`
focuses on generating samples next to the original samples which are wrongly
classified using a k-Nearest Neighbors classifier while the basic
implementation of :class:`SMOTE` will not make any distinction between easy and
hard samples to be classified using the nearest neighbors rule. Therefore, the
decision function found during training will be different among the algorithms.

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_005.png
   :target: ./auto_examples/over-sampling/plot_comparison_over_sampling.html
   :align: center

The sampling particularities of these two algorithms can lead to some peculiar
behavior as shown below.

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_006.png
   :target: ./auto_examples/over-sampling/plot_comparison_over_sampling.html
   :scale: 60
   :align: center

SMOTE variants
--------------

SMOTE might connect inliers and outliers while ADASYN might focus solely on
outliers which, in both cases, might lead to a sub-optimal decision
function. In this regard, SMOTE offers three additional options to generate
samples. Those methods focus on samples near the border of the optimal
decision function and will generate samples in the opposite direction of the
nearest neighbors class. Those variants are presented in the figure below.

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_007.png
   :target: ./auto_examples/over-sampling/plot_comparison_over_sampling.html
   :scale: 60
   :align: center


The :class:`BorderlineSMOTE` :cite:`han2005borderline`,
:class:`SVMSMOTE` :cite:`nguyen2009borderline`, and
:class:`KMeansSMOTE` :cite:`last2017oversampling` offer some variant of the
SMOTE algorithm::

  >>> from imblearn.over_sampling import BorderlineSMOTE
  >>> X_resampled, y_resampled = BorderlineSMOTE().fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 4674), (1, 4674), (2, 4674)]

When dealing with mixed data type such as continuous and categorical features,
none of the presented methods (apart of the class :class:`RandomOverSampler`)
can deal with the categorical features. The :class:`SMOTENC`
:cite:`chawla2002smote` is an extension of the :class:`SMOTE` algorithm for
which categorical data are treated differently::

  >>> # create a synthetic data set with continuous and categorical features
  >>> rng = np.random.RandomState(42)
  >>> n_samples = 50
  >>> X = np.empty((n_samples, 3), dtype=object)
  >>> X[:, 0] = rng.choice(['A', 'B', 'C'], size=n_samples).astype(object)
  >>> X[:, 1] = rng.randn(n_samples)
  >>> X[:, 2] = rng.randint(3, size=n_samples)
  >>> y = np.array([0] * 20 + [1] * 30)
  >>> print(sorted(Counter(y).items()))
  [(0, 20), (1, 30)]

In this data set, the first and last features are considered as categorical
features. One needs to provide this information to :class:`SMOTENC` via the
parameters ``categorical_features`` either by passing the indices, the feature
names when `X` is a pandas DataFrame, a boolean mask marking these features,
or relying on `dtype` inference if the columns are using the
:class:`pandas.CategoricalDtype`::

  >>> from imblearn.over_sampling import SMOTENC
  >>> smote_nc = SMOTENC(categorical_features=[0, 2], random_state=0)
  >>> X_resampled, y_resampled = smote_nc.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 30), (1, 30)]
  >>> print(X_resampled[-5:])
  [['A' 0.19... 2]
   ['B' -0.36... 2]
   ['B' 0.87... 2]
   ['B' 0.37... 2]
   ['B' 0.33... 2]]

Therefore, it can be seen that the samples generated in the first and last
columns are belonging to the same categories originally presented without any
other extra interpolation.

However, :class:`SMOTENC` is only working when data is a mixed of numerical and
categorical features. If data are made of only categorical data, one can use
the :class:`SMOTEN` variant :cite:`chawla2002smote`. The algorithm changes in
two ways:

* the nearest neighbors search does not rely on the Euclidean distance. Indeed,
  the value difference metric (VDM) also implemented in the class
  :class:`~imblearn.metrics.ValueDifferenceMetric` is used.
* a new sample is generated where each feature value corresponds to the most
  common category seen in the neighbors samples belonging to the same class.

Let's take the following example::

   >>> import numpy as np
   >>> X = np.array(["green"] * 5 + ["red"] * 10 + ["blue"] * 7,
   ...              dtype=object).reshape(-1, 1)
   >>> y = np.array(["apple"] * 5 + ["not apple"] * 3 + ["apple"] * 7 +
   ...              ["not apple"] * 5 + ["apple"] * 2, dtype=object)

We generate a dataset associating a color to being an apple or not an apple.
We strongly associated "green" and "red" to being an apple. The minority class
being "not apple", we expect new data generated belonging to the category
"blue"::

   >>> from imblearn.over_sampling import SMOTEN
   >>> sampler = SMOTEN(random_state=0)
   >>> X_res, y_res = sampler.fit_resample(X, y)
   >>> X_res[y.size:]
   array([['blue'],
           ['blue'],
           ['blue'],
           ['blue'],
           ['blue'],
           ['blue']], dtype=object)
   >>> y_res[y.size:]
   array(['not apple', 'not apple', 'not apple', 'not apple', 'not apple',
          'not apple'], dtype=object)

Mathematical formulation
========================

Sample generation
-----------------

Both :class:`SMOTE` and :class:`ADASYN` use the same algorithm to generate new
samples. Considering a sample :math:`x_i`, a new sample :math:`x_{new}` will be
generated considering its k neareast-neighbors (corresponding to
``k_neighbors``). For instance, the 3 nearest-neighbors are included in the
blue circle as illustrated in the figure below. Then, one of these
nearest-neighbors :math:`x_{zi}` is selected and a sample is generated as
follows:

.. math::

   x_{new} = x_i + \lambda \times (x_{zi} - x_i)

where :math:`\lambda` is a random number in the range :math:`[0, 1]`. This
interpolation will create a sample on the line between :math:`x_{i}` and
:math:`x_{zi}` as illustrated in the image below:

.. image:: ./auto_examples/over-sampling/images/sphx_glr_plot_illustration_generation_sample_001.png
   :target: ./auto_examples/over-sampling/plot_illustration_generation_sample.html
   :scale: 60
   :align: center

SMOTE-NC slightly change the way a new sample is generated by performing
something specific for the categorical features. In fact, the categories of a
new generated sample are decided by picking the most frequent category of the
nearest neighbors present during the generation.

.. warning::
   Be aware that SMOTE-NC is not designed to work with only categorical data.

The other SMOTE variants and ADASYN differ from each other by selecting the
samples :math:`x_i` ahead of generating the new samples.

The **regular** SMOTE algorithm --- cf. to the :class:`SMOTE` object --- does not
impose any rule and will randomly pick-up all possible :math:`x_i` available.

The **borderline** SMOTE --- cf. to the :class:`BorderlineSMOTE` with the
parameters ``kind='borderline-1'`` and ``kind='borderline-2'`` --- will
classify each sample :math:`x_i` to be (i) noise (i.e. all nearest-neighbors
are from a different class than the one of :math:`x_i`), (ii) in danger
(i.e. at least half of the nearest neighbors are from the same class than
:math:`x_i`, or (iii) safe (i.e. all nearest neighbors are from the same class
than :math:`x_i`). **Borderline-1** and **Borderline-2** SMOTE will use the
samples *in danger* to generate new samples. In **Borderline-1** SMOTE,
:math:`x_{zi}` will belong to the same class than the one of the sample
:math:`x_i`. On the contrary, **Borderline-2** SMOTE will consider
:math:`x_{zi}` which can be from any class.

**SVM** SMOTE --- cf. to :class:`SVMSMOTE` --- uses an SVM classifier to find
support vectors and generate samples considering them. Note that the ``C``
parameter of the SVM classifier allows to select more or less support vectors.

For both borderline and SVM SMOTE, a neighborhood is defined using the
parameter ``m_neighbors`` to decide if a sample is in danger, safe, or noise.

**KMeans** SMOTE --- cf. to :class:`KMeansSMOTE` --- uses a KMeans clustering
method before to apply SMOTE. The clustering will group samples together and
generate new samples depending of the cluster density.

ADASYN works similarly to the regular SMOTE. However, the number of
samples generated for each :math:`x_i` is proportional to the number of samples
which are not from the same class than :math:`x_i` in a given
neighborhood. Therefore, more samples will be generated in the area that the
nearest neighbor rule is not respected. The parameter ``m_neighbors`` is
equivalent to ``k_neighbors`` in :class:`SMOTE`.

Multi-class management
----------------------

All algorithms can be used with multiple classes as well as binary classes
classification.  :class:`RandomOverSampler` does not require any inter-class
information during the sample generation. Therefore, each targeted class is
resampled independently. In the contrary, both :class:`ADASYN` and
:class:`SMOTE` need information regarding the neighbourhood of each sample used
for sample generation. They are using a one-vs-rest approach by selecting each
targeted class and computing the necessary statistics against the rest of the
data set which are grouped in a single class.

---

### Doc/Under Sampling

.. _under-sampling:

==============
Under-sampling
==============

.. currentmodule:: imblearn.under_sampling

One way of handling imbalanced datasets is to reduce the number of observations from
all classes but the minority class. The minority class is that with the least number
of observations. The most well known algorithm in this group is random
undersampling, where samples from the targeted classes are removed at random.

But there are many other algorithms to help us reduce the number of observations in the
dataset. These algorithms can be grouped based on their undersampling strategy into:

- Prototype generation methods.
- Prototype selection methods.

And within the latter, we find:

- Controlled undersampling
- Cleaning methods

We will discuss the different algorithms throughout this document.

Check also
:ref:`sphx_glr_auto_examples_under-sampling_plot_comparison_under_sampling.py`.

.. _cluster_centroids:

Prototype generation
====================

Given an original data set :math:`S`, prototype generation algorithms will
generate a new set :math:`S'` where :math:`|S'| < |S|` and :math:`S' \not\subset
S`. In other words, prototype generation techniques will reduce the number of
samples in the targeted classes but the remaining samples are generated --- and
not selected --- from the original set.

:class:`ClusterCentroids` makes use of K-means to reduce the number of
samples. Therefore, each class will be synthesized with the centroids of the
K-means method instead of the original samples::

  >>> from collections import Counter
  >>> from sklearn.datasets import make_classification
  >>> X, y = make_classification(n_samples=5000, n_features=2, n_informative=2,
  ...                            n_redundant=0, n_repeated=0, n_classes=3,
  ...                            n_clusters_per_class=1,
  ...                            weights=[0.01, 0.05, 0.94],
  ...                            class_sep=0.8, random_state=0)
  >>> print(sorted(Counter(y).items()))
  [(0, 64), (1, 262), (2, 4674)]
  >>> from imblearn.under_sampling import ClusterCentroids
  >>> cc = ClusterCentroids(random_state=0)
  >>> X_resampled, y_resampled = cc.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 64), (2, 64)]

The figure below illustrates such under-sampling.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_comparison_under_sampling_001.png
   :target: ./auto_examples/under-sampling/plot_comparison_under_sampling.html
   :scale: 60
   :align: center

:class:`ClusterCentroids` offers an efficient way to represent the data cluster
with a reduced number of samples. Keep in mind that this method requires that
your data are grouped into clusters. In addition, the number of centroids
should be set such that the under-sampled clusters are representative of the
original one.

.. warning::

   :class:`ClusterCentroids` supports sparse matrices. However, the new samples
   generated are not specifically sparse. Therefore, even if the resulting
   matrix will be sparse, the algorithm will be inefficient in this regard.

Prototype selection
===================

Prototype selection algorithms will select samples from the original set :math:`S`,
generating a dataset :math:`S'`, where :math:`|S'| < |S|` and :math:`S' \subset S`. In
other words, :math:`S'` is a subset of :math:`S`.

Prototype selection algorithms can be divided into two groups: (i) controlled
under-sampling techniques and (ii) cleaning under-sampling techniques.

Controlled under-sampling methods reduce the number of observations in the majority
class or classes to an arbitrary number of samples specified by the user. Typically,
they reduce the number of observations to the number of samples observed in the
minority class.

In contrast, cleaning under-sampling techniques "clean" the feature space by removing
either "noisy" or "too easy to classify" observations, depending on the method. The
final number of observations in each class varies with the cleaning method and can't be
specified by the user.

.. _controlled_under_sampling:

Controlled under-sampling techniques
------------------------------------

Controlled under-sampling techniques reduce the number of observations from the
targeted classes to a number specified by the user.

Random under-sampling
^^^^^^^^^^^^^^^^^^^^^

:class:`RandomUnderSampler` is a fast and easy way to balance the data by
randomly selecting a subset of data for the targeted classes::

  >>> from imblearn.under_sampling import RandomUnderSampler
  >>> rus = RandomUnderSampler(random_state=0)
  >>> X_resampled, y_resampled = rus.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 64), (2, 64)]

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_comparison_under_sampling_002.png
   :target: ./auto_examples/under-sampling/plot_comparison_under_sampling.html
   :scale: 60
   :align: center

:class:`RandomUnderSampler` allows bootstrapping the data by setting
``replacement`` to ``True``. When there are multiple classes, each targeted class is
under-sampled independently::

  >>> import numpy as np
  >>> print(np.vstack([tuple(row) for row in X_resampled]).shape)
  (192, 2)
  >>> rus = RandomUnderSampler(random_state=0, replacement=True)
  >>> X_resampled, y_resampled = rus.fit_resample(X, y)
  >>> print(np.vstack(np.unique([tuple(row) for row in X_resampled], axis=0)).shape)
  (181, 2)

:class:`RandomUnderSampler` handles heterogeneous data types, i.e. numerical,
categorical, dates, etc.::

  >>> X_hetero = np.array([['xxx', 1, 1.0], ['yyy', 2, 2.0], ['zzz', 3, 3.0]],
  ...                     dtype=object)
  >>> y_hetero = np.array([0, 0, 1])
  >>> X_resampled, y_resampled = rus.fit_resample(X_hetero, y_hetero)
  >>> print(X_resampled)
  [['xxx' 1 1.0]
   ['zzz' 3 3.0]]
  >>> print(y_resampled)
  [0 1]

:class:`RandomUnderSampler` also supports pandas dataframes as input for
undersampling::

  >>> from sklearn.datasets import fetch_openml
  >>> df_adult, y_adult = fetch_openml(
  ...     'adult', version=2, as_frame=True, return_X_y=True)
  >>> df_adult.head()  # doctest: +SKIP
  >>> df_resampled, y_resampled = rus.fit_resample(df_adult, y_adult)
  >>> df_resampled.head()  # doctest: +SKIP

:class:`NearMiss` adds some heuristic rules to select samples
:cite:`mani2003knn`. :class:`NearMiss` implements 3 different types of
heuristic which can be selected with the parameter ``version``::

  >>> from imblearn.under_sampling import NearMiss
  >>> nm1 = NearMiss(version=1)
  >>> X_resampled_nm1, y_resampled = nm1.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 64), (2, 64)]

As later stated in the next section, :class:`NearMiss` heuristic rules are
based on nearest neighbors algorithm. Therefore, the parameters ``n_neighbors``
and ``n_neighbors_ver3`` accept classifier derived from ``KNeighborsMixin``
from scikit-learn. The former parameter is used to compute the average distance
to the neighbors while the latter is used for the pre-selection of the samples
of interest.

Mathematical formulation
^^^^^^^^^^^^^^^^^^^^^^^^

Let *positive samples* be the samples belonging to the targeted class to be
under-sampled. *Negative sample* refers to the samples from the minority class
(i.e., the most under-represented class).

NearMiss-1 selects the positive samples for which the average distance
to the :math:`N` closest samples of the negative class is the smallest.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_illustration_nearmiss_001.png
   :target: ./auto_examples/under-sampling/plot_illustration_nearmiss.html
   :scale: 60
   :align: center

NearMiss-2 selects the positive samples for which the average distance to the
:math:`N` farthest samples of the negative class is the smallest.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_illustration_nearmiss_002.png
   :target: ./auto_examples/under-sampling/plot_illustration_nearmiss.html
   :scale: 60
   :align: center

NearMiss-3 is a 2-steps algorithm. First, for each negative sample, their
:math:`M` nearest-neighbors will be kept. Then, the positive samples selected
are the one for which the average distance to the :math:`N` nearest-neighbors
is the largest.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_illustration_nearmiss_003.png
   :target: ./auto_examples/under-sampling/plot_illustration_nearmiss.html
   :scale: 60
   :align: center

In the next example, the different :class:`NearMiss` variant are applied on the
previous toy example. It can be seen that the decision functions obtained in
each case are different.

When under-sampling a specific class, NearMiss-1 can be altered by the presence
of noise. In fact, it will implied that samples of the targeted class will be
selected around these samples as it is the case in the illustration below for
the yellow class. However, in the normal case, samples next to the boundaries
will be selected. NearMiss-2 will not have this effect since it does not focus
on the nearest samples but rather on the farthest samples. We can imagine that
the presence of noise can also altered the sampling mainly in the presence of
marginal outliers. NearMiss-3 is probably the version which will be less
affected by noise due to the first step sample selection.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_comparison_under_sampling_003.png
   :target: ./auto_examples/under-sampling/plot_comparison_under_sampling.html
   :scale: 60
   :align: center

Cleaning under-sampling techniques
----------------------------------

Cleaning under-sampling methods "clean" the feature space by removing
either "noisy" observations or observations that are "too easy to classify", depending
on the method. The final number of observations in each targeted class varies with the
cleaning method and cannot be specified by the user.

.. _tomek_links:

Tomek's links
^^^^^^^^^^^^^

A Tomek's link exists when two samples from different classes are closest neighbors to
each other.

Mathematically, a Tomek's link between two samples from different classes :math:`x`
and :math:`y` is defined such that for any sample :math:`z`:

.. math::

   d(x, y) < d(x, z) \text{ and } d(x, y) < d(y, z)

where :math:`d(.)` is the distance between the two samples.

:class:`TomekLinks` detects and removes Tomek's links :cite:`tomek1976two`. The
underlying idea is that Tomek's links are noisy or hard to classify observations and
would not help the algorithm find a suitable discrimination boundary.

In the following figure, a Tomek's link between an observation of class :math:`+` and
class :math:`-` is highlighted in green:

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_illustration_tomek_links_001.png
   :target: ./auto_examples/under-sampling/plot_illustration_tomek_links.html
   :scale: 60
   :align: center

When :class:`TomekLinks` finds a Tomek's link, it can either remove the sample of the
majority class, or both. The parameter ``sampling_strategy`` controls which samples
from the link will be removed. By default (i.e., ``sampling_strategy='auto'``), it will
remove the sample from the majority class. Both samples, that is that from the majority
and the one from the minority class, can be removed by setting ``sampling_strategy`` to
``'all'``.

The following figure illustrates this behaviour: on the left, only the sample from the
majority class is removed, whereas on the right, the entire Tomek's link is removed.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_illustration_tomek_links_002.png
   :target: ./auto_examples/under-sampling/plot_illustration_tomek_links.html
   :scale: 60
   :align: center

.. _edited_nearest_neighbors:

Editing data using nearest neighbours
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Edited nearest neighbours
~~~~~~~~~~~~~~~~~~~~~~~~~

The edited nearest neighbours methodology uses K-Nearest Neighbours to identify the
neighbours of the targeted class samples, and then removes observations if any or most
of their neighbours are from a different class :cite:`wilson1972asymptotic`.

:class:`EditedNearestNeighbours` carries out the following steps:

1. Train a K-Nearest neighbours using the entire dataset.
2. Find each observations' K closest neighbours (only for the targeted classes).
3. Remove observations if any or most of its neighbours belong to a different class.

Below the code implementation::

  >>> sorted(Counter(y).items())
  [(0, 64), (1, 262), (2, 4674)]
  >>> from imblearn.under_sampling import EditedNearestNeighbours
  >>> enn = EditedNearestNeighbours()
  >>> X_resampled, y_resampled = enn.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 213), (2, 4568)]


To paraphrase step 3, :class:`EditedNearestNeighbours` will retain observations from
the majority class when **most**, or **all** of its neighbours are from the same class.
To control this behaviour we set ``kind_sel='mode'`` or ``kind_sel='all'``,
respectively. Hence, `kind_sel='all'` is less conservative than `kind_sel='mode'`,
resulting in the removal of more samples::

  >>> enn = EditedNearestNeighbours(kind_sel="all")
  >>> X_resampled, y_resampled = enn.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 213), (2, 4568)]
  >>> enn = EditedNearestNeighbours(kind_sel="mode")
  >>> X_resampled, y_resampled = enn.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 234), (2, 4666)]

The parameter ``n_neighbors`` accepts integers. The integer refers to the number of
neighbours to examine for each sample. It can also take a classifier subclassed from
``KNeighborsMixin`` from scikit-learn. When passing a classifier, note that, if you
pass a 3-Nearest Neighbors classifier, only 2 neighbours will be examined for the cleaning, as the
third sample is the one being examined for undersampling since it is part of the
samples provided at `fit`.

Repeated Edited Nearest Neighbours
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:class:`RepeatedEditedNearestNeighbours` extends
:class:`EditedNearestNeighbours` by repeating the algorithm multiple times
:cite:`tomek1976experiment`. Generally, repeating the algorithm will delete
more data::

   >>> from imblearn.under_sampling import RepeatedEditedNearestNeighbours
   >>> renn = RepeatedEditedNearestNeighbours()
   >>> X_resampled, y_resampled = renn.fit_resample(X, y)
   >>> print(sorted(Counter(y_resampled).items()))
   [(0, 64), (1, 208), (2, 4551)]

The user can set up the number of times the edited nearest neighbours method should be
repeated through the parameter `max_iter`.

The repetitions will stop when:

1. the maximum number of iterations is reached, or
2. no more observations are removed, or
3. one of the majority classes becomes a minority class, or
4. one of the majority classes disappears during the undersampling.

All KNN
~~~~~~~

:class:`AllKNN` is a variation of the
:class:`RepeatedEditedNearestNeighbours` where the number of neighbours evaluated at
each round of :class:`EditedNearestNeighbours` increases. It starts by editing based on
1-Nearest Neighbour, and it increases the neighbourhood by 1 at each iteration
:cite:`tomek1976experiment`::

  >>> from imblearn.under_sampling import AllKNN
  >>> allknn = AllKNN()
  >>> X_resampled, y_resampled = allknn.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 220), (2, 4601)]

:class:`AllKNN` stops cleaning when the maximum number of neighbours to examine, which
is determined by the user through the parameter `n_neighbors` is reached, or when the
majority class becomes the minority class.

In the example below, we see that :class:`EditedNearestNeighbours`,
:class:`RepeatedEditedNearestNeighbours` and :class:`AllKNN` have similar impact when
cleaning "noisy" samples at the boundaries between classes.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_comparison_under_sampling_004.png
   :target: ./auto_examples/under-sampling/plot_comparison_under_sampling.html
   :scale: 60
   :align: center

.. _condensed_nearest_neighbors:

Condensed nearest neighbors
^^^^^^^^^^^^^^^^^^^^^^^^^^^

:class:`CondensedNearestNeighbour` uses a 1 nearest neighbor rule to
iteratively decide if a sample should be removed
:cite:`hart1968condensed`. The algorithm runs as follows:

1. Get all minority samples in a set :math:`C`.
2. Add a sample from the targeted class (class to be under-sampled) in
   :math:`C` and all other samples of this class in a set :math:`S`.
3. Train a 1-Nearest Neigbhour on :math:`C`.
4. Go through the samples in set :math:`S`, sample by sample, and classify each one
   using a 1 nearest neighbor rule (trained in 3).
5. If the sample is misclassified, add it to :math:`C`, and go to step 6.
6. Repeat steps 3 to 5 until all observations in :math:`S` have been examined.

The final dataset is :math:`S`, containing all observations from the minority class and
those from the majority that were miss-classified by the successive
1-Nearest Neigbhour algorithms.

The :class:`CondensedNearestNeighbour` can be used in the following manner::

  >>> from imblearn.under_sampling import CondensedNearestNeighbour
  >>> cnn = CondensedNearestNeighbour(random_state=0)
  >>> X_resampled, y_resampled = cnn.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 24), (2, 115)]

:class:`CondensedNearestNeighbour` is sensitive to noise and may add noisy samples
(see figure later on).

One Sided Selection
~~~~~~~~~~~~~~~~~~~

In an attempt to remove the noisy observations introduced by
:class:`CondensedNearestNeighbour`, :class:`OneSidedSelection`
will first find the observations that are hard to classify, and then will use
:class:`TomekLinks` to remove noisy samples :cite:`hart1968condensed`.
:class:`OneSidedSelection` runs as follows:

1. Get all minority samples in a set :math:`C`.
2. Add a sample from the targeted class (class to be under-sampled) in
   :math:`C` and all other samples of this class in a set :math:`S`.
3. Train a 1-Nearest Neighbors on :math:`C`.
4. Using a 1 nearest neighbor rule trained in 3, classify all samples in
   set :math:`S`.
5. Add all misclassified samples to :math:`C`.
6. Remove Tomek Links from :math:`C`.

The final dataset is :math:`S`, containing all observations from the minority class,
plus the observations from the majority that were added at random, plus all
those from the majority that were miss-classified by the 1-Nearest Neighbors algorithms.

Note that differently from :class:`CondensedNearestNeighbour`, :class:`OneSidedSelection`
does not train a K-Nearest Neighbors after each sample is misclassified. It uses the
1-Nearest Neighbors from step 3 to classify all samples from the majority in 1 pass.
The class can be used as::

  >>> from imblearn.under_sampling import OneSidedSelection
  >>> oss = OneSidedSelection(random_state=0)
  >>> X_resampled, y_resampled = oss.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 174), (2, 4404)]

Our implementation offers the possibility to set the number of observations
to put at random in the set :math:`C` through the parameter ``n_seeds_S``.

:class:`NeighbourhoodCleaningRule` will focus on cleaning the data than
condensing them :cite:`laurikkala2001improving`. Therefore, it will used the
union of samples to be rejected between the :class:`EditedNearestNeighbours`
and the output a 3 nearest neighbors classifier. The class can be used as::

  >>> from imblearn.under_sampling import NeighbourhoodCleaningRule
  >>> ncr = NeighbourhoodCleaningRule(n_neighbors=11)
  >>> X_resampled, y_resampled = ncr.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 193), (2, 4535)]

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_comparison_under_sampling_005.png
   :target: ./auto_examples/under-sampling/plot_comparison_under_sampling.html
   :scale: 60
   :align: center

.. _instance_hardness_threshold:

Additional undersampling techniques
-----------------------------------

Instance hardness threshold
^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Instance Hardness** is a measure of how difficult it is to classify an instance or
observation correctly. In other words, hard instances are observations that are hard to
classify correctly.

Fundamentally, instances that are hard to classify correctly are those for which the
learning algorithm or classifier produces a low probability of predicting the correct
class label.

If we removed these hard instances from the dataset, the logic goes, we would help the
classifier better identify the different classes :cite:`smith2014instance`.

:class:`InstanceHardnessThreshold` trains a classifier on the data and then removes the
samples with lower probabilities :cite:`smith2014instance`. Or in other words, it
retains the observations with the higher class probabilities.

In our implementation, :class:`InstanceHardnessThreshold` is (almost) a controlled
under-sampling method: it will retain a specific number of observations of the target
class(es), which is specified by the user (see caveat below).

The class can be used as::

  >>> from sklearn.linear_model import LogisticRegression
  >>> from imblearn.under_sampling import InstanceHardnessThreshold
  >>> iht = InstanceHardnessThreshold(random_state=0,
  ...                                 estimator=LogisticRegression())
  >>> X_resampled, y_resampled = iht.fit_resample(X, y)
  >>> print(sorted(Counter(y_resampled).items()))
  [(0, 64), (1, 64), (2, 64)]

:class:`InstanceHardnessThreshold` has 2 important parameters. The parameter
``estimator`` accepts any scikit-learn classifier with a method ``predict_proba``.
This classifier will be used to identify the hard instances. The training is performed
with cross-validation which can be specified through the parameter ``cv`.

.. note::

   :class:`InstanceHardnessThreshold` could almost be considered as a
   controlled under-sampling method. However, due to the probability outputs, it
   is not always possible to get the specified number of samples.

The figure below shows examples of instance hardness undersampling on a toy dataset.

.. image:: ./auto_examples/under-sampling/images/sphx_glr_plot_comparison_under_sampling_006.png
   :target: ./auto_examples/under-sampling/plot_comparison_under_sampling.html
   :scale: 60
   :align: center

---

### Doc/User Guide

.. title:: User guide: contents

.. _user_guide:

==========
User Guide
==========

.. Ensure that the references will be alphabetically collected last
.. Check https://github.com/mcmtroffaes/sphinxcontrib-bibtex/issues/113

.. toctree::
   :numbered:

   introduction.rst
   over_sampling.rst
   under_sampling.rst
   combine.rst
   ensemble.rst
   miscellaneous.rst
   metrics.rst
   model_selection.rst
   common_pitfalls.rst
   Dataset loading utilities <datasets/index.rst>
   developers_utils.rst
   zzz_references.rst

---

### Doc/Whats New

.. currentmodule:: imblearn

===============
Release history
===============

.. include:: whats_new/v0.15.rst

.. include:: whats_new/v0.14.rst

.. include:: whats_new/v0.13.rst

.. include:: whats_new/v0.12.rst

.. include:: whats_new/v0.11.rst

.. include:: whats_new/v0.10.rst

.. include:: whats_new/v0.9.rst

.. include:: whats_new/v0.8.rst

.. include:: whats_new/v0.7.rst

.. include:: whats_new/v0.6.rst

.. include:: whats_new/v0.5.rst

.. include:: whats_new/v0.4.rst

.. include:: whats_new/v0.3.rst

.. include:: whats_new/v0.2.rst

.. include:: whats_new/v0.1.rst

---