### Doc/ Templates/Class
:mod:`{{module}}`.{{objname}}
{{ underline }}==============
.. currentmodule:: {{ module }}
.. autoclass:: {{ objname }}
---
### Doc/ Templates/Class Without Init
:mod:`{{module}}`.{{objname}}
{{ underline }}==============
.. currentmodule:: {{ module }}
.. autoclass:: {{ objname }}
.. include:: {{module}}.{{objname}}.examples
.. raw:: html
---
### Doc/ Templates/Function
:mod:`{{module}}`.{{objname}}
{{ underline }}====================
.. currentmodule:: {{ module }}
.. autofunction:: {{ objname }}
.. raw:: html
---
### Doc/Api
:orphan:
.. _api:
APIs
****
============
Main modules
============
~~~~~~~~~~~~~~
Classification
~~~~~~~~~~~~~~
.. autoclass:: autosklearn.classification.AutoSklearnClassifier
:members:
:inherited-members: show_models, fit_ensemble, refit, sprint_statistics
.. autoclass:: autosklearn.experimental.askl2.AutoSklearn2Classifier
:inherited-members: show_models, fit_ensemble, refit, sprint_statistics, fit, predict, predict_proba
~~~~~~~~~~
Regression
~~~~~~~~~~
.. autoclass:: autosklearn.regression.AutoSklearnRegressor
:members:
:inherited-members: show_models, fit_ensemble, refit, sprint_statistics
=======
Metrics
=======
.. autofunction:: autosklearn.metrics.make_scorer
~~~~~~~~~~~~~~~~
Built-in Metrics
~~~~~~~~~~~~~~~~
Classification metrics
~~~~~~~~~~~~~~~~~~~~~~
Note: The default ``autosklearn.metrics.f1``, ``autosklearn.metrics.precision`` and ``autosklearn.metrics.recall``
built-in metrics are applicable only for binary classification. In order to apply them on multilabel and multiclass
classification, please use the corresponding metrics with an appropriate averaging mechanism, such as ``autosklearn.metrics.f1_macro``.
For more information about how these metrics are used, please read
`this scikit-learn documentation `_.
.. autoclass:: autosklearn.metrics.accuracy
.. autoclass:: autosklearn.metrics.balanced_accuracy
.. autoclass:: autosklearn.metrics.f1
.. autoclass:: autosklearn.metrics.f1_macro
.. autoclass:: autosklearn.metrics.f1_micro
.. autoclass:: autosklearn.metrics.f1_samples
.. autoclass:: autosklearn.metrics.f1_weighted
.. autoclass:: autosklearn.metrics.roc_auc
.. autoclass:: autosklearn.metrics.precision
.. autoclass:: autosklearn.metrics.precision_macro
.. autoclass:: autosklearn.metrics.precision_micro
.. autoclass:: autosklearn.metrics.precision_samples
.. autoclass:: autosklearn.metrics.precision_weighted
.. autoclass:: autosklearn.metrics.average_precision
.. autoclass:: autosklearn.metrics.recall
.. autoclass:: autosklearn.metrics.recall_macro
.. autoclass:: autosklearn.metrics.recall_micro
.. autoclass:: autosklearn.metrics.recall_samples
.. autoclass:: autosklearn.metrics.recall_weighted
.. autoclass:: autosklearn.metrics.log_loss
Regression metrics
~~~~~~~~~~~~~~~~~~
.. autoclass:: autosklearn.metrics.r2
.. autoclass:: autosklearn.metrics.mean_squared_error
.. autoclass:: autosklearn.metrics.mean_absolute_error
.. autoclass:: autosklearn.metrics.median_absolute_error
====================
Extension Interfaces
====================
.. autoclass:: autosklearn.pipeline.components.base.AutoSklearnClassificationAlgorithm
:members:
.. autoclass:: autosklearn.pipeline.components.base.AutoSklearnRegressionAlgorithm
:members:
.. autoclass:: autosklearn.pipeline.components.base.AutoSklearnPreprocessingAlgorithm
:members:
.. _api_ensemble:
=========
Ensembles
=========
~~~~~~~~~~~~~~~~
Single objective
~~~~~~~~~~~~~~~~
.. autoclass:: autosklearn.ensembles.EnsembleSelection
:members:
Single model classes
~~~~~~~~~~~~~~~~~~~~
These classes wrap a single model to provide a unified interface in Auto-sklearn.
.. autoclass:: autosklearn.ensembles.SingleBest
:members:
.. autoclass:: autosklearn.ensembles.SingleModelEnsemble
:members:
.. autoclass:: autosklearn.ensembles.SingleBestFromRunhistory
:members:
~~~~~~~~~~~~~~~
Multi-objective
~~~~~~~~~~~~~~~
.. autoclass:: autosklearn.ensembles.MultiObjectiveDummyEnsemble
:members:
---
### Doc/Extending
:orphan:
.. _extending:
======================
Extending auto-sklearn
======================
auto-sklearn can be easily extended with new classification, regression and
feature preprocessing methods. In order to do so, a user has to implement a
wrapper class and register it to auto-sklearn. This manual will walk you
through the process.
Writing a component
===================
Depending on the purpose, the component has to be a subclass of one of the
following base classes:
* classification: :class:`autosklearn.pipeline.components.base.AutoSklearnClassificationAlgorithm`
* regression: :class:`autosklearn.pipeline.components.base.AutoSklearnRegressionAlgorithm`
* preprocessing: :class:`autosklearn.pipeline.components.base.AutoSklearnPreprocessingAlgorithm`
In general, these classes are wrappers around existing machine learning
models and only add the functionality auto-sklearn needs. Of course you can
also implement a machine learning algorithm directly inside a component.
Each component has to implement a method which returns its configuration
space, a method for querying properties of the component and methods like
`fit()`, `predict()` or `transform()` based on the task of the component.
These are described in the subsections
:ref:`get_hyperparameter_search_space` and :ref:`get_properties`
After writing a component class, you have to tell auto-sklearn about its
existence. You have to add it with the following function calls, depending on
the type of component:
.. autofunction:: autosklearn.pipeline.components.classification.add_classifier
.. autofunction:: autosklearn.pipeline.components.regression.add_regressor
.. autofunction:: autosklearn.pipeline.components.feature_preprocessing.add_preprocessor
.. _get_hyperparameter_search_space:
get_hyperparameter_search_space()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Return an instance of ``ConfigSpace.configuration_space.ConfigurationSpace``.
See also the abstract definitions:
:meth:`AutoSklearnClassificationAlgorithm.get_hyperparameter_search_space() `
:meth:`AutoSklearnRegressionAlgorithm.get_hyperparameter_search_space() `
:meth:`AutoSklearnPreprocessingAlgorithm.get_hyperparameter_search_space() `
To find out about how to create a ``ConfigurationSpace``-object, please look
at the source code on `github.com `_.
.. _get_properties:
get_properties()
~~~~~~~~~~~~~~~~
Return a dictionary which defines how the component can be used when
constructing a machine learning pipeline. The following fields must be
specified:
* shortname : str
an abbreviation of the component
* name : str
the full name of the component
* handles_regression : bool
whether the component can handle regression data
* handles_classification : bool
whether the component can handle classification data
* handles_multiclass : bool
whether the component can handle multiclass classification data
* handles_multilabel : bool
whether the component can multilabel classification data
* is_deterministic : bool
whether the component gives the same result when using several times,
but with the same random seed
* input : tuple
type of input data the component can handle, can have multiple values:
* **autosklearn.constants.DENSE**
dense data arrays, mutually exclusive with autosklearn.constants.SPARSE
* **autosklearn.constants.SPARSE**
sparse data matrices, mutually exclusive with autosklearn.constants.DENSE
* **autosklearn.constants.UNSIGNED_DATA**
unsigned data array, meaning only positive input, mutually exclusive
with autosklearn.constants.SIGNED_DATA
* **autosklearn.constants.SIGNED_DATA**
signed data array, meaning both positive and negative input values,
mutually exclusive with autosklearn.constants.UNSIGNED_DATA
* output : tuple
type of output data the component produces
* **autosklearn.constants.PREDICTIONS**
predictions, for example by a classifier
* **autosklearn.constants.INPUT**
data in the same form as the input
* **autosklearn.constants.DENSE**
dense data arrays, mutually exclusive with autosklearn.constants.SPARSE.
This implies that sparse data will be converted into a dense
representation.
* **autosklearn.constants.SPARSE**
sparse data matrices, mutually exclusive with
autosklearn.constants.DENSE. This implies that dense data will
be converted into a sparse representation
* **autosklearn.constants.UNSIGNED_DATA**
unsigned data array, meaning only positive input, mutually exclusive
with autosklearn.constants.SIGNED_DATA. This allows for algorithms which
can only work on positive data.
* **autosklearn.constants.SIGNED_DATA**
signed data array, meaning both positive and negative input values,
mutually exclusive with autosklearn.constants.UNSIGNED_DATA
Classification
==============
In addition two `get_properties()` and `get_hyperparameter_search_space()`
you have to implement
:meth:`AutoSklearnClassificationAlgorithm.fit() `
and
:meth:`AutoSklearnClassificationAlgorithm.predict() `
. These are an implementation of the `scikit-learn predictor API
`_.
Regression
==========
In addition two `get_properties()` and `get_hyperparameter_search_space()`
you have to implement
:meth:`AutoSklearnRegressionAlgorithm.fit() `
and
:meth:`AutoSklearnRegressionAlgorithm.predict() `
. These are an implementation of the `scikit-learn predictor API
`_.
Feature Preprocessing
=====================
In addition two `get_properties()` and `get_hyperparameter_search_space()`
you have to implement
:meth:`AutoSklearnPreprocessingAlgorithm.fit() `
and
:meth:`AutoSklearnPreprocessingAlgorithm.transform() `
. These are an implementation of the `scikit-learn predictor API
`_.
---
### Doc/Faq
:orphan:
.. _faq:
===
FAQ
===
General
=======
.. collapse:: Where can I find examples on how to use auto-sklearn?
We provide examples on using *auto-sklearn* for multiple use cases ranging from
simple classification to advanced uses such as feature importance, parallel runs
and customization. They can be found in the :ref:`examples`.
.. collapse:: What type of tasks can auto-sklearn tackle?
*auto-sklearn* can accept targets for the following tasks (more details on `Sklearn algorithms `_):
* Binary Classification
* Multiclass Classification
* Multilabel Classification
* Regression
* Multioutput Regression
You can provide feature and target training pairs (X_train/y_train) to *auto-sklearn* to fit an
ensemble of pipelines as described in the next section. This X_train/y_train dataset must belong
to one of the supported formats: np.ndarray, pd.DataFrame, scipy.sparse.csr_matrix and python lists.
Optionally, you can measure the ability of this fitted model to generalize to unseen data by
providing an optional testing pair (X_test/Y_test). For further details, please refer to the
Example :ref:`sphx_glr_examples_40_advanced_example_pandas_train_test.py`.
Regarding the features, there are multiple things to consider:
* Providing a X_train/X_test numpy array with the optional flag feat_type. For further details, you
can check the Example :ref:`sphx_glr_examples_40_advanced_example_feature_types.py`.
* You can provide a pandas DataFrame with properly formatted columns. If a column has numerical
dtype, *auto-sklearn* will not encode it and it will be passed directly to scikit-learn. *auto-sklearn*
supports both categorical or string as column type. Please ensure that you are using the correct
dtype for your task. By default *auto-sklearn* treats object and string columns as strings and
encodes the data using `sklearn.feature_extraction.text.CountVectorizer `_
* If your data contains categorical values (in the features or targets), ensure that you explicitly label them as categorical.
Data labeled as categorical is encoded by using a `sklearn.preprocessing.LabelEncoder `_
for unidimensional data and a `sklearn.preprodcessing.OrdinalEncoder `_ for multidimensional data.
* For further details on how to properly encode your data, you can check the Pandas Example
`Working with categorical data `_). If you are working with time series, it is recommended that you follow this approach
`Working with time data `_.
* If you prefer not using the string option at all you can disable this option. In this case
objects, strings and categorical columns are encoded as categorical.
.. code:: python
import autosklearn.classification
automl = autosklearn.classification.AutoSklearnClassifier(allow_string_features=False)
automl.fit(X_train, y_train)
Regarding the targets (y_train/y_test), if the task involves a classification problem, such features will be
automatically encoded. It is recommended to provide both y_train and y_test during fit, so that a common encoding
is created between these splits (if only y_train is provided during fit, the categorical encoder will not be able
to handle new classes that are exclusive to y_test). If the task is regression, no encoding happens on the
targets.
.. collapse:: Where can I find slides and notebooks from talks and tutorials?
We provide resources for talks, tutorials and presentations on *auto-sklearn* under `auto-sklearn-talks `_
.. collapse:: How should I cite auto-sklearn in a scientific publication?
If you've used auto-sklearn in scientific publications, we would appreciate citations.
.. code-block::
@inproceedings{feurer-neurips15a,
title = {Efficient and Robust Automated Machine Learning},
author = {Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina Springenberg, Jost and Blum, Manuel and Hutter, Frank},
booktitle = {Advances in Neural Information Processing Systems 28 (2015)},
pages = {2962--2970},
year = {2015}
}
Or this, if you've used auto-sklearn 2.0 in your work:
.. code-block::
@article{feurer-arxiv20a,
title = {Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning},
author = {Feurer, Matthias and Eggensperger, Katharina and Falkner, Stefan and Lindauer, Marius and Hutter, Frank},
booktitle = {arXiv:2007.04074 [cs.LG]},
year = {2020}
}
.. collapse:: I want to contribute. What can I do?
This sounds great. Please have a look at our `contribution guide `_
.. collapse:: I have a question which is not answered here. What should I do?
Thanks a lot. We regularly update this section with questions from our issue tracker. So please use the
`issue tracker `_
Resource Management
===================
.. collapse:: How should I set the time and memory limits?
While *auto-sklearn* alleviates manual hyperparameter tuning, the user still
has to set memory and time limits. For most datasets a memory limit of 3GB or
6GB as found on most modern computers is sufficient. For the time limits it
is harder to give clear guidelines. If possible, a good default is a total
time limit of one day, and a time limit of 30 minutes for a single run.
Further guidelines can be found in
`auto-sklearn/issues/142 `_.
.. collapse:: How many CPU cores does auto-sklearn use by default?
By default, *auto-sklearn* uses **one core**. See also :ref:`parallel` on how to configure this.
.. collapse:: How can I run auto-sklearn in parallel?
Nevertheless, *auto-sklearn* also supports parallel Bayesian optimization via the use of
`Dask.distributed `_. By providing the arguments ``n_jobs``
to the estimator construction, one can control the number of cores available to *auto-sklearn*
(As shown in the Example :ref:`sphx_glr_examples_60_search_example_parallel_n_jobs.py`).
Distributed processes are also supported by providing a custom client object to *auto-sklearn* like
in the Example: :ref:`sphx_glr_examples_60_search_example_parallel_manual_spawning_cli.py`. When
multiple cores are
available, *auto-sklearn* will create a worker per core, and use the available workers to both search
for better machine learning models as well as building an ensemble with them until the time resource
is exhausted.
**Note:** *auto-sklearn* requires all workers to have access to a shared file system for storing training data and models.
*auto-sklearn* employs `threadpoolctl `_ to control the number of threads employed by scientific libraries like numpy or scikit-learn. This is done exclusively during the building procedure of models, not during inference. In particular, *auto-sklearn* allows each pipeline to use at most 1 thread during training. At predicting and scoring time this limitation is not enforced by *auto-sklearn*. You can control the number of resources
employed by the pipelines by setting the following variables in your environment, prior to running *auto-sklearn*:
.. code-block:: shell-session
$ export OPENBLAS_NUM_THREADS=1
$ export MKL_NUM_THREADS=1
$ export OMP_NUM_THREADS=1
For further information about how scikit-learn handles multiprocessing, please check the `Parallelism, resource management, and configuration `_ documentation from the library.
.. collapse:: Auto-sklearn is extremely memory hungry in a sequential setting
Auto-sklearn can appear very memory hungry (i.e. requiring a lot of memory for small datasets) due
to the use of ``fork`` for creating new processes when running in sequential manner (if this
happens in a parallel setting or if you pass your own dask client this is due to a different
issue, see the other issues below).
Let's go into some more detail and discuss how to fix it:
Auto-sklearn executes each machine learning algorithm in its own process to be able to apply a
memory limit and a time limit. To start such a process, Python gives three options: ``fork``,
``forkserver`` and ``spawn``. The default ``fork`` copies the whole process memory into the
subprocess. If the main process already uses 1.5GB of main memory and we apply a 3GB memory
limit to Auto-sklearn, executing a machine learning pipeline is limited to use at most 1.5GB.
We would have loved to use ``forkserver`` or ``spawn`` as the default option instead, which both
copy only relevant data into the subprocess and thereby alleaviate the issue of eating up a lot
of your main memory
(and also do not suffer from potential deadlocks as ``fork`` does, see
`here `_),
but they have the downside that code must be guarded by ``if __name__ == "__main__"`` or executed
in a notebook, and we decided that we do not want to require this by default.
There are now two possible solutions:
1. Use Auto-sklearn in parallel: if you use Auto-sklean in parallel, it defaults to ``forkserver``
as the parallelization mechanism itself requires Auto-sklearn the code to be guarded. Please
find more information on how to do this in the following two examples:
1. :ref:`sphx_glr_examples_60_search_example_parallel_n_jobs.py`
2. :ref:`sphx_glr_examples_60_search_example_parallel_manual_spawning_cli.py`
.. note::
This requires all code to be guarded by ``if __name__ == "__main__"``.
2. Pass a `dask client `_. If the user passes
a dask client, Auto-sklearn can no longer assume that it runs in sequential mode and will use
a ``forkserver`` to start new processes.
.. note::
This requires all code to be guarded by ``if __name__ == "__main__"``.
We therefore suggest using one of the above settings by default.
.. collapse:: Auto-sklearn is extremely memory hungry in a parallel setting
When running Auto-sklearn in a parallel setting it starts new processes for evaluating machine
learning models using the ``forkserver`` mechanism. Code that is in the main script and that is
not guarded by ``if __name__ == "__main__"`` will be executed for each subprocess. If, for example,
you are loading your dataset outside of the guarded code, your dataset will be loaded for each
evaluation of a machine learning algorithm and thus blocking your RAM.
We therefore suggest moving all code inside functions or the main block.
.. collapse:: Auto-sklearn crashes with a segmentation fault
Please make sure that you have read and followed the :ref:`installation` section! In case
everything is set up correctly, this is most likely due to the dependency
`pyrfr `_ not being compiled correctly. If this is the
case please execute:
.. code:: python
import pyrfr.regression as reg
data = reg.default_data_container(64)
If this fails, the pyrfr dependency is most likely not compiled correctly. We advice you to do the
following:
1. Check if you can use a pre-compiled version of the pyrfr to avoid compiling it yourself. We
provide pre-compiled versions of the pyrfr on `pypi `_.
2. Check if the dependencies specified under :ref:`installation` are correctly installed,
especially that you have ``swig`` and a ``C++`` compiler.
3. If you are not yet using Conda, consider using it; it simplifies installation of the correct
dependencies.
4. Install correct build dependencies before installing the pyrfr, you can check the following
github issues for suggestions: `1025 `_,
`856 `_
Results, Log Files and Output
=============================
.. collapse:: How can I get an overview of the run statistics?
``sprint_statistics()`` is a method that prints the name of the dataset, the metric used, and the best validation score
obtained by running *auto-sklearn*. It additionally prints the number of both successful and unsuccessful
algorithm runs.
.. collapse:: What was the performance over time?
``performance_over_time_`` returns a DataFrame containing the models performance over time data, which can
be used for plotting directly (Here is an example: :ref:`sphx_glr_examples_40_advanced_example_pandas_train_test.py`).
.. code:: python
automl.performance_over_time_.plot(
x='Timestamp',
kind='line',
legend=True,
title='Auto-sklearn accuracy over time',
grid=True,
)
plt.show()
.. collapse:: Which models were evaluated?
You can see all models evaluated using :meth:`automl.leaderboard(ensemble_only=False) `.
.. collapse:: Which models are in the final ensemble?
Use either :meth:`automl.leaderboard(ensemble_only=True) ` or ``automl.show_models()``
.. collapse:: Is there more data I can look at?
``cv_results_`` returns a dict with keys as column headers and values as columns, that can be imported into
a pandas DataFrame, e.g. ``df = pd.DataFrame(automl.cv_results_)``
.. collapse:: Where does Auto-sklearn output files by default?
*Auto-sklearn* heavily uses the hard drive to store temporary data, models and log files which can
be used to inspect the behavior of Auto-sklearn. Each run of Auto-sklearn requires
its own directory. If not provided by the user, *Auto-sklearn* requests a temporary directory from
Python, which by default is located under ``/tmp`` and starts with ``autosklearn_tmp_`` followed
by a random string. By default, this directory is deleted when the *Auto-sklearn* object is
finished fitting. If you want to keep these files you can pass the argument
``delete_tmp_folder_after_terminate=True`` to the *Auto-sklearn* object.
The :class:`autosklearn.classification.AutoSklearnClassifier` and all other *auto-sklearn*
estimators accept the argument ``tmp_folder`` which change where such output is written to.
There's an additional argument ``output_directory`` which can be passed to *Auto-sklearn* and it
controls where test predictions of the ensemble are stored if the test set is passed to ``fit()``.
.. collapse:: Auto-sklearn's logfiles eat up all my disk space. What can I do?
*Auto-sklearn* heavily uses the hard drive to store temporary data, models and log files which can
be used to inspect the behavior of Auto-sklearn. By default, *Auto-sklearn* stores 50
models and their predictions on the validation data (which is a subset of the training data in
case of holdout and the full training data in case of cross-validation) on the hard drive.
Redundant models and their predictions (i.e. when we have more than 50 models) are removed
everytime the ensemble builder finishes an iteration, which means that the number of models stored
on disk can temporarily be higher if a model is output while the ensemble builder is running.
One can therefore change the number of models that will be stored on disk by passing an integer
for the argument ``max_models_on_disc`` to *Auto-sklearn*, for example reduce the number of models
stored on disk if you have space issues.
As the number of models is only an indicator of the disk space used it is also possible to pass
the memory in MB the models are allowed to use as a ``float`` (also via the ``max_models_on_disc``
arguments). As above, this is rather a guideline on how much memory is used as redundant models
are only removed from disk when the ensemble builder finishes an iteration.
.. note::
Especially when running in parallel it can happen that multiple models are constructed during
one run of the ensemble builder and thus *Auto-sklearn* can exceed the given limit.
.. note::
These limits do only apply to models and their predictions, but not to other files stored in
the temporary directory such as the log files.
The Search Space
================
.. collapse:: How can I restrict the searchspace?
The following shows an example of how to exclude all preprocessing methods and restrict the configuration space to
only random forests.
.. code:: python
import autosklearn.classification
automl = autosklearn.classification.AutoSklearnClassifier(
include = {
'classifier': ["random_forest"],
'feature_preprocessor': ["no_preprocessing"]
},
exclude=None
)
automl.fit(X_train, y_train)
predictions = automl.predict(X_test)
**Note:** The strings used to identify estimators and preprocessors are the filenames without *.py*.
For a full list please have a look at the source code (in `autosklearn/pipeline/components/`):
* `Classifiers `_
* `Regressors `_
* `Preprocessors `_
We do also provide an example on how to restrict the classifiers to search over
:ref:`sphx_glr_examples_40_advanced_example_interpretable_models.py`.
.. collapse:: How can I turn off data preprocessing?
Data preprocessing includes One-Hot encoding of categorical features, imputation
of missing values and the normalization of features or samples. These ensure that
the data the gets to the sklearn models is well formed and can be used for
training models.
While this is necessary in general, if you'd like to disable this step, please
refer to this :ref:`example `.
.. collapse:: How can I turn off feature preprocessing?
Feature preprocessing is a single transformer which implements for example feature
selection or transformation of features into a different space (i.e. PCA).
This can be turned off by setting
``include={'feature_preprocessor'=["no_preprocessing"]}`` as shown in the example above.
.. collapse:: Will non-scikit-learn models be added to Auto-sklearn?
The short answer: no.
The long answer answer is a bit more nuanced: maintaining Auto-sklearn requires a lot of time and
effort, which would grow even larger when depending on more libraries. Also, adding more
libraries would require us to generate meta-data more often. Lastly, having more choices does not
guarantee a better performance for most users as having more choices demands a longer search for
good models and can lead to more overfitting.
Nevertheless, everyone can still add their favorite model to Auto-sklearn's search space by
following the `examples on how to extend Auto-sklearn
`_.
If there is interest in creating a Auto-sklearn-contrib repository with 3rd-party models please
open an issue for that.
.. collapse:: How can I only search for interpretable models
Auto-sklearn can be restricted to only use interpretable models and preprocessing algorithms.
Please see the Section :ref:`space` to learn how to restrict the models
which are searched over or see the Example
:ref:`sphx_glr_examples_40_advanced_example_interpretable_models.py`.
We don't provide a judgement which of the models are interpretable as this is very much up to the
specific use case, but would like to note that decision trees and linear models usually most
interpretable.
Ensembling
==========
.. collapse:: What can I configure wrt the ensemble building process?
The following hyperparameters control how the ensemble is constructed:
* ``ensemble_class`` class object implementing :class:`autosklearn.ensembles.AbstractEnsemble`,
will be instantiated by *auto-sklearn*'s ensemble builder.
* ``ensemble_kwargs`` are keyword arguments that are passed to the ``ensemble_class`` upon
instantiation. See below for an example argument.
* ``ensemble_nbest`` allows the user to directly specify the number of models considered for the ensemble. This hyperparameter can be an integer *n*, such that only the best *n* models are used in the final ensemble. If a float between 0.0 and 1.0 is provided, ``ensemble_nbest`` would be interpreted as a fraction suggesting the percentage of models to use in the ensemble building process (namely, if ensemble_nbest is a float, library pruning is implemented as described in `Caruana et al. (2006) `_).
* ``max_models_on_disc`` defines the maximum number of models that are kept on the disc, as a mechanism to control the amount of disc space consumed by *auto-sklearn*. Throughout the automl process, different individual models are optimized, and their predictions (and other metadata) is stored on disc. The user can set the upper bound on how many models are acceptable to keep on disc, yet this variable takes priority in the definition of the number of models used by the ensemble builder (that is, the minimum of ``ensemble_size``, ``ensemble_nbest`` and ``max_models_on_disc`` determines the maximal amount of models used in the ensemble). If set to None, this feature is disabled.
The default method for Auto-sklearn is :class:`autosklearn.ensembles.EnsembleSelection`,
which features the argument ``ensemble_size``. that determines the maximal size of the
ensemble. Models can be added repeatedly, so the number of different models is usually
less than the ``ensemble_size``.
.. collapse:: Which models are in the final ensemble?
The results obtained from the final ensemble can be printed by calling ``show_models()`` or ``leaderboard()``.
The *auto-sklearn* ensemble is composed of scikit-learn models that can be inspected as exemplified
in the Example :ref:`sphx_glr_examples_40_advanced_example_get_pipeline_components.py`.
.. collapse:: Can I fit an ensemble also only post-hoc?
It is possible to build ensembles post-hoc. An example on how to do this (first searching for individual models, and then building an ensemble from them) can be seen in :ref:`sphx_glr_examples_60_search_example_sequential.py`.
Configuring the Search Procedure
================================
.. collapse:: Can I change the resampling strategy?
Examples for using holdout and cross-validation can be found in :ref:`example `
If using a custom resampling strategy with predefined splits, you may need to disable
the subsampling performed with particularly large datasets or if using a small ``memory_limit``.
Please see the manual section on :ref:`limits`
:class:`AutoSklearnClassifier(dataset_compression=...) `.
for more details.
.. collapse:: Can I use a custom metric
Examples for using a custom metric can be found in :ref:`example `
Meta-Learning
=============
.. collapse:: Which datasets are used for meta-learning?
We updated the list of datasets used for meta-learning several times and this list now differs
significantly from the original 140 datasets we used in 2015 when the paper and the package were
released. An up-to-date list of `OpenML task IDs `_ can be found
on `github `_.
.. collapse:: How can datasets from the meta-data be excluded?
For *Auto-sklearn 1.0* one can pass the dataset name via the ``fit()`` function. If a dataset
with the same name is within the meta-data, that datasets will not be used.
For *Auto-sklearn 2.0* it is not possible to do so because of the method used to construct the
meta-data.
.. collapse:: Which meta-features are used for meta-learning?
We do not have a user guide on meta-features but they are all pretty simple and can be found
`in the source code `_.
.. collapse:: How is the meta-data generated for Auto-sklearn 1.0?
We currently generate meta-data the following way. First, for each of the datasets mentioned
above, we run Auto-sklearn without meta-learning for a total of two days on multiple metrics (for
classification these are accuracy, balanced accuracy, log loss and the area under the curce).
Second, for each run we then have a look at each models that improved the score, i.e. the
trajectory of the best known model at a time, and refit it on the whole training data. Third, for
each of these models we then compute all scores we're interested in, these also include other
ones such F1 and precision. Finally, for each combination of dataset and metric we store the best
model we know of.
.. collapse:: How is the meta-data generated for Auto-sklearn 2.0?
Please check `our paper `_ for details.
Issues and Debugging
====================
.. collapse:: How can I limit the number of model evaluations for debugging?
In certain cases, for example for debugging, it can be helpful to limit the number of
model evaluations. We do not provide this as an argument in the API as we believe that it
should NOT be used in practice, but that the user should rather provide time limits.
An example on how to add the number of models to try as an additional stopping condition
can be found `in this github issue `_.
Please note that Auto-sklearn will stop when either the time limit or the number of
models termination condition is reached.
.. collapse:: Why does the final ensemble contains only a dummy model?
This is a symptom of the problem that all runs started by Auto-sklearn failed. Usually, the issue
is that the runtime or memory limit were too tight. Please check the output of
``sprint_statistics()`` to see the distribution of why runs failed. If there are mostly crashed
runs, please check the log file for further details. If there are mostly runs that exceed the
memory or time limit, please increase the respective limit and rerun the optimization.
.. collapse:: Auto-sklearn does not use the specified amount of resources?
Auto-sklearn wraps scikit-learn and therefore inherits its parallelism implementation. In short,
scikit-learn uses two modes of parallelizing computations:
1. By using joblib to distribute independent function calls on multiple cores.
2. By using lower level libraries such as OpenMP and numpy to distribute more fine-grained
computation.
This means that Auto-sklearn can use more resources than expected by the user. For technical
reasons we can only control the 1st way of parallel execution, but not the 2nd. Thus, the user
needs to make sure that the lower level parallelization libraries only use as many cores as
allocated (on a laptop or workstation running a single copy of Auto-sklearn it can be fine to not
adjust this, but when using a compute cluster it is necessary to align the parallelism setting
with the number of requested CPUs). This can be done by setting the following environment
variables: ``MKL_NUM_THREADS``, ``OPENBLAS_NUM_THREADS``, ``BLIS_NUM_THREADS`` and
``OMP_NUM_THREADS``.
More details can be found in the `scikit-learn docs `_.
Other
=====
.. collapse:: Model persistence
*auto-sklearn* is mostly a wrapper around scikit-learn. Therefore, it is
possible to follow the
`persistence Example `_
from scikit-learn.
.. collapse:: Vanilla auto-sklearn
In order to obtain *vanilla auto-sklearn* as used in `Efficient and Robust Automated Machine Learning
`_
set ``ensemble_class=autosklearn.ensembles.SingleBest`` and ``initial_configurations_via_metalearning=0``:
.. code:: python
import autosklearn.classification
import autosklearn.ensembles
automl = autosklearn.classification.AutoSklearnClassifier(
ensemble_class=autosklearn.ensembles.SingleBest,
initial_configurations_via_metalearning=0
)
This will always choose the best model according to the validation set.
Setting the initial configurations found by meta-learning to zero makes
*auto-sklearn* use the regular SMAC algorithm for suggesting new
hyperparameter configurations.
---
### Doc/Index
************
auto-sklearn
************
.. role:: bash(code)
:language: bash
.. role:: python(code)
:language: python
*auto-sklearn* is an automated machine learning toolkit and a drop-in
replacement for a scikit-learn estimator:
.. code:: python
import autosklearn.classification
cls = autosklearn.classification.AutoSklearnClassifier()
cls.fit(X_train, y_train)
predictions = cls.predict(X_test)
*auto-sklearn* frees a machine learning user from algorithm selection and
hyperparameter tuning. It leverages recent advantages in *Bayesian
optimization*, *meta-learning* and *ensemble construction*. Learn more about
the technology behind *auto-sklearn* by reading our paper published at
`NeurIPS 2015 `_
.
.. topic:: NEW: Text feature support
Auto-sklearn now supports text features, check our new example:
:ref:`sphx_glr_examples_40_advanced_example_text_preprocessing.py`
Example
*******
.. code:: python
import autosklearn.classification
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics
if __name__ == "__main__":
X, y = sklearn.datasets.load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = \
sklearn.model_selection.train_test_split(X, y, random_state=1)
automl = autosklearn.classification.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, y_hat))
This will run for one hour and should result in an accuracy above 0.98.
Manual
******
* :ref:`installation`
* :ref:`manual`
* :ref:`api`
* :ref:`extending`
* :ref:`faq`
Additional Material
*******************
We provide slides and notebooks from talks and tutorials here:
`auto-sklearn-talks `_
License
*******
*auto-sklearn* is licensed the same way as *scikit-learn*,
namely the 3-clause BSD license.
Citing auto-sklearn
*******************
If you use auto-sklearn in a scientific publication, we would appreciate a
reference to the following paper:
`Efficient and Robust Automated Machine Learning
`_,
Feurer *et al.*, Advances in Neural Information Processing Systems 28 (NIPS 2015).
Bibtex entry::
@inproceedings{feurer-neurips15a,
title = {Efficient and Robust Automated Machine Learning},
author = {Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina and Springenberg, Jost and Blum, Manuel and Hutter, Frank},
booktitle = {Advances in Neural Information Processing Systems 28 (2015)},
pages = {2962--2970},
year = {2015}
}
If you are using Auto-sklearn 2.0, please also cite
`Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning `_, Feurer *et al.*, (arXiv, 2020).
Bibtex entry::
@article{feurer-arxiv20a,
title = {Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning},
author = {Feurer, Matthias and Eggensperger, Katharina and Falkner, Stefan and Lindauer, Marius and Hutter, Frank},
journal = {arXiv:2007.04074 [cs.LG]},
year = {2020},
}
Contributing
************
We appreciate all contribution to auto-sklearn, from bug reports and
documentation to new features. If you want to contribute to the code, you can
pick an issue from the `issue tracker `_.
Check out our `contribution guide on github `_ if you want to know more!
We've catered it for both new and experienced contributers.
.. note::
To avoid spending time on duplicate work or features that are unlikely to
get merged, it is highly advised that you contact the developers
by opening a `github issue `_ before starting to work.
---
### Doc/Installation
:orphan:
.. _installation:
============
Installation
============
System requirements
===================
auto-sklearn has the following system requirements:
* Linux operating system (for example Ubuntu) (`get Linux here `_)
* Python (>=3.7) (`get Python here `_),
* C++ compiler (with C++11 supports) (`get GCC here `_).
In case you try to install Auto-sklearn on a system where no wheel files for the pyrfr package
are provided (see `here `_ for available wheels) you also
need:
* SWIG (`get SWIG here `_).
For an explanation of missing Microsoft Windows and macOS support please
check the Section `Windows/macOS compatibility`_.
Installing auto-sklearn
=======================
You can install *auto-sklearn* with `pip` in the usual manner:
.. code:: bash
pip3 install auto-sklearn
We recommend installing *auto-sklearn* into a
`virtual environment `_
or an
`Anaconda environment `_.
If the ``pip3`` installation command fails, make sure you have the `System requirements`_ installed correctly.
Ubuntu installation
===================
To provide Python 3, a C++11 building environment and the latest SWIG version on Ubuntu,
run:
.. code:: bash
sudo apt-get install build-essential swig python3-dev
Anaconda installation
=====================
You need to enable conda-forge to install *auto-sklearn* via anaconda. This section explains how to enable conda-forge so
installation can be done with the command `conda install auto-sklearn`.
Optionally, you can also install *auto-sklearn* with `pip` as detailed in the Section `Installing auto-sklearn`_.
A common installation problem under recent Linux distribution is the
incompatibility of the compiler version used to compile the Python binary
shipped by AnaConda and the compiler installed by the distribution. This can
be solved by installing the *gcc* compiler shipped with AnaConda (as well as
*swig*):
.. code:: bash
conda install gxx_linux-64 gcc_linux-64 swig
Conda-forge
~~~~~~~~~~~
Installing `auto-sklearn` from the `conda-forge` channel can be achieved by adding `conda-forge` to your channels with:
.. code:: bash
conda config --add channels conda-forge
conda config --set channel_priority strict
You must have `conda >=4.9`. To update conda or check your current conda version, please follow the instructions from `the official anaconda documentation `_ . Once the `conda-forge` channel has been enabled, `auto-sklearn` can be installed with:
.. code:: bash
conda install auto-sklearn
It is possible to list all of the versions of `auto-sklearn` available on your platform with:
.. code:: bash
conda search auto-sklearn --channel conda-forge
to read in more details check
`auto sklearn feedstock `_.
for more information about Conda forge check
`conda-forge documentations `_.
Source Installation
===================
You can install auto-sklearn directly form source by following the below:
.. code:: bash
git clone --recurse-submodules git@github.com:automl/auto-sklearn.git
cd auto-sklearn
# Install it in editable mode with all optional dependencies
pip install -e ".[test,doc,examples]"
We use submodules so you will have to make sure the submodule is initialized if you
missed the `--recurse-submodules` option.
.. code:: bash
git clone git@github.com:automl/auto-sklearn.git
cd auto-sklearn
git submodule update --init --recursive
pip install -e ".[test,doc,examples]"
Windows/macOS compatibility
===========================
Windows
~~~~~~~
*auto-sklearn* relies heavily on the Python module ``resource``. ``resource``
is part of Python's `Unix Specific Services `_
and not available on a Windows machine. Therefore, it is not possible to run
*auto-sklearn* on a Windows machine.
Possible solutions:
* Windows 10 bash shell (see `431 `_ and
`860 `_ for suggestions)
* virtual machine
* docker image
macOS
~~~~~
We currently do not know if *auto-sklearn* works on macOS. There are at least two
issues holding us back from actively supporting macOS:
* The ``resource`` module cannot enforce a memory limit on a Python process
(see `SMAC3/issues/115 `_).
* Not all dependencies we are using are set up to work on macOS.
In case you're having issues installing the `pyrfr package `_, check out
`this installation suggestion on github `_.
Possible other:
* virtual machine
* docker image
Docker Image
============
A Docker image is also provided on dockerhub. To download from dockerhub,
use:
.. code:: bash
docker pull mfeurer/auto-sklearn:master
You can also verify that the image was downloaded via:
.. code:: bash
docker images # Verify that the image was downloaded
This image can be used to start an interactive session as follows:
.. code:: bash
docker run -it mfeurer/auto-sklearn:master
To start a Jupyter notebook, you could instead run e.g.:
.. code:: bash
docker run -it -v ${PWD}:/opt/nb -p 8888:8888 mfeurer/auto-sklearn:master /bin/bash -c "mkdir -p /opt/nb && jupyter notebook --notebook-dir=/opt/nb --ip='0.0.0.0' --port=8888 --no-browser --allow-root"
Alternatively, it is possible to use the development version of auto-sklearn by replacing all
occurences of ``master`` by ``development``.
---
### Doc/Manual
:orphan:
.. _manual:
======
Manual
======
This manual gives an overview of different aspects of *auto-sklearn*. For each section, we either references examples or
give short explanations (click the title to expand text), e.g.
.. collapse:: Code examples
We provide examples on using *auto-sklearn* for multiple use cases ranging from
simple classification to advanced uses such as feature importance, parallel runs
and customization. They can be found in the :ref:`examples`.
.. collapse:: Material from talks and presentations
We provide resources for talks, tutorials and presentations on *auto-sklearn* under `auto-sklearn-talks `_
.. _askl2:
Auto-sklearn 2.0
================
Auto-sklearn 2.0 includes latest research on automatically configuring the AutoML system itself
and contains a multitude of improvements which speed up the fitting the AutoML system.
Concretely, Auto-sklearn 2.0 automatically sets the :ref:`bestmodel`, decides whether it can use
the efficient bandit strategy *Successive Halving* and uses meta-feature free *Portfolios* for
efficient meta-learning.
*auto-sklearn 2.0* has the same interface as regular *auto-sklearn* and you can use it via
.. code:: python
from autosklearn.experimental.askl2 import AutoSklearn2Classifier
A paper describing our advances is available on `arXiv `_.
.. _limits:
Resource limits
===============
A crucial feature of *auto-sklearn* is limiting the resources (memory and time) which the scikit-learn algorithms are
allowed to use. Especially for large datasets, on which algorithms can take several hours and make the machine swap,
it is important to stop the evaluations after some time in order to make progress in a reasonable amount of time.
Setting the resource limits is therefore a tradeoff between optimization time and the number of models that can be
tested.
.. collapse:: Time and memory limits
While *auto-sklearn* alleviates manual hyperparameter tuning, the user still
has to set memory and time limits. For most datasets a memory limit of 3GB or
6GB as found on most modern computers is sufficient. For the time limits it
is harder to give clear guidelines. If possible, a good default is a total
time limit of one day, and a time limit of 30 minutes for a single run.
Further guidelines can be found in
`auto-sklearn/issues/142 `_.
.. collapse:: CPU cores
By default, *auto-sklearn* uses **one core**. See also :ref:`parallel` on how to configure this.
.. collapse:: Managing data compression
.. _manual_managing_data_compression:
Auto-sklearn will attempt to fit the dataset into 1/10th of the ``memory_limit``.
This won't happen unless your dataset is quite large or you have small a
``memory_limit``. This is done using two methods, reducing **precision** and
to **subsample**. One reason you may want to control this is if you require high
precision or you rely on predefined splits for which subsampling does not account
for.
To turn off data preprocessing:
.. code:: python
AutoSklearnClassifier(
dataset_compression = False
)
You can specify which of the methods are performed using:
.. code:: python
AutoSklearnClassifier(
dataset_compression = { "methods": ["precision", "subsample"] },
)
You can change the memory allocation for the dataset to a percentage of ``memory_limit``
or an absolute amount using:
.. code:: python
AutoSklearnClassifier(
dataset_compression = { "memory_allocation": 0.2 },
)
The default arguments are used when ``dataset_compression = True`` are:
.. code:: python
{
"memory_allocation": 0.1,
"methods": ["precision", "subsample"]
}
The full description is given at :class:`AutoSklearnClassifier(dataset_compression=...) `.
.. _space:
The search space
================
*Auto-sklearn* by default searches a large space to find a well performing configuration. However, it is also possible
to restrict the searchspace:
.. collapse:: Restricting the searchspace
The following shows an example of how to exclude all preprocessing methods and restrict the configuration space to
only random forests.
.. code:: python
import autosklearn.classification
automl = autosklearn.classification.AutoSklearnClassifier(
include = {
'classifier': ["random_forest"],
'feature_preprocessor': ["no_preprocessing"]
},
exclude=None
)
automl.fit(X_train, y_train)
predictions = automl.predict(X_test)
**Note:** The strings used to identify estimators and preprocessors are the filenames without *.py*.
For a full list please have a look at the source code (in `autosklearn/pipeline/components/`):
* `Classifiers `_
* `Regressors `_
* `Preprocessors `_
We do also provide an example on how to restrict the classifiers to search over
:ref:`sphx_glr_examples_40_advanced_example_interpretable_models.py`.
.. collapse:: Turn off data preprocessing
Data preprocessing includes One-Hot encoding of categorical features, imputation
of missing values and the normalization of features or samples. These ensure that
the data the gets to the sklearn models is well formed and can be used for
training models.
While this is necessary in general, if you'd like to disable this step, please
refer to this :ref:`example `.
.. collapse:: Turn off feature preprocessing
Feature preprocessing is a single transformer which implements for example feature
selection or transformation of features into a different space (i.e. PCA).
This can be turned off by setting
``include={'feature_preprocessor'=["no_preprocessing"]}`` as shown in the example above.
.. _bestmodel:
Model selection
===============
*Auto-sklearn* implements different strategies to identify the best performing model. For some use cases it might be
necessary to adapt the resampling strategy or define a custom metric:
.. collapse:: Use different resampling strategies
Examples for using holdout and cross-validation can be found in :ref:`example `
.. collapse:: Use a custom metric
Examples for using a custom metric can be found in :ref:`example `
.. _ensembles:
Ensembling
==========
To get the best performance out of the evaluated models, *auto-sklearn* uses ensemble selection by `Caruana et al. (2004) `_
to build an ensemble based on the models’ prediction for the validation set.
.. collapse:: Configure the ensemble building process
The following hyperparameters control how the ensemble is constructed:
* ``ensemble_size`` determines the maximal size of the ensemble. If it is set to zero, no ensemble will be constructed.
* ``ensemble_nbest`` allows the user to directly specify the number of models considered for the ensemble. This hyperparameter can be an integer *n*, such that only the best *n* models are used in the final ensemble. If a float between 0.0 and 1.0 is provided, ``ensemble_nbest`` would be interpreted as a fraction suggesting the percentage of models to use in the ensemble building process (namely, if ensemble_nbest is a float, library pruning is implemented as described in `Caruana et al. (2006) `_).
* ``max_models_on_disc`` defines the maximum number of models that are kept on the disc, as a mechanism to control the amount of disc space consumed by *auto-sklearn*. Throughout the automl process, different individual models are optimized, and their predictions (and other metadata) is stored on disc. The user can set the upper bound on how many models are acceptable to keep on disc, yet this variable takes priority in the definition of the number of models used by the ensemble builder (that is, the minimum of ``ensemble_size``, ``ensemble_nbest`` and ``max_models_on_disc`` determines the maximal amount of models used in the ensemble). If set to None, this feature is disabled.
.. collapse:: Inspect the final ensemble
The results obtained from the final ensemble can be printed by calling ``show_models()``.
The *auto-sklearn* ensemble is composed of scikit-learn models that can be inspected as exemplified
in the Example :ref:`sphx_glr_examples_40_advanced_example_get_pipeline_components.py`.
.. collapse:: Fit ensemble post-hoc
To use a single core only, it is possible to build ensembles post-hoc. An example on how to do this (first searching
for individual models, and then building an ensemble from them) can be seen in
:ref:`sphx_glr_examples_60_search_example_sequential.py`.
.. _inspect:
Inspecting the results
======================
*auto-sklearn* allows users to inspect the training results and statistics. Assume we have a fitted estimator:
.. code:: python
import autosklearn.classification
automl = autosklearn.classification.AutoSklearnClassifier()
automl.fit(X_train, y_train)
*auto-sklearn* offers the following ways to inspect the results
.. collapse:: Basic statistics
``sprint_statistics()`` is a method that prints the name of the dataset, the metric used, and the best validation score
obtained by running *auto-sklearn*. It additionally prints the number of both successful and unsuccessful
algorithm runs.
.. collapse:: Performance over Time
``performance_over_time_`` returns a DataFrame containing the models performance over time data, which can
be used for plotting directly (Here is an example: :ref:`sphx_glr_examples_40_advanced_example_pandas_train_test.py`).
.. code:: python
automl.performance_over_time_.plot(
x='Timestamp',
kind='line',
legend=True,
title='Auto-sklearn accuracy over time',
grid=True,
)
plt.show()
.. collapse:: Evaluated models
The results obtained from the final ensemble can be printed by calling ``show_models()``.
.. collapse:: Leaderboard
``automl.leaderboard()`` shows the ensemble members, check the :meth:`docs ` for using leaderboard for getting information on *all* runs.
.. collapse:: Other
``cv_results_`` returns a dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame.
.. _parallel:
Parallel computation
====================
In it's default mode, *auto-sklearn* uses **one core** and interleaves ensemble building with evaluating new
configurations.
.. collapse:: Parallelization with Dask
Nevertheless, *auto-sklearn* also supports parallel Bayesian optimization via the use of
`Dask.distributed `_. By providing the arguments ``n_jobs``
to the estimator construction, one can control the number of cores available to *auto-sklearn*
(As shown in the Example :ref:`sphx_glr_examples_60_search_example_parallel_n_jobs.py`).
Distributed processes are also supported by providing a custom client object to *auto-sklearn* like
in the Example: :ref:`sphx_glr_examples_60_search_example_parallel_manual_spawning_cli.py`. When
multiple cores are
available, *auto-sklearn* will create a worker per core, and use the available workers to both search
for better machine learning models as well as building an ensemble with them until the time resource
is exhausted.
**Note:** *auto-sklearn* requires all workers to have access to a shared file system for storing training data and models.
*auto-sklearn* employs `threadpoolctl `_ to control the number of threads employed by scientific libraries like numpy or scikit-learn. This is done exclusively during the building procedure of models, not during inference. In particular, *auto-sklearn* allows each pipeline to use at most 1 thread during training. At predicting and scoring time this limitation is not enforced by *auto-sklearn*. You can control the number of resources
employed by the pipelines by setting the following variables in your environment, prior to running *auto-sklearn*:
.. code-block:: shell-session
$ export OPENBLAS_NUM_THREADS=1
$ export MKL_NUM_THREADS=1
$ export OMP_NUM_THREADS=1
For further information about how scikit-learn handles multiprocessing, please check the `Parallelism, resource management, and configuration `_ documentation from the library.
.. _othermanual:
Other
=====
.. collapse:: Supported input types
*auto-sklearn* can accept targets for the following tasks (more details on `Sklearn algorithms `_):
* Binary Classification
* Multiclass Classification
* Multilabel Classification
* Regression
* Multioutput Regression
You can provide feature and target training pairs (X_train/y_train) to *auto-sklearn* to fit an
ensemble of pipelines as described in the next section. This X_train/y_train dataset must belong
to one of the supported formats: np.ndarray, pd.DataFrame, scipy.sparse.csr_matrix and python lists.
Optionally, you can measure the ability of this fitted model to generalize to unseen data by
providing an optional testing pair (X_test/Y_test). For further details, please refer to the
Example :ref:`sphx_glr_examples_40_advanced_example_pandas_train_test.py`.
Regarding the features, there are multiple things to consider:
* Providing a X_train/X_test numpy array with the optional flag feat_type. For further details, you
can check the Example :ref:`sphx_glr_examples_40_advanced_example_feature_types.py`.
* You can provide a pandas DataFrame with properly formatted columns. If a column has numerical
dtype, *auto-sklearn* will not encode it and it will be passed directly to scikit-learn. *auto-sklearn*
supports both categorical or string as column type. Please ensure that you are using the correct
dtype for your task. By default *auto-sklearn* treats object and string columns as strings and
encodes the data using `sklearn.feature_extraction.text.CountVectorizer `_
* If your data contains categorical values (in the features or targets), ensure that you explicitly label them as categorical.
Data labeled as categorical is encoded by using a `sklearn.preprocessing.LabelEncoder `_
for unidimensional data and a `sklearn.preprodcessing.OrdinalEncoder `_ for multidimensional data.
* For further details on how to properly encode your data, you can check the Pandas Example
`Working with categorical data `_). If you are working with time series, it is recommended that you follow this approach
`Working with time data `_.
* If you prefer not using the string option at all you can disable this option. In this case
objects, strings and categorical columns are encoded as categorical.
.. code:: python
import autosklearn.classification
automl = autosklearn.classification.AutoSklearnClassifier(allow_string_features=False)
automl.fit(X_train, y_train)
Regarding the targets (y_train/y_test), if the task involves a classification problem, such features will be
automatically encoded. It is recommended to provide both y_train and y_test during fit, so that a common encoding
is created between these splits (if only y_train is provided during fit, the categorical encoder will not be able
to handle new classes that are exclusive to y_test). If the task is regression, no encoding happens on the
targets.
.. collapse:: Model persistence
*auto-sklearn* is mostly a wrapper around scikit-learn. Therefore, it is
possible to follow the
`persistence Example `_
from scikit-learn.
.. collapse:: Vanilla auto-sklearn
In order to obtain *vanilla auto-sklearn* as used in `Efficient and Robust Automated Machine Learning
`_
set ``ensemble_size=1``, ``initial_configurations_via_metalearning=0`` and ``allow_string_features=False``:
.. code:: python
import autosklearn.classification
automl = autosklearn.classification.AutoSklearnClassifier(
ensemble_size=1,
initial_configurations_via_metalearning=0,
allow_string_features=False,
)
An ensemble of size one will result in always choosing the current best model
according to its performance on the validation set. Setting the initial
configurations found by meta-learning to zero makes *auto-sklearn* use the
regular SMAC algorithm for suggesting new hyperparameter configurations.
.. collapse:: Early stopping and Callbacks
By using the parameter ``get_trials_callback``, we can get access to the results
of runs as they occur. See this example :ref:`Early Stopping And Callbacks ` for more!
---
### Doc/Releases
:orphan:
..
The following command allows to retrieve all commiters since a specified
commit. From https://stackoverflow.com/questions/6482436/list-of-authors-in-git-since-a-given-commit
git log 6cc8bb179fcb023d1c341cf33d2958a16a6935be.. --format="%aN <%aE>" --reverse | perl -e 'my %dedupe; while () { print unless $dedupe{$_}++}'
========
Releases
========
Version 0.15.0
==============
* ADD #1317, #1455, #1485, #1501, #1518, #1523: Initial support for multi-objective Auto-sklearn.
* ADD #1300, #1410, #1414, #1415, #1420, #1468, #1500: Intial support for text features Auto-sklearn. You can now pass in columns identified as `"string"` columns which will be tokenized using pure sklearn methods.
* ADD #1475: Support for passing `X` data to metrics, as required by [`fairlearn`](https://github.com/fairlearn/fairlearn) metrics
* ADD #1341, #1250: Expose interface to interact with how auto-sklearn performs dataset compression when required
* DOC #1304: This adds documentation for SMAC callbacks that can be used by Auto-sklearn.
* DOC #1476: Example on how to interupt autosklearn with a callback, implementing a very naive early stopping
* MAINT #1364, #1473: Improve import time of Auto-sklearn 2 by moving the construction of the selector
model from import time to construction time.
* MAINT #1425: Update `StopWatch` to be context manager.
* MAINT #1454: Rename interal bool parameters `categorical` to `feat_type` to reflect the use of different feature types
* MAINT #1474: remove left-overs of a "public test set" from the code. This has no influence on
any user-facing code.
* MAINT #1487: Replace deprecated of `DataFrame.append`
* MAINT #1504: Rename `rval` to `return_value` or `run_value` to remove ambiguity
* MAINT #1506: Increase the time given to meta-learning-related unit tests to decrease the amount
of timeouts on github.
* MAINT #1527: Relax MLPRegressor unit tests precision.
* MAINT #1545: Add explicit lower bound subsample check in the train evaluator
* MAINT #1551: Fix issue with updated scipy skew see [here](https://github.com/scipy/scipy/issues/16765).
* MAINT #1434: Refactor the ensemble building process
* MAINT #1464: Improve testing, with caching (#1464), modularity (#1417)
* MAINT #1358: Add tooling Mypy, Flake8, isort, black
* FIX #741: Disable hyperparameters for a special data modality if it is not present, for example
disable one hot encoding if no categorical features are present.
* FIX #1365, #1369: Fix an issue with `ensemble_size == 0`.
* FIX #1374: Pass random state to all components of a pipeline.
* FIX #1432: Fixes an issue in which the `AutoSklearnClassifier.leaderboard()` or
`AutoSklearnRegressor.leaderboard()` could fail to display results.
* FIX #1480: Properly terminate Auto-sklearn on an exception or a keyboard interrupt.
* FIX #1532: Removes exception printing at shutdown for latest dask versions. The printed
exceptions did not impact performance at all and were only confusing as they suggested failures
of Auto-sklearn.
* FIX #1547: Fixes a bug in Auto-sklearn 2 that could silently break it when passing in pandas
DataFrames.
* FIX #1550: Fix recent bug when performing evaluations with pandas Y.
Contributors v0.15.0
********************
* Matthias Feurer
* Eddie Bergman
* Katharina Eggensperger
* Sagar Kaushik
* partev
* Lukas Strack
* Basavasagar K Patil
* Eric Pedley
* Aseem Kannal
* SkBlaz
Version 0.14.7
==============
* HOTFIX #1445: Locks `ConfigSpace` to `<0.5.0` and `smac` to `<1.3`. Adds upper bounds on `automl` packages to help prevent further issues.
Contributors v0.14.7
********************
* Eddie Bergman
Version 0.14.6
==============
* HOTFIX #1407: Catches keyword arguments in `SingleThreadedClient` so they don't get passed to it's executing `func`.
Contributors v0.14.6
********************
* Eddie Bergman
Version 0.14.5
==============
* HOTFIX: Release PyPi package with ``automl_common`` included
Contributors v0.14.5
********************
* Eddie Bergman
Version 0.14.4
==============
* Fix #1356: SVR degree hyperparameter now only active with "poly" kernel.
* Add #1311: Black format checking (non-strict).
* Maint #1306: Run history is now saved every iteration
* Doc #1309: Updated the doc faqs to include many use cases and the manual for early introductions
* Doc #1322: Fix typo in contribution guide
* Maint #1326: Add isort checker (non-strict)
* Maint #1238, #1346, #1368, #1370: Update warnings in tests
* Maint #1325: Test workflow can now be manually triggered
* Maint #1332: Update docstring and typing of ``include`` and ``exclude`` params
* Add #1260: Support for Python 3.10
* Add #1318: First update to use the shared backend in a new submodule `automl_common `_
* Fix #1339: Resolve dependancy issues with ``sphinx_toolbox``
* Fix #1335: Fix issue where some regression algorithm gave incorrect output dimensions as raised in #1297
* Doc #1340: Update example for predefined splits
* Fix #1329: Fix random state not being passed to the ConfigurationSpace
* Maint #1348: Stop double triggering of github workflows
* Doc #1349: Rename OSX to macOS in docs
* Add #1321: Change ``show_models()`` to produce actual pipeline objects and not a ``str``
* Maint #1361: Remove ``flaky`` dependency
* Maint #1366: Make ``SimpleClassificationPipeline`` tests more deterministic
* Maint #1367: Update test values for ``MLPRegressor`` with newer numpy
Contributors v0.14.4
********************
* Eddie Bergman
* Matthias Feurer
* Katharina Eggensperger
* UserFindingSelf
* partev
Version 0.14.3
==============
* HOTFIX #1356: Updates dask to ``dask.distributed >=2012.12``.
Contributors v0.14.3
********************
* Eddie Bergman
Version 0.14.2
==============
* FIX #1290: Fixes a bug where it was not possible to extend Auto-sklearn and run it in parallel.
Contributors v0.14.2
********************
* Matthias Feurer
Version 0.14.1
==============
* FIX #1248: Allow for sparse ``y_test``.
* FIX #1259: Fix an issue that could result in ``setup.py`` not working due to relative paths
being chosen.
* MAINT #1261: Include a CITATION.cff file
* MAINT #1263: Make unit test deterministic.
* DOC #1269: Fix example on extending data preprocessing.
* DOC #1270: Remove ``>>>`` from code examples in the documentation.
* DOC #1271: Fix a typo in an example in the documentation.
* DOC #1282: Add a contribution guide.
Contributors v0.14.1
********************
* Eddie Bergman
* Michael Becker
* Katharina Eggensperger
Version 0.14.0
==============
* ADD #900: Make data preprocessing more configurable, for example allow to completely disable it.
* ADD #1128: Adds new functionality to retrieve data for an accuracy over time plot from
Auto-sklearn without additional code.
* FIX #1149: Stops Auto-sklearn from printing weird warnings (`Exception ignored in [...]`) at
shutdown.
* FIX #1169: Fixes a bug which made cross-validation and multi-output regression incompatible.
* FIX #1170: Make all preprocessing techniques deterministic.
* FIX #1190: Fixes a bug which could make predictive probabilities contain too few classes in
case one class was only present a single time.
* FIX #1209: Pass random states to pipeline objects.
* FIX #1204: Add support for sparse data in Auto-sklearn 2.0.
* FIX #1210: Add support for sparse `y` labels.
* FIX #1245: Fixes a bug which could result in Auto-sklearn crashing in case a class was present
only once.
* DOC #532,#1242: Simplify installation instructions.
* DOC #1144: Document installation via `conda`
* DOC #1195,#1201,#1214: Fix a few typos and links. Make some http links https links.
* DOC #1200: Fixes variable name in an example.
* DOC #1229: Improve code formatting in the documentation.
* DOC #1235: Improve docker startup command so it also work on Windows.
* MAINT #1198: Use latest Ubuntu LTS (20:04) for github actions.
* MAINT #1231: The command `make linkcheck` no longer builds the documentation, speeding up
link-checking.
* MAINT #1233: Enable regression testing with 3 classification and 3 regression datasets on
github actions.
* MAINT #1239: Increase the timeout for github actions to 60 minutes.
Contributors v0.14.0
********************
* Pieter Gijsbers
* Taneli Mielikäinen
* Rohit Agarwal
* hnishi
* Francisco Rivera Valverde
* Eddie Bergman
* Satyam Jha
* Joel Jose
* Oli
* Matthias Feurer
Version 0.13.0
==============
* ADD #1100: Provide access to the callbacks of SMAC.
* ADD #1185: New leaderboard functionality to visualize models
* FIX #1133: Refer to the correct attribute in an error message.
* FIX #1154: Allow running Auto-sklearn on a 32-bit system.
* MAINT #924: Instead of passing classes for the resampling strategy one has now to pass objects.
* MAINT #1108: Limit the number of threads used by numpy and/or scikit-learn via `threadpoolctl`.
* MAINT #1135: Simplify internal workflow of pandas handling. This results in pandas being passed
directly passed to scikit-learn models instead of being internally converted into a numpy array.
However, this should neither impact the behavior nor the performance of Auto-sklearn.
* MAINT #1157: Drop support for Python 3.6, enable support for Python 3.9.
* MAINT #1159: Remove the output directory argument to the classifier and regressor. Despite the
name, the output directory was not used and was a leftover from participating in the AutoML
challenges.
* MAINT #1187: Bump requires SMAC version to at least 0.14.
* DOC #1109: Add an FAQ.
* DOC #1126: Add new examples on how to use scikit-learn's inspect module.
* DOC #1136: Add a new example on how to perform multi-output regression.
* DOC #1152: Enable link checking when building the documentation.
* DOC #1158: New example on how to configure the logger for Auto-sklearn.
* DOC #1165: Improve the readme page.
Contributors v0.13.0
********************
* Matthias Feurer
* Eddie Bergman
* bitsbuffer
* Francisco Rivera Valverde
Version 0.12.8
==============
* MAINT #1183: Introduce an upper bound on the dask version to retain compatibility with SMAC3.
Contributors v0.12.8
********************
* Eddie Bergman
Version 0.12.7
==============
* ADD #1178: Reduce precision if dataset is too large for given memory limit.
* ADD #1179: Improve Auto-sklearn 2.0 meta-data by providing new meta-data for the metrics
`roc_auc` and `logloss`.
* DOC: Fix reference to arXiv paper
* MAINT #1134,#1142,#1143: Improvements to the stale bot - the stale bot now marks issues labeled
with `feedback required` as stale if there is nothing happening for 30 days. After another 7
days it then closes the issue.
* MAINT: Added a new issue template for questions.
* MAINT #1168: Upper-bound scipy to `1.6.3` as `1.7.0` is incompatible with `SMAC`.
* MAINT #1173: Update the license files to be recognized by github.
Contributors v0.12.7
********************
* Francisco Rivera Valverde
* Matthias Feurer
* JJ Ben-Joseph
* Isaac Chung
* Katharina Eggensperger
* bitsbuffer
* Eddie Bergman
* olehb007
Version 0.12.6
==============
* ADD #886: Provide new function which allows fitting only a single configuration.
* DOC #1070: Clarify example on how successive halving and Bayesian optimization play together.
* DOC #1112: Fix type.
* DOC #1122: Add Python 3 to the installation command for Ubuntu.
* FIX #1114: Fix a bug which made printing dummy models fail.
* FIX #1117: Fix a bug previously made `memory_limit=None` fail.
* FIX #1121: Fix an edge case which could decrease performance in Auto-sklearn 2.0 when using
cross-validation with iterative fitting.
* FIX #1123: Fix a bug `autosklearn.metrics.calculate_score` for metrics/scores which need
to be minimized where the function previously returned the loss and not the score.
* FIX #1115/#1124: Fix a bug which would prevent Auto-sklearn from computing meta-features in the
multiprocessing case.
Contributors v0.12.6
********************
* Francisco Rivera Valverde
* stock90975
* Lucas Nildaimon dos Santos Silva
* Matthias Feurer
* Rohit Agarwal
Version 0.12.5
==============
* MAINT: Remove ``Cython`` and ``numpy`` as installation requirements.
Contributors v0.12.5
********************
* Matthias Feurer
Version 0.12.4
==============
* ADD #660: Enable scikit-learn's power transformation for input features.
* MAINT: Bump the ``pyrfr`` minimum dependency to 0.8.1 to automatically download wheels from pypi
if possible.
* FIX #732: Add a missing size check into the GMEANS clustering used for the NeurIPS 2015 paper.
* FIX #1050: Add missing arguments to the ``AutoSklearn2Classifier`` signature.
* FIX #1072: Fixes a bug where the ``AutoSklearn2Classifier`` could not be created due to trying to
cache to the wrong directory.
Contributors v0.12.4
********************
* Matthias Feurer
* Francisco Rivera
* Maximilian Greil
* Pepe Berba
Version 0.12.3
==============
* FIX #1061: Fixes a bug where the model could not be printed in a jupyter notebook.
* FIX #1075: Fixes a bug where the ensemble builder would wrongly prune good models for loss
functions (i.e. functions that need to be minimized such as ``logloss`` or ``mean_squared_error``.
* FIX #1079: Fixes a bug where ``AutoMLClassifier.cv_results`` and ``AutoMLRegressor.cv_results``
could rank results in opposite order for loss functions (i.e. functions that need to be minimized
such as ``logloss`` or ``mean_squared_error``.
* FIX: Fixes a bug in offline meta-data generation that could lead to a deadlock.
* MAINT #1076: Uses the correct multiprocessing context for computing meta-features
* MAINT: Cleanup readme and main directory
Contributors v0.12.3
********************
* Matthias Feurer
* ROHIT AGARWAL
* Francisco Rivera
Version 0.12.2
==============
* ADD #1045: New example demonstrating how to log multiple metrics during a run of Auto-sklearn.
* DOC #1052: Add links to mybinder
* DOC #1059: Improved the example on manually starting workers for Auto-sklearn.
* FIX #1046: Add the final result of the ensemble builder to the ensemble builder trajectory.
* MAINT: Two log outputs of level warning about metadata were turned reduced to the info loglevel
as they are not actionable for the user.
* MAINT #1062: Use threads for local dask workers and forkserver to start subprocesses to reduce
overhead.
* MAINT #1053: Remove the restriction to guard single-core Auto-sklearn by
``__main__ == "__name__"`` again.
Contributors v0.12.2
********************
* Matthias Feurer
* ROHIT AGARWAL
* Francisco Rivera
* Katharina Eggensperger
Version 0.12.1
==============
* ADD: A new heuristic which gives a warning and subsamples the data if it is too large for the
given ``memory_limit``.
* ADD #1024: Tune scikit-learn's ``MLPClassifier`` and ``MLPRegressor``.
* MAINT #1017: Improve the logging server introduced in release 0.12.0.
* MAINT #1024: Move to scikit-learn 0.24.X.
* MAINT #1038: Use new datasets for regression and classification and also update the metadata
used for Auto-sklearn 1.0.
* MAINT #1040: Minor speed improvements in the ensemble selection algorithm.
Contributors v0.12.1
********************
* Matthias Feurer
* Katharina Eggensperger
* Francisco Rivera
Version 0.12.0
==============
* BREAKING: Auto-sklearn must now be guarded by ``__name__ == "__main__"`` due to the use of the
``spawn`` multiprocessing context.
* ADD #1026: Adds improved meta-data for Auto-sklearn 2.0 which results in strong improved
performance.
* MAINT #984 and #1008: Move to scikit-learn 0.23.X
* MAINT #1004: Move from travis-ci to github actions.
* MAINT 8b67af6: drop the requirement to the lockfile package.
* FIX #990: Fixes a bug that made Auto-sklearn fail if there are missing values in a pandas
DataFrame.
* FIX #1007, #1012 and #1014: Log multiprocessing output via a new log server. Remove several
potential deadlocks related to the joint use of multi-processing, multi-threading and logging.
Contributors v0.12.0
********************
* Matthias Feurer
* ROHIT AGARWAL
* Francisco Rivera
Version 0.11.1
==============
* FIX #989: Fixes a bug where `y` was not passed to all data preprocessors which made 3rd party
category encoders fail.
* FIX #1001: Fixes a bug which could make Auto-sklearn fail at random.
* MAINT #1000: Introduce a minimal version for ``dask.distributed``.
Contributors v0.11.1
********************
* Matthias Feurer
Version 0.11.0
==============
* ADD #992: Move ensemble building from being a separate process to a job submitted to the dask
cluster. This allows for better control of the memory used in multiprocessing settings.
* FIX #905: Make ``AutoSklearn2Classifier`` picklable.
* FIX #970: Fix a bug where Auto-sklearn would fail if categorical features are passed as a
Pandas Dataframe.
* MAINT #772: Improve error message in case of dummy prediction failure.
* MAINT #948: Finally use Pandas >= 1.0.
* MAINT #973: Improve meta-data by running meta-data generation for more time and separately for
important metrics.
* MAINT #997: Improve memory handling in the ensemble building process. This allows building
ensembles for larger datasets.
Contributors v0.11.0
********************
* Matthias Feurer
* Francisco Rivera
* Karl Leswing
* ROHIT AGARWAL
Version 0.10.0
==============
* ADD #325: Allow to separately optimize metrics for metadata generation.
* ADD #946: New dask backend for parallel Auto-sklearn.
* BREAKING #947: Drop Python3.5 support.
* BREAKING #946: Remove shared model mode for parallel Auto-sklearn.
* FIX #351: No longer pass un-picklable logger instances to the target function.
* FIX #840: Fixes a bug which prevented computing metadata for regression datasets. Also
adds a unit test for regression metadata computation.
* FIX #897: Allow custom splitters to be used with multi-ouput regression.
* FIX #951: Fixes a lot of bugs in the regression pipeline that caused bad performance for
regression datasets.
* FIX #953: Re-add `liac-arff` as a dependency.
* FIX #956: Fixes a bug which could cause Auto-sklearn not to find a model on disk which
is part of the ensemble.
* FIX #961: Fixes a bug which caused Auto-sklearn to load bad meta-data for metrics which cannot
be computed on multiclass datasets (especially ROC_AUC).
* DOC #498: Improve the example on resampling strategies by showing how to pass scikit-learn's
splitter objects to Auto-sklearn.
* DOC #670: Demonstrate how to give access to training accuracy.
* DOC #872: Improve an example on how obtain the best model.
* DOC #940: Improve documentation of the docker image.
* MAINT: Improve the docker file by setting environment variable that restrict BLAS and OMP to only
use a single core.
* MAINT #949: Replace `pip` by `pip3` in the installation guidelines.
* MAINT #280, #535, #956: Update meta-data and include regression meta-data again.
Contributors v0.10.0
********************
* Francisco Rivera
* Matthias Feurer
* felixleungsc
* Chu-Cheng Fu
* Francois Berenger
Version 0.9.0
=============
* ADD #157,#889: Improve handling of pandas dataframes, including the possibility to use pandas'
categorical column type.
* ADD #375: New `SelectRates` feature preprocessing component for regression.
* ADD #891: Improve the robustness of Auto-sklearn by using the single best model if no ensemble
is found.
* ADD #902: Track performance of the ensemble over time.
* ADD #914: Add an example on using pandas dataframes as input to Auto-sklearn.
* ADD #919: Add an example for multilabel classification.
* MAINT #909: Fix broken links in the documentation.
* MAINT #907,#911: Add initial support for mypy.
* MAINT #881,#927: Automatically build docker images on pushes to the master and development
branch and also push them to dockerhub and the github docker registry.
* MAINT #918: Remove old dependencies from requirements.txt.
* MAINT #931: Add information about the host system and installed packages to the log file.
* MAINT #933: Reduce the number of warnings raised when building the documentation by sphinx.
* MAINT #936: Completely restructure the examples section.
* FIX #558: Provide better error message when the ensemble process fails due to a memory issue.
* FIX #901: Allow custom resampling strategies again (was broken due to an upgrade of SMAC).
* FIX #916: Fixes a bug where the data preprocessing configurations were ignored.
* FIX #925: make internal data preprocessing objects clonable.
Contributors v0.9.0
*******************
* Francisco Rivera
* Matthias Feurer
* felixleungsc
* Vladislav Skripniuk
Version 0.8
===========
* ADD #803: multi-output regression
* ADD #893: new Auto-sklearn mode Auto-sklearn 2.0
Contributors v0.8.0
*******************
* Chu-Cheng Fu
* Matthias Feurer
Version 0.7.1
=============
* ADD #764: support for automatic per_run_time_limit selection
* ADD #864: add the possibility to predict with cross-validation
* ADD #874: support to limit the disk space consumption
* MAINT #862: improved documentation and render examples in web page
* MAINT #869: removal of competition data manager support
* MAINT #870: memory improvements when building ensemble
* MAINT #882: memory improvements when performing ensemble selection
* FIX #701: scaling factors for metafeatures should not be learned using test data
* FIX #715: allow unlimited ML memory
* FIX #771: improved worst possible result calculation
* FIX #843: default value for SelectPercentileRegression
* FIX #852: clip probabilities within [0-1]
* FIX #854: improved tmp file naming
* FIX #863: SMAC exceptions also registered in log file
* FIX #876: allow Auto-sklearn model to be cloned
* FIX #879: allow 1-D binary predictions
Contributors v0.7.1
*******************
* Matthias Feurer
* Xiaodong DENG
* Francisco Rivera
Version 0.7.0
=============
* ADD #785: user control to reduce the hard drive memory required to store ensembles
* ADD #794: iterative fit for gradient boosting
* ADD #795: add successive halving evaluation strategy
* ADD #814: new sklearn.metrics.balanced_accuracy_score instead of custom metric
* ADD #815: new experimental evaluation mode called iterative_cv
* MAINT #774: move from scikit-learn 0.21.X to 0.22.X
* MAINT #791: move from smac 0.8 to 0.12
* MAINT #822: make autosklearn modules PEP8 compliant
* FIX #733: fix for n_jobs=-1
* FIX #739: remove unnecessary warning
* FIX ##769: fixed error in calculation of meta features
* FIX #778: support for python 3.8
* FIX #781: support for pandas 1.x
Contributors v0.7.0
*******************
* Andrew Nader
* Gui Miotto
* Julian Berman
* Katharina Eggensperger
* Matthias Feurer
* Maximilian Peters
* Rong-Inspur
* Valentin Geffrier
* Francisco Rivera
Version 0.6.0
=============
* MAINT: move from scikit-learn 0.19.X to 0.21.X
* MAINT #688: allow for pyrfr version 0.8.X
* FIX #680: Remove unnecessary print statement
* FIX #600: Remove unnecessary warning
Contributors v0.6.0
*******************
* Guilherme Miotto
* Matthias Feurer
* Jin Woo Ahn
Version 0.5.2
=============
* FIX #669: Correctly handle arguments to the ``AutoMLRegressor``
* FIX #667: Auto-sklearn works with numpy 1.16.3 again.
* ADD #676: Allow brackets [ ] inside the temporary and output directory paths.
* ADD #424: (Experimental) scripts to reproduce the results from the original Auto-sklearn paper.
Contributors v0.5.2
*******************
* Jin Woo Ahn
* Herilalaina Rakotoarison
* Matthias Feurer
* yazanobeidi
Version 0.5.1
=============
* ADD #650: Auto-sklearn will immediately stop if prediction using scikit-learn's dummy predictor
fail.
* ADD #537: Auto-sklearn will no longer start for time limits less than 30 seconds.
* FIX #655: Fixes an issue where predictions using models from parallel Auto-sklearn runs could
be wrong.
* FIX #648: Fixes an issue with custom meta-data directories.
* FIX #626: Fixes an issue where losses were not minimized, but maximized.
* MAINT #646: Do no longer restrict the numpy version to be less than 1.14.5.
Contributors v0.5.1
*******************
* Jin Woo Ahn
* Taneli Mielikäinen
* Matthias Feurer
* jianswang
Version 0.5.0
=============
* ADD #593: Auto-sklearn supports the ``n_jobs`` argument for parallel
computing on a single machine.
* DOC #618: Added links to several system requirements.
* Fixes #611: Improved installation from pip.
* TEST #614: Test installation with clean Ubuntu on travis-ci.
* MAINT: Fixed broken link and typo in the documentation.
Contributors v0.5.0
*******************
* Mohd Shahril
* Adrian
* Matthias Feurer
* Jirka Borovec
* Pradeep Reddy Raamana
Version 0.4.2
=============
* Fixes #538: Remove rounding errors when giving a training set fraction for
holdout.
* Fixes #558: Ensemble script now uses less memory and the memory limit can be
given to Auto-sklearn.
* Fixes #585: Auto-sklearn's ensemble script produced wrong results when
called directly (and not via one of Auto-sklearn's estimator classes).
* Fixes an error in the ensemble script which made it non-deterministic.
* MAINT #569: Rename hyperparameter to have a different name than a
scikit-learn hyperparameter with different meaning.
* MAINT #592: backwards compatible requirements.txt
* MAINT #588: Fix SMAC version to 0.8.0
* MAINT: remove dependency on the six package
* MAINT: upgrade to XGBoost 0.80
Contributors v0.4.2
*******************
* Taneli Mielikäinen
* Matthias Feurer
* Diogo Bastos
* Zeyi Wen
* Teresa Conceição
* Jin Woo Ahn
Version 0.4.1
=============
* Added documentation on `how to extend Auto-sklearn `_
with custom classifier, regressor, and preprocessor.
* Auto-sklearn now requires numpy version between 1.9.0 and 1.14.5, due to higher versions
causing travis failure.
* Examples now use ``sklearn.datasets.load_breast_cancer()`` instead of ``sklearn.datasets.load_digits()``
to reduce memory usage for travis build.
* Fixes future warnings on non-tuple sequence for indexing.
* Fixes `#500 `_: fixes
ensemble builder to correctly evaluate model score with any metrics.
See this `PR `_.
* Fixes `#482 `_ and
`#491 `_: Users can now set up
custom logger configuration by passing a dictionary created by a yaml file to
``logging_config``.
* Fixes `#566 `_: ensembles are now sorted correctly.
* Fixes `#293 `_: Auto-sklearn checks if appropriate
target type was given for classification and regression before call to ``fit()``.
* Travis-ci now runs flake8 to enforce pep8 style guide, and uses travis-ci instead of circle-ci
for deployment.
Contributors v0.4.1
*******************
* Matthias Feurer
* Manuel Streuhofer
* Taneli Mielikäinen
* Katharina Eggensperger
* Jin Woo Ahn
Version 0.4.0
=============
* Fixes `#409 `_: fixes
``predict_proba`` to no longer raise an `AttributeError`.
* Improved documentation of the parallel example.
* Classifiers are now tested to be idempotent as `required by scikit-learn
`_.
* Fixes the usage of the shrinkage parameter in LDA.
* Fixes `#410 `_ and changes
the SGD hyperparameters
* Fixes `#425 `_ which
caused the non-linear support vector machine to always crash on OSX.
* Implements `#149 `_: it
is now possible to pass a custom cross-validation split following
scikit-learn's ``model_selection`` module.
* It is now possible to decide whether or not to shuffle the data in
Auto-sklearn by passing a bool `shuffle` in the dictionary of
``resampling_strategy_arguments``.
* Added functionality to track the test performance over time.
* Re-factored the ensemble building to be faster, read less data from the
hard drive and perform random tie breaking in case of equally
well-performing models.
* Implements `#438 `_: To
be consistent with the output of SMAC (which minimizes the loss of a target
function), the output of the ensemble builder is now also the output of a
minimization problem.
* Implements `#271 `_:
XGBoost is available again, even configuring the new dropout functionality.
* New documentation section :ref:`inspect`.
* Fixes `#444 `_:
Auto-sklearn now only loads models for refit which are actually relevant
for the ensemble.
* Adds an operating system check at import and installation time to make sure
to not accidentaly run on a Windows machine.
* New examples gallery using sphinx gallery: :ref:`examples`
* Safeguard Auto-sklearn against deleting directories it did not create (Issue
`#317 `_.
Contributors v0.4.0
*******************
* Matthias Feurer
* kaa
* Josh Mabry
* Katharina Eggensperger
* Vladimir Glazachev
* Jesper van Engelen
* Jin Woo Ahn
* Enrico Testa
* Marius Lindauer
* Yassine Morakakam
Version 0.3.0
=============
* Upgrade to scikit-learn 0.19.1.
* Do not use the ``DummyClassifier`` or ``DummyRegressor`` as part of an
ensemble. Fixes `#140 `_.
* Fixes #295 by loading the data in the subprocess instead of the main process.
* Fixes #326: refitting could result in a type error. This is now fixed by
better type checking in the classification components.
* Updated search space for ``RandomForestClassifier``, ``ExtraTreesClassifier``
and ``GradientBoostingClassifier`` (fixes #358).
* Removal of constant features is now a part of the pipeline.
* Allow passing an SMBO object into the ``AutoSklearnClassifier`` and
``AutoSklearnRegressor``.
Contributors v0.3.0
*******************
* Matthias Feurer
* Jesper van Engelen
Version 0.2.1
=============
* Allows the usage of scikit-learn 0.18.2.
* Upgrade to latest SMAC version (``0.6.0``) and latest random forest version
(``0.6.1``).
* Added a Dockerfile.
* Added the possibility to change the size of the holdout set when
using holdout resampling strategy.
* Fixed a bug in QDA's hyperparameters.
* Typo fixes in print statements.
* New method to retrieve the models used in the final ensemble.
Contributors v0.2.1
*******************
* Matthias Feurer
* Katharina Eggensperger
* Felix Leung
* caoyi0905
* Young Ryul Bae
* Vicente Alencar
* Lukas Großberger
Version 0.2.0
=============
* **auto-sklearn supports custom metrics and all metrics included in
scikit-learn**. Different metrics can now be passed to the ``fit()``-method
estimator objects, for example
``AutoSklearnClassifier.fit(metric='roc_auc')``.
* Upgrade to scikit-learn 0.18.1.
* Drop XGBoost as the latest release (0.6a2) does not work when spawned by
the pyninsher.
* *auto-sklearn* can use multiprocessing in calls to ``predict()`` and
``predict_proba``. By `Laurent Sorber `_.
Contributors v0.2.0
*******************
* Matthias Feurer
* Katharina Eggensperger
* Laurent Sorber
* Rafael Calsaverini
Version 0.1.x
=============
There are no release notes for auto-sklearn prior to version 0.2.0.
Contributors v0.1.x
*******************
* Matthias Feurer
* Katharina Eggensperger
* Aaron Klein
* Jost Tobias Springenberg
* Anatolii Domashnev
* Stefan Falkner
* Alexander Sapronov
* Manuel Blum
* Diego Kobylkin
* Jaidev Deshpande
* Jongheon Jeong
* Hector Mendoza
* Timothy J Laurent
* Marius Lindauer
* _329_
* Iver Jordal
---
### CONTRIBUTING
# Contributing to auto-sklearn
Thanks for checking out the contribution guide!
We included a [quick overview](#pull-request-overview) at the end for anyone familiar with open-source contribution or familiar with auto-sklearn and simply wants to see our workflow.
If you're new to contributing then hopefully this guide helps with contributing to open-source and auto-sklearn, whether it be a simple doc fix, a small bug fix or even new features that everyone can get use out of.
If you're looking for a particular project to work on, check out the [Issues](https://github.com/automl/auto-sklearn/issues) for things you might be interested in!
For experienced contributors, you can skip the overview and find the quick walk-through [here](#pull-request-overview)!
This guide is only aimed towards Unix command line users as that's what we know but the same principles apply.
# Contributing
There are many kinds of contributions you can make to auto-sklearn but we'll focus on three main ones **Documentation**, **Bug Fixes** and **Features**, each of which require a little bit of a different flow.
We need to perform several checks to make sure it meets code standards and won't cause any issues later on.
We tend to follow a development cycle which could be called _Gitflow_ which you are new to git or git-based projects, you can see a nice summary [here](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow).
First we'll go over the general flow, what each step does and then later look at making more specific kinds of changes, what we'd like to see and how you might create a workflow.
Following that we'll tell you about how you can test your changes locally and then how to submit your pull request!
## General steps
* The first thing to do is create your own [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo).
This is to give you a nice place to work on your changes without impacting any code from the original repository.
To do this, navigate to [automl/auto-sklearn](https://github.com/automl/auto-sklearn) and hit the **fork** button in the top-right corner.
This will copy the repository to your own account, including all of its different branches.
You'll be able to access this at `https://github.com/{your-username}/auto-sklearn`.
* The next steps are to download **your own fork** and to create a new [branch](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-branches) where all your changes will go.
It's important to work off the latest changes on the **development** branch.
```bash
# With https
# Note the --recurse-submodules args, we use a submodule autosklearn/automl_common
# so it needs to be downloaded too
git clone --recurse-submodules https://github.com/your-username/auto-sklearn
# ... or with ssh
git clone --recurse-submodules git@github.com:your-username/auto-sklearn.git
# Navigate into the cloned repo
cd auto-sklearn
# Create a new branch based off the development one
git checkout -b my_new_branch development
# If you missed the --recurse-submodules arg during clone or need to install the
# submodule manually, then execute the following line:
#
# git submodule udate --init --recursive
# ... Alternatively, if you would prefer a more manual method
# Show all the available branches with a * beside your current one
git branch
# Switch to the development branch
git checkout development
# Create a new branch based on the currently active branch
git checkout -b my_new_branch
# If you missed the --recurse-submodules arg during clone or need to install the
# submodule manually, then execute the following line:
#
# git submodule update --init --recursive
```
The reason to create a new branch is two fold:
* One, it keeps the commit history for your changes much cleaner once we merge them in.
* If you have to perform a **rebase** or a **merge** later on, this will be much easier.
* You'll need a [virtual environment](https://docs.python.org/3/tutorial/venv.html) to work in.
If you've never used them before, now is definitely the time to start as a virtual environment lets you keep packages for a project separate.
```bash
# Create a virtual environment in a folder called my-virtual-env
python -m venv my-virtual-env
# Activate the virtual environment
source my-virtual-env/bin/activate
```
* A popular alternative to managing Python projects is [conda](https://docs.conda.io/en/latest/).
* The folder `my-virtual-env` is where dependency packages that are required for auto-sklearn will go.
* In general, once you have activated the virtual environment with `source my-virtual-env/bin/activate`, anything you install with `pip install` will now go into the virtual environment.
While this environment is active, any python you run will have access to the packages here.
If at any time you want to deactivate it simply type `deactivate` in the shell or just close the shell and open a new one.
* If you use Python 3.6 or lower on your machine then unfortunately auto-sklearn doesn't support this.
Fortunately, you can check out [pyenv](https://github.com/pyenv/pyenv) which lets you switch between Python versions on the fly!
* Now that we have a virtual environment, it's time to install all the dependencies into it.
We've provided a simple `make` command to help do this.
```bash
make install-dev
# Manually
pip install -e .[test,examples,doc]
# If you're using shells other than bash you'll need to use
pip install -e ".[test,examples,doc]"
```
* If your only exposure to using pip is `pip install package_name` then this might be a bit confusing.
* If we type `pip install -e .` (notice the 'dot'), this tells `pip` to install a package located here, in this directory, `.`.
The `-e` flag indicates that it should be editable, meaning you will not have to run `pip install .` every time you make a change and want to try it.
* Finally the `[test,examples,doc]` tells `pip` that there's some extra optional dependencies that we want to install.
These are dependencies used in development but ones that are not required to actually run auto-sklearn itself.
You can check out what these are in the `setup.py` file.
* If you're new to virtual environments, after performing all this, it's a great time to check out what actually exists in the `my-virtual-env` folder.
* You can check out some functionality we have captured in a `Makefile` by running `make help`
* Now it's time to make some changes, whether it be for [documentation](#documentation), a [bug fix](#bug-fixes) or a new [features](#features).
## Making Changes
We'll go over three main categories of contributions but don't feel limited by these headers, adding to our tests, improving the typing of functions and methods or even some compliance changes are also super useful!
#### Bug Fixes
Auto-sklearn has been through quite a few iterations, been used for many purposes and is constantly used in ways we didn't even think of.
Like any maturing software, there will be bugs, old and new.
While we try to create unit tests to catch as many of these as we can, some slip through and new bugs get introduced as dependencies are updated and new features are introduced.
If you're looking to help by fixing some bugs, or you've encountered your own bugs you'd like fixed in the official version of auto-sklearn, we would greatly appreciate any help!
We'd be happy to guide you through what we think may be the underlying cause or at least point you in the right direction if it's something you wish to work on.
Some core things to consider in fixing a bug:
* What's the minimal working example that reproduces this bug?
* This is usually the first step.
The process of creating a minimal piece of code that reproduces a bug often illuminates what needs to be fixed.
* What's the quick fix and what's the long term fix?
* Sometimes it's a code typo, a quick correction and problem solved.
Other times, the bug is an artifact of some larger underlying issue that has gone unnoticed and might require some restructuring.
If this is the case, let us know as you work on it!
If it requires breaking, such as changing default behaviour or public API, sometimes a quick patch and fix will do and larger restricting fixes can be tackled in a timely manner.
As a rule of thumb, if a bug requires modifying more then 50-100 lines of code it's probably something we would like to talk through on how best to tackle it.
* How can we create a test for this bug in the future?
* Of course, once a bug is squashed, we'd like it to not show again and having a test to catch it for the future will future-proof against any changes down the line.
Thankfully, most of this is usually captured in the minimal working example and all that is left is to turn it into a comprehensive test!
What's important once fixing a bug is [writing a good PR](#creating-the-pr) that let's us now how you identified the bug, what the problem was and how it was fixed.
This lets use review your code with all this in mind and follow the same thought process that lead you to fix it in the first place!
#### Documentation
Anything to contribute to better documentation is always appreciated and the main way users can get to know about auto-sklearn.
Whether it's a typo fix, something you didn't find clear or something you think we didn't explain properly, we'd love to improve it!
All of our documentation is done with [`sphinx`](https://www.sphinx-doc.org/en/master/) with some various
plugins that you can see in [`doc/conf.py`](https://github.com/automl/auto-sklearn/blob/master/doc/conf.py#L42).
All of your changes can be viewed by first [building the docs](#testing) and then opening `doc/build/html/index.html` in a browser.
* If you're simply fixing a typo, there shouldn't be much to do except make the PR and we'll accept it without much issue.
* If you want to fix a link you should know how linking with `sphinx` works
* For links to internal documentation, you can create a label with
```.rst
.. _mylabel:
```
* Later on you can reference this label by
```.rst
I am reference to :ref:`the above label`
```
* Now if you want to link some external documentation you'll need
to do something like
```.rst
Here's a `link`_ to the external documentation on linking for sphinx
```
Notice the trailing `_` which is important
* If you want to make some more detailed documentation about some feature that you introduced or you think is not well documented, you'll have to think about a few things.
* Can you include a code snippet to illustrate what you mean?
* Are there other relevant parts of the documentation or code that should be linked to?
* How much other parts of auto-sklearn are you relying on readers to know before hand, maybe link to those sections if you do.
* If you want to contribute an example, it's a great way to really illustrate an entire flow of some feature.
`sphinx-gallery` will run any python file `example_*.py` in one of the example folders.
This allows you to have both ReStructured Markdown (rst) and python code with it's output into a [single html page](https://automl.github.io/auto-sklearn/master/examples/40_advanced/example_calc_multiple_metrics.html#sphx-glr-examples-40-advanced-example-calc-multiple-metrics-py)!
You'll want to check out some of the [other examples](https://github.com/automl/auto-sklearn/tree/master/examples) to see how to embed rst into a `.py` file.
#### Features
While auto-sklearn has many features we're proud of, there's always room for more and better functionality.
Features don't have to be performance driven, in fact, most of the new features we'd love to see are to improve a users ability to interact with auto-sklearn, whether it be usability or an ability to inspect the inner workings in an intuitive manner!
However, features are usually a bigger project and for this we'd really advise getting in touch with us first about the feature in mind.
There are some things we believe best left for external libraries or integrations that we don't wish to consider at this time.
Another reason to get in touch is to nail exactly what this feature will look like beforehand, the more direct and concise the feature is, the better.
Some things to keep in mind with new features:
* A new feature is great, but will this change any existing default behaviours?
Unexpected changes for existing users can be detrimental and unexpected.
Sometimes it has to be done but often these new features can be presented as an option the user can enable.
* If you're introducing some new API:
* Creating some code samples of how you'd like your feature to be used is a great start.
* What current way is there to do the same thing, can any functionality already present be used to help with this new API?
* Are you going to deprecate any current API? This is an important fact to consider and something that will definitely have to be discussed.
**Testing Features** - Writing features are great but new features means new bugs, but thankfully that's what we can write tests for.
How to [test your feature](#testing) is always tricky, especially if the feature is big in scope.
Unfortunately there's no secret or rule of thumb other than try to cover every case you can think of.
The more it's tested the better!
Bugs will still get through, that's okay, we will have done what we can and we can fix those in the future but as long as the usual use-cases are covered, this shouldn't be too much of a problem.
**Documenting Features** - Now how are people going to know about your new feature you've introduced?
This is what [documentation](#documentation) is so great for and it's how almost all software functionality is expressed.
If the feature is enabled by a parameter, great, almost all the documentation is already present in the code docstring, automatically being rendered in the online docs ... that docstring that was updated when you made changes ... right?
Sometimes, the new functionality isn't so clear from a simple parameter description and so maybe something needs to be added to the `manual.rst`, a short paragraph suffices and much appreciated.
Lastly, if the feature really is a game changer or you're very proud of it, consider making an `example_*.py` that will be run and rendered in the online docs!
## Testing
* Let's assume you've made some changes, now we have to make sure they work.
Begin by simply running all the tests.
If there's any errors, they'll pop up once it's complete.
```bash
pytest
```
* Note that these may take a while so check out `pytest --help` to see how you can run tests so that only previous failures run or only certain tests are run.
This can help you try changes and get results faster.
Do however run one last full `pytest` once you are finished and happy!
* Here are some we find particularly useful
```
# Run tests in specific file like 'test_estimators.py'
pytest "test/test_automl/test_estimators.py"
# Run an entire directory of tests such as 'pipeline'
pytest "test/test_pipeline"
# Run a specific test 'test_mytest' in a specific directory 'test_automl'
pytest -k "test_mytest" "test/test_automl"
# Rerun all the tests that failed in the last `pytest` command
pytest --last-failed
# Rerun all tests but run the failed ones first
pytest --failed-first
# Exit on the first test failure
pytest -x
```
* More advanced editors like PyCharm may have built in integrations which could be good to check out!
* Running all unittests will take a while, here's how you can run them in parallel
```
export OPENBLAS_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OMP_NUM_THREADS=1
pytest -n 4
```
* Now we are going to use [sphinx](https://www.sphinx-doc.org/en/master/) to generate all the documentation and make sure there are no issues.
```bash
make doc
```
* If you're unfamiliar with sphinx, it's a documentation generator which can read comments and docstrings from within the code and generate html documentation.
* If you've added documentation, we also has a command `linkcheck` for making sure all the links correctly go to some destination.
This helps tests for dead links or accidental typos.
```bash
make linkcheck
```
* We also use sphinx-gallery which can take python files (such as those in the `examples` folder) and run them, creating html which shows the code and the output it generates.
```bash
make examples
```
* To view the documentation itself, make sure it is built with the above commands and then open `doc/build/html/index.html` with your favourite browser:
```bash
# Firefox
firefox ./doc/build/html/index.html
# Using your default browser
xdg-open ./doc/build/html/index.html
```
* Once you've made all your changes and all the tests pass successfully, we need to make sure that the code fits a certain format and that the [typing](https://docs.python.org/3/library/typing.html) is correct.
* Formatting and import sorting can helps keep things uniform across all coding styles. We use [`black`](https://black.readthedocs.io/en/stable/) and [`isort`](https://isort.readthedocs.io/en/latest/) to do this for us. To automatically run these formatters across the code base, just run the following command:
```bash
make format
```
* To then check for issues using [`black`](https://black.readthedocs.io/en/stable/), [`isort`](https://isort.readthedocs.io/en/latest/), [`mypy`](http://mypy-lang.org/), [`flake8`](https://flake8.pycqa.org/en/latest/) and [`pydocstyle`](http://www.pydocstyle.org/en/stable/), run
```bash
make check
```
* To do this checking automatically, we use `pre-commit` which if you already installed everything with `make install-dev` then this has been done for you.
This will happen every time you make a commit and warn you of any issues.
Otherwise you can run the following to install pre-commit.
```bash
pre-commit install
```
* To run `pre-commit` manually:
```bash
pre-commit run --all-files
```
* The reason we use tools like [`flake8`](https://flake8.pycqa.org/en/latest/), [`mypy`](http://mypy-lang.org/), [`black`](https://black.readthedocs.io/en/stable/), [`isort`](https://isort.readthedocs.io/en/latest/) and [`pydocstyle`](http://www.pydocstyle.org/en/stable/) is to make sure that when we review code:
* There are no extra blank spaces and blank lines. (`flake8`, `black`)
* Lines don't end up too long. (`flake8`, `black`)
* Code from multiple source keeps a similar appearance. (`black`)
* Importing things is consistently ordered. (`isort`)
* Functions are type annotated and correct with static type checking. (`mypy`)
* Function and classes have docstrings. (`pydocstyle`)
If you are new to Python types, or stuck with how something should be 'typed', please feel free to push the pull request in the following steps and we should be able to help you out.
* If interested, the configuration for `pre-commit` can be found in `.pre-commit-config.yaml` with the other tools mainly being configured in `pyproject.toml` and `.flake8`.
## Creating the PR
* We've made sure all the changes work, we've maybe added a test for them and run all the tests locally.
It's time to commit the changes, push them up to your fork and create a pull request!
```bash
# Get an overview of all the files changes
git status
# Add your changed files
git add {changed files}
git commit -m "Something as meaningful as possible"
# This will push my_new_branch to your fork located at `origin`
git push --set-upstream origin my_new_branch
```
* At this point, we need to create a pull request (PR) to the [automl/auto-sklearn](https://github.com/automl/auto-sklearn) repository with our new changes.
This can be done simply by going to your own forked repo, clicking **'Contribute'**, and selecting
the **development** branch of `automl/auto-sklearn`.
* `automl/auto-sklearn` | `development` <- `your-username/auto-sklearn` | `my_new_branch`
The reason we don't want to directly merge new PR's into master is to make
sure we always have a stable version. With a development branch, we can safely
accumulate certain changes and makes sure they all work together before creating
a new master version.
* Now you've got to describe what you've changed.
You'll likely want to check out [this blogpost](https://hugooodias.medium.com/the-anatomy-of-a-perfect-pull-request-567382bb6067) which we believe to give a good overview of what a good PR looks like and will help us get your changes in sooner rather than later!
Some key things to include here are:
* A high level overview of what you've done such as fixing a problem or
introducing a new feature.
If it's a simple doc fix, don't worry too much about this.
* Have you introduced any breaking changes to the API?
We may not realise it while reviewing the changes but if you are aware,
it definitely helps to tell us!
* Do you think this might have any implications in the future or how would
further work on this look like?
* If you've introduced some new feature, write about who might use it,
give a brief code sample to show it and let us know how you tested it!
* If you've fixed a bug, write about why this bug exists in the first place,
how you solved it and how a test makes sure it won't pop up again!
* Once you've submitted a PR, we have it set up so github will automatically schedule some unit tests and documentation building to run.
This will make sure all the tests run smoothly, make sure the documentation builds correctly and do some quick check on code quality.
You'll be able to see these run in the **Checks** tab or at the bottom of the PR.
If you see a red x, that means somethings probably gone wrong which should have been caught by running the tests locally but we also do some checks in environments you were not developing on.
Sometimes it is the case that there is a bug unrelated to your changes that cause the tests to fail but we are aware of these.
If you see one of these failures, feel free to ask!
* Meanwhile, we'll also review your code.
Some common review points are:
* This seems odd, why was this done?
* Could you see if you can use the functionality from place X?
* Could you create a test or documentation for this?
* This could be a nicer way of doing this, what do you think?
Occasionally there will be some major point which will require more discussion but those are more on a case-by-case basis.
* This process of review, fix and testing may go on a few times.
The simplest way to reduce the time needed and to help us too is to run the tests, code formatting check and doc building locally.
If they all pass locally they will very often have no issues in the automated tests.
Once everyone is happy, it's time for us to hit (*squash*) merge, get it into the development branch and have one more contribution to auto-sklearn!
# Pull Request Overview
* Create a [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) of the [automl/auto-sklearn](https://github.com/automl/auto-sklearn) git repo
* Check out what's available by running `make help`.
* Clone your own fork and create a new branch from the branch to work on
```bash
git clone git@github.com:your-username/auto-sklearn.git
cd auto-sklearn
git checkout -b my_new_branch development
# Initialize autosklearn/automl_common submodule
git submodule update --init --recursive
# Create a virtual environment and activate it so there are no package
# conflicts
python -m venv my-virtual-env
source my-virtual-env/bin/activate
make install-dev
# pip install -e ".[test,docs,examples]" # To manually install things
# Edit files...
# Format code
make format
# Check for any issues
make check
# ... fix any issues
# If you changed documentation:
# This will generate all documentation and check links
make doc
make linkcheck
make examples # mainly needed if you modified some examples
# ... fix any issues
# If you edited any code
# Check out pytest --help if you want to only run specific tests
pytest
# ... fix any issues
# If you want to run pre-commit, the formatting checks we run on github
pre-commit install
pre-commit run --all-files
# ... fix any issues
# Check the changed files
git status
# Add the changes
git add {changed files}
git commit -m "Meaningful as you can make it message"
# Push back to your fork
git push --set-upstream origin my_new_branch
```
* Go to github, go to your fork and then make a pull request using the **Contribute** button.
* `automl/auto-sklearn` | `development` <- `your-username/auto-sklearn` | `my_new_branch`
* Write a [PR](#creating-the-pr) with a description of the changes, why you implemented them and any implications.
* Check out this [blog post](https://hugooodias.medium.com/the-anatomy-of-a-perfect-pull-request-567382bb6067) for some inspiration!
* Once we see this, we will run some automated tests on the pull request. These
tests are the same as the ones you can run manually and are mentioned in the
[test](#testing) section.
* We'll review the code and perhaps ask for some changes
* Once we're happy with the result, we'll merge it in!
# General FAQ
### I've finished my pull request or made new changes and want it reviewed, now what?
* We'll actively monitor what pull requests are already in progress.
Once you believe you pull request to be ready or you need some feedback, feel free to comment on your pull request, tagging @eddiebergman or @mfeurer and we'll provide feedback as soon as we can!
### I've been asked to rebase my pull request, why and what do I do?
* It can often be the case that while you were working on some changes, we may have merged something new into the development branch.
This can be a problem because the branch you created was based off the old development branch.
This means there are new changes in the automl/auto-sklearn code base you don't have in your forked repo or locally.
This is not always an issue, generally if different parts of the code were touched they can be merged safely.
Either way, if we ask you to [rebase](https://www.atlassian.com/git/tutorials/merging-vs-rebasing), it is because it won't merge or we think there may have been overlapping changes between what you were working on and the new code put into the development branch.
* First, update the development branch on **your fork**
* The easiest way to do this is go to *https://github.com/your-username/auto-sklearn*, navigate to the development branch and hit **fetch upstream**.
* This will make it so your fork of auto-sklearn is now up to date
* Second, we need to go to our clone and pull in these new changes to development
```bash
git checkout development
git pull
```
* Lastly, we need to rebase the my_new_branch on top of the new development
```bash
git checkout my_new_branch
git rebase development
```
* Now if there were no conflicts, that's it, you can continue as normal.
If there are conflicts, you'll have to sort these out which you can find out how to do [here](https://docs.github.com/en/get-started/using-git/resolving-merge-conflicts-after-a-git-rebase) and [here](https://docs.github.com/en/github/collaborating-with-pull-requests/addressing-merge-conflicts/resolving-a-merge-conflict-using-the-command-line).
---
### README
# auto-sklearn
**auto-sklearn** is an automated machine learning toolkit and a drop-in replacement for a [scikit-learn](https://scikit-learn.org) estimator.
Find the documentation **[here](https://automl.github.io/auto-sklearn/)**. Quick links:
* [Installation Guide](https://automl.github.io/auto-sklearn/master/installation.html)
* [Releases](https://automl.github.io/auto-sklearn/master/releases.html)
* [Manual](https://automl.github.io/auto-sklearn/master/manual.html)
* [Examples](https://automl.github.io/auto-sklearn/master/examples/index.html)
* [API](https://automl.github.io/auto-sklearn/master/api.html)
## auto-sklearn in one image
## auto-sklearn in four lines of code
```python
import autosklearn.classification
cls = autosklearn.classification.AutoSklearnClassifier()
cls.fit(X_train, y_train)
predictions = cls.predict(X_test)
```
## Relevant publications
If you use auto-sklearn in scientific publications, we would appreciate citations.
**Efficient and Robust Automated Machine Learning**
*Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum and Frank Hutter*
Advances in Neural Information Processing Systems 28 (2015)
[Link](https://papers.neurips.cc/paper/5872-efficient-and-robust-automated-machine-learning.pdf) to publication.
```
@inproceedings{feurer-neurips15a,
title = {Efficient and Robust Automated Machine Learning},
author = {Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina and Springenberg, Jost and Blum, Manuel and Hutter, Frank},
booktitle = {Advances in Neural Information Processing Systems 28 (2015)},
pages = {2962--2970},
year = {2015}
}
```
----------------------------------------
**Auto-Sklearn 2.0: The Next Generation**
*Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer and Frank Hutter**
arXiv:2007.04074 [cs.LG], 2020
[Link](https://arxiv.org/abs/2007.04074) to publication.
```
@article{feurer-arxiv20a,
title = {Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning},
author = {Feurer, Matthias and Eggensperger, Katharina and Falkner, Stefan and Lindauer, Marius and Hutter, Frank},
booktitle = {arXiv:2007.04074 [cs.LG]},
year = {2020}
}
```
----------------------------------------
Also, have a look at the blog on [automl.org](https://automl.org) where we regularly release blogposts.
---