Doc/ Templates/Class

:mod:{{module}}.{{objname}}
{{ underline }}==============

.. currentmodule:: {{ module }}

.. autoclass:: {{ objname }}

---

Doc/ Templates/Class Without Init

:mod:{{module}}.{{objname}}
{{ underline }}==============

.. currentmodule:: {{ module }}

.. autoclass:: {{ objname }}

.. include:: {{module}}.{{objname}}.examples

.. raw:: html

---

Doc/ Templates/Function

:mod:{{module}}.{{objname}}
{{ underline }}====================

.. currentmodule:: {{ module }}

.. autofunction:: {{ objname }}

.. raw:: html

---

Doc/Api

:orphan:

.. _api:

APIs

============
Main modules
============

~~~~~~~~~~~~~~
Classification
~~~~~~~~~~~~~~

.. autoclass:: autosklearn.classification.AutoSklearnClassifier
:members:
:inherited-members: show_models, fit_ensemble, refit, sprint_statistics

.. autoclass:: autosklearn.experimental.askl2.AutoSklearn2Classifier
:inherited-members: show_models, fit_ensemble, refit, sprint_statistics, fit, predict, predict_proba

~~~~~~~~~~
Regression
~~~~~~~~~~

.. autoclass:: autosklearn.regression.AutoSklearnRegressor
:members:
:inherited-members: show_models, fit_ensemble, refit, sprint_statistics

=======
Metrics
=======

.. autofunction:: autosklearn.metrics.make_scorer

~~~~~~~~~~~~~~~~
Built-in Metrics
~~~~~~~~~~~~~~~~

Classification metrics
~~~~~~~~~~~~~~~~~~~~~~
Note: The default `autosklearn.metrics.f1, autosklearn.metrics.precision and autosklearn.metrics.recallbuilt-in metrics are applicable only for binary classification. In order to apply them on multilabel and multiclass classification, please use the corresponding metrics with an appropriate averaging mechanism, such asautosklearn.metrics.f1_macro. For more information about how these metrics are used, please readthis scikit-learn documentation <https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures>_.

.. autoclass:: autosklearn.metrics.accuracy

.. autoclass:: autosklearn.metrics.balanced_accuracy

.. autoclass:: autosklearn.metrics.f1

.. autoclass:: autosklearn.metrics.f1_macro

.. autoclass:: autosklearn.metrics.f1_micro

.. autoclass:: autosklearn.metrics.f1_samples

.. autoclass:: autosklearn.metrics.f1_weighted

.. autoclass:: autosklearn.metrics.roc_auc

.. autoclass:: autosklearn.metrics.precision

.. autoclass:: autosklearn.metrics.precision_macro

.. autoclass:: autosklearn.metrics.precision_micro

.. autoclass:: autosklearn.metrics.precision_samples

.. autoclass:: autosklearn.metrics.precision_weighted

.. autoclass:: autosklearn.metrics.average_precision

.. autoclass:: autosklearn.metrics.recall

.. autoclass:: autosklearn.metrics.recall_macro

.. autoclass:: autosklearn.metrics.recall_micro

.. autoclass:: autosklearn.metrics.recall_samples

.. autoclass:: autosklearn.metrics.recall_weighted

.. autoclass:: autosklearn.metrics.log_loss

Regression metrics ~~~~~~~~~~~~~~~~~~

.. autoclass:: autosklearn.metrics.r2

.. autoclass:: autosklearn.metrics.mean_squared_error

.. autoclass:: autosklearn.metrics.mean_absolute_error

.. autoclass:: autosklearn.metrics.median_absolute_error

==================== Extension Interfaces ====================

.. autoclass:: autosklearn.pipeline.components.base.AutoSklearnClassificationAlgorithm :members:

.. autoclass:: autosklearn.pipeline.components.base.AutoSklearnRegressionAlgorithm :members:

.. autoclass:: autosklearn.pipeline.components.base.AutoSklearnPreprocessingAlgorithm :members:

.. _api_ensemble:

========= Ensembles =========

~~~~~~~~~~~~~~~~ Single objective ~~~~~~~~~~~~~~~~

.. autoclass:: autosklearn.ensembles.EnsembleSelection :members:

Single model classes ~~~~~~~~~~~~~~~~~~~~

These classes wrap a single model to provide a unified interface in Auto-sklearn.

.. autoclass:: autosklearn.ensembles.SingleBest :members:

.. autoclass:: autosklearn.ensembles.SingleModelEnsemble :members:

.. autoclass:: autosklearn.ensembles.SingleBestFromRunhistory :members:

~~~~~~~~~~~~~~~ Multi-objective ~~~~~~~~~~~~~~~

.. autoclass:: autosklearn.ensembles.MultiObjectiveDummyEnsemble :members:

---

`Doc/Extending`

:orphan:

.. _extending:

====================== Extending auto-sklearn ======================

auto-sklearn can be easily extended with new classification, regression and feature preprocessing methods. In order to do so, a user has to implement a wrapper class and register it to auto-sklearn. This manual will walk you through the process.

Writing a component ===================

Depending on the purpose, the component has to be a subclass of one of the following base classes:

* classification: :class:autosklearn.pipeline.components.base.AutoSklearnClassificationAlgorithm* regression: :class:autosklearn.pipeline.components.base.AutoSklearnRegressionAlgorithm* preprocessing: :class:autosklearn.pipeline.components.base.AutoSklearnPreprocessingAlgorithm

In general, these classes are wrappers around existing machine learning models and only add the functionality auto-sklearn needs. Of course you can also implement a machine learning algorithm directly inside a component.

Each component has to implement a method which returns its configuration space, a method for querying properties of the component and methods likefit(), predict() or transform()based on the task of the component. These are described in the subsections :ref:get_hyperparameter_search_space and :ref:get_properties

After writing a component class, you have to tell auto-sklearn about its existence. You have to add it with the following function calls, depending on the type of component:

.. autofunction:: autosklearn.pipeline.components.classification.add_classifier

.. autofunction:: autosklearn.pipeline.components.regression.add_regressor

.. autofunction:: autosklearn.pipeline.components.feature_preprocessing.add_preprocessor

.. _get_hyperparameter_search_space:

get_hyperparameter_search_space() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Return an instance of ConfigSpace.configuration_space.ConfigurationSpace.

See also the abstract definitions: :meth:AutoSklearnClassificationAlgorithm.get_hyperparameter_search_space() <autosklearn.pipeline.components.base.AutoSklearnClassificationAlgorithm.get_hyperparameter_search_space>:meth:AutoSklearnRegressionAlgorithm.get_hyperparameter_search_space() <autosklearn.pipeline.components.base.AutoSklearnRegressionAlgorithm.get_hyperparameter_search_space>:meth:AutoSklearnPreprocessingAlgorithm.get_hyperparameter_search_space() <autosklearn.pipeline.components.base.AutoSklearnPreprocessingAlgorithm.get_hyperparameter_search_space>

To find out about how to create a ConfigurationSpace-object, please look at the source code ongithub.com <https://github.com/automl/auto-sklearn/tree/master/autosklearn/pipeline/components/classification>_.

.. _get_properties:

get_properties() ~~~~~~~~~~~~~~~~

Return a dictionary which defines how the component can be used when constructing a machine learning pipeline. The following fields must be specified:

* shortname : str an abbreviation of the component * name : str the full name of the component * handles_regression : bool whether the component can handle regression data * handles_classification : bool whether the component can handle classification data * handles_multiclass : bool whether the component can handle multiclass classification data * handles_multilabel : bool whether the component can multilabel classification data * is_deterministic : bool whether the component gives the same result when using several times, but with the same random seed * input : tuple type of input data the component can handle, can have multiple values:

* autosklearn.constants.DENSE dense data arrays, mutually exclusive with autosklearn.constants.SPARSE * autosklearn.constants.SPARSE sparse data matrices, mutually exclusive with autosklearn.constants.DENSE * autosklearn.constants.UNSIGNED_DATA unsigned data array, meaning only positive input, mutually exclusive with autosklearn.constants.SIGNED_DATA * autosklearn.constants.SIGNED_DATA signed data array, meaning both positive and negative input values, mutually exclusive with autosklearn.constants.UNSIGNED_DATA * output : tuple type of output data the component produces

* autosklearn.constants.PREDICTIONS predictions, for example by a classifier * autosklearn.constants.INPUT data in the same form as the input * autosklearn.constants.DENSE dense data arrays, mutually exclusive with autosklearn.constants.SPARSE. This implies that sparse data will be converted into a dense representation. * autosklearn.constants.SPARSE sparse data matrices, mutually exclusive with autosklearn.constants.DENSE. This implies that dense data will be converted into a sparse representation * autosklearn.constants.UNSIGNED_DATA unsigned data array, meaning only positive input, mutually exclusive with autosklearn.constants.SIGNED_DATA. This allows for algorithms which can only work on positive data. * autosklearn.constants.SIGNED_DATA signed data array, meaning both positive and negative input values, mutually exclusive with autosklearn.constants.UNSIGNED_DATA

Classification ==============

In addition two get_properties() and get_hyperparameter_search_space()you have to implement :meth:AutoSklearnClassificationAlgorithm.fit() <autosklearn.pipeline.components.base.AutoSklearnClassificationAlgorithm.fit>and :meth:AutoSklearnClassificationAlgorithm.predict() <autosklearn.pipeline.components.base.AutoSklearnClassificationAlgorithm.predict>. These are an implementation of thescikit-learn predictor API
<https://scikit-learn.org/stable/developers/develop.html#apis-of-scikit-learn-objects>_.

Regression ==========

In addition two get_properties() and get_hyperparameter_search_space()you have to implement :meth:AutoSklearnRegressionAlgorithm.fit() <autosklearn.pipeline.components.base.AutoSklearnRegressionAlgorithm.fit>and :meth:AutoSklearnRegressionAlgorithm.predict() <autosklearn.pipeline.components.base.AutoSklearnRegressionAlgorithm.predict>. These are an implementation of thescikit-learn predictor API
<https://scikit-learn.org/stable/developers/develop.html#apis-of-scikit-learn-objects>_.

Feature Preprocessing =====================

In addition two get_properties() and get_hyperparameter_search_space()you have to implement :meth:AutoSklearnPreprocessingAlgorithm.fit() <autosklearn.pipeline.components.base.AutoSklearnPreprocessingAlgorithm.fit>and :meth:AutoSklearnPreprocessingAlgorithm.transform() <autosklearn.pipeline.components.base.AutoSklearnPreprocessingAlgorithm.transform>. These are an implementation of thescikit-learn predictor API
<https://scikit-learn.org/stable/developers/develop.html#apis-of-scikit-learn-objects>_.

---

`Doc/Faq`

:orphan:

.. _faq:

=== FAQ ===

General =======

.. collapse:: Where can I find examples on how to use auto-sklearn?

We provide examples on using auto-sklearn for multiple use cases ranging from simple classification to advanced uses such as feature importance, parallel runs and customization. They can be found in the :ref:examples.

.. collapse:: What type of tasks can auto-sklearn tackle?

auto-sklearn can accept targets for the following tasks (more details on Sklearn algorithms <https://scikit-learn.org/stable/modules/multiclass.html>_):

* Binary Classification * Multiclass Classification * Multilabel Classification * Regression * Multioutput Regression

You can provide feature and target training pairs (X_train/y_train) to auto-sklearn to fit an ensemble of pipelines as described in the next section. This X_train/y_train dataset must belong to one of the supported formats: np.ndarray, pd.DataFrame, scipy.sparse.csr_matrix and python lists. Optionally, you can measure the ability of this fitted model to generalize to unseen data by providing an optional testing pair (X_test/Y_test). For further details, please refer to the Example :ref:sphx_glr_examples_40_advanced_example_pandas_train_test.py.

Regarding the features, there are multiple things to consider:

* Providing a X_train/X_test numpy array with the optional flag feat_type. For further details, you can check the Example :ref:sphx_glr_examples_40_advanced_example_feature_types.py. * You can provide a pandas DataFrame with properly formatted columns. If a column has numerical dtype, auto-sklearn will not encode it and it will be passed directly to scikit-learn. auto-sklearn supports both categorical or string as column type. Please ensure that you are using the correct dtype for your task. By default auto-sklearn treats object and string columns as strings and encodes the data usingsklearn.feature_extraction.text.CountVectorizer <https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html>_ * If your data contains categorical values (in the features or targets), ensure that you explicitly label them as categorical. Data labeled as categorical is encoded by using asklearn.preprocessing.LabelEncoder <https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html>_ for unidimensional data and asklearn.preprodcessing.OrdinalEncoder <https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html>_ for multidimensional data. * For further details on how to properly encode your data, you can check the Pandas ExampleWorking with categorical data <https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html>_). If you are working with time series, it is recommended that you follow this approachWorking with time data <https://stats.stackexchange.com/questions/311494/>_. * If you prefer not using the string option at all you can disable this option. In this case objects, strings and categorical columns are encoded as categorical.

.. code:: python

import autosklearn.classification automl = autosklearn.classification.AutoSklearnClassifier(allow_string_features=False) automl.fit(X_train, y_train)

Regarding the targets (y_train/y_test), if the task involves a classification problem, such features will be automatically encoded. It is recommended to provide both y_train and y_test during fit, so that a common encoding is created between these splits (if only y_train is provided during fit, the categorical encoder will not be able to handle new classes that are exclusive to y_test). If the task is regression, no encoding happens on the targets.

.. collapse:: Where can I find slides and notebooks from talks and tutorials?

We provide resources for talks, tutorials and presentations on auto-sklearn under auto-sklearn-talks <https://github.com/automl/auto-sklearn-talks>_

.. collapse:: How should I cite auto-sklearn in a scientific publication?

If you've used auto-sklearn in scientific publications, we would appreciate citations.

.. code-block::

@inproceedings{feurer-neurips15a, title = {Efficient and Robust Automated Machine Learning}, author = {Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina Springenberg, Jost and Blum, Manuel and Hutter, Frank}, booktitle = {Advances in Neural Information Processing Systems 28 (2015)}, pages = {2962--2970}, year = {2015} }

Or this, if you've used auto-sklearn 2.0 in your work:

.. code-block::

@article{feurer-arxiv20a, title = {Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning}, author = {Feurer, Matthias and Eggensperger, Katharina and Falkner, Stefan and Lindauer, Marius and Hutter, Frank}, booktitle = {arXiv:2007.04074 [cs.LG]}, year = {2020} }

.. collapse:: I want to contribute. What can I do?

This sounds great. Please have a look at our contribution guide <https://github.com/automl/auto-sklearn/blob/master/CONTRIBUTING.md>_

.. collapse:: I have a question which is not answered here. What should I do?

Thanks a lot. We regularly update this section with questions from our issue tracker. So please use theissue tracker <https://github.com/automl/auto-sklearn/issues>_

Resource Management ===================

.. collapse:: How should I set the time and memory limits?

While auto-sklearn alleviates manual hyperparameter tuning, the user still has to set memory and time limits. For most datasets a memory limit of 3GB or 6GB as found on most modern computers is sufficient. For the time limits it is harder to give clear guidelines. If possible, a good default is a total time limit of one day, and a time limit of 30 minutes for a single run.

Further guidelines can be found inauto-sklearn/issues/142 <https://github.com/automl/auto-sklearn/issues/142>_.

.. collapse:: How many CPU cores does auto-sklearn use by default?

By default, auto-sklearn uses one core. See also :ref:parallel on how to configure this.

.. collapse:: How can I run auto-sklearn in parallel?

Nevertheless, auto-sklearn also supports parallel Bayesian optimization via the use ofDask.distributed <https://distributed.dask.org/>_. By providing the arguments n_jobsto the estimator construction, one can control the number of cores available to auto-sklearn (As shown in the Example :ref:sphx_glr_examples_60_search_example_parallel_n_jobs.py). Distributed processes are also supported by providing a custom client object to auto-sklearn like in the Example: :ref:sphx_glr_examples_60_search_example_parallel_manual_spawning_cli.py. When multiple cores are available, auto-sklearn will create a worker per core, and use the available workers to both search for better machine learning models as well as building an ensemble with them until the time resource is exhausted.

Note: auto-sklearn requires all workers to have access to a shared file system for storing training data and models.

auto-sklearn employs threadpoolctl <https://github.com/joblib/threadpoolctl/>_ to control the number of threads employed by scientific libraries like numpy or scikit-learn. This is done exclusively during the building procedure of models, not during inference. In particular, auto-sklearn allows each pipeline to use at most 1 thread during training. At predicting and scoring time this limitation is not enforced by auto-sklearn. You can control the number of resources employed by the pipelines by setting the following variables in your environment, prior to running auto-sklearn:

.. code-block:: shell-session

$ export OPENBLAS_NUM_THREADS=1 $ export MKL_NUM_THREADS=1 $ export OMP_NUM_THREADS=1

For further information about how scikit-learn handles multiprocessing, please check the Parallelism, resource management, and configuration <https://scikit-learn.org/stable/computing/parallelism.html>_ documentation from the library.

.. collapse:: Auto-sklearn is extremely memory hungry in a sequential setting

Auto-sklearn can appear very memory hungry (i.e. requiring a lot of memory for small datasets) due to the use offorkfor creating new processes when running in sequential manner (if this happens in a parallel setting or if you pass your own dask client this is due to a different issue, see the other issues below).

Let's go into some more detail and discuss how to fix it: Auto-sklearn executes each machine learning algorithm in its own process to be able to apply a memory limit and a time limit. To start such a process, Python gives three options:fork,forkserver and spawn. The default forkcopies the whole process memory into the subprocess. If the main process already uses 1.5GB of main memory and we apply a 3GB memory limit to Auto-sklearn, executing a machine learning pipeline is limited to use at most 1.5GB. We would have loved to useforkserver or spawnas the default option instead, which both copy only relevant data into the subprocess and thereby alleaviate the issue of eating up a lot of your main memory (and also do not suffer from potential deadlocks asforkdoes, seehere <https://pythonspeed.com/articles/python-multiprocessing/>_), but they have the downside that code must be guarded byif __name__ == "__main__"or executed in a notebook, and we decided that we do not want to require this by default.

There are now two possible solutions:

1. Use Auto-sklearn in parallel: if you use Auto-sklean in parallel, it defaults to forkserveras the parallelization mechanism itself requires Auto-sklearn the code to be guarded. Please find more information on how to do this in the following two examples:

1. :ref:sphx_glr_examples_60_search_example_parallel_n_jobs.py2. :ref:sphx_glr_examples_60_search_example_parallel_manual_spawning_cli.py

.. note::

This requires all code to be guarded by if __name__ == "__main__".

2. Pass a dask client <https://distributed.dask.org/en/latest/client.html>_. If the user passes a dask client, Auto-sklearn can no longer assume that it runs in sequential mode and will use aforkserver to start new processes.

.. note::

This requires all code to be guarded by if __name__ == "__main__".

We therefore suggest using one of the above settings by default.

.. collapse:: Auto-sklearn is extremely memory hungry in a parallel setting

When running Auto-sklearn in a parallel setting it starts new processes for evaluating machine learning models using theforkservermechanism. Code that is in the main script and that is not guarded byif __name__ == "__main__"will be executed for each subprocess. If, for example, you are loading your dataset outside of the guarded code, your dataset will be loaded for each evaluation of a machine learning algorithm and thus blocking your RAM.

We therefore suggest moving all code inside functions or the main block.

.. collapse:: Auto-sklearn crashes with a segmentation fault

Please make sure that you have read and followed the :ref:installationsection! In case everything is set up correctly, this is most likely due to the dependencypyrfr <https://github.com/automl/random_forest_run>_ not being compiled correctly. If this is the case please execute:

.. code:: python

import pyrfr.regression as reg data = reg.default_data_container(64)

If this fails, the pyrfr dependency is most likely not compiled correctly. We advice you to do the following:

1. Check if you can use a pre-compiled version of the pyrfr to avoid compiling it yourself. We provide pre-compiled versions of the pyrfr onpypi <https://pypi.org/project/pyrfr/#files>_. 2. Check if the dependencies specified under :ref:installationare correctly installed, especially that you haveswig and a C++compiler. 3. If you are not yet using Conda, consider using it; it simplifies installation of the correct dependencies. 4. Install correct build dependencies before installing the pyrfr, you can check the following github issues for suggestions:1025 <https://github.com/automl/auto-sklearn/issues/1025>_,856 <https://github.com/automl/auto-sklearn/issues/856>_

Results, Log Files and Output =============================

.. collapse:: How can I get an overview of the run statistics?

sprint_statistics()is a method that prints the name of the dataset, the metric used, and the best validation score obtained by running auto-sklearn. It additionally prints the number of both successful and unsuccessful algorithm runs.

.. collapse:: What was the performance over time?

performance_over_time_returns a DataFrame containing the models performance over time data, which can be used for plotting directly (Here is an example: :ref:sphx_glr_examples_40_advanced_example_pandas_train_test.py).

.. code:: python

automl.performance_over_time_.plot( x='Timestamp', kind='line', legend=True, title='Auto-sklearn accuracy over time', grid=True, ) plt.show()

.. collapse:: Which models were evaluated?

You can see all models evaluated using :meth:automl.leaderboard(ensemble_only=False) <autosklearn.classification.AutoSklearnClassifier.leaderboard>.

.. collapse:: Which models are in the final ensemble?

Use either :meth:automl.leaderboard(ensemble_only=True) <autosklearn.classification.AutoSklearnClassifier.leaderboard> or automl.show_models()

.. collapse:: Is there more data I can look at?

cv_results_returns a dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame, e.g.df = pd.DataFrame(automl.cv_results_)

.. collapse:: Where does Auto-sklearn output files by default?

Auto-sklearn heavily uses the hard drive to store temporary data, models and log files which can be used to inspect the behavior of Auto-sklearn. Each run of Auto-sklearn requires its own directory. If not provided by the user, Auto-sklearn requests a temporary directory from Python, which by default is located under/tmp and starts with autosklearn_tmp_followed by a random string. By default, this directory is deleted when the Auto-sklearn object is finished fitting. If you want to keep these files you can pass the argumentdelete_tmp_folder_after_terminate=True to the Auto-sklearn object.

The :class:autosklearn.classification.AutoSklearnClassifierand all other auto-sklearn estimators accept the argumenttmp_folder which change where such output is written to.

There's an additional argument output_directorywhich can be passed to Auto-sklearn and it controls where test predictions of the ensemble are stored if the test set is passed tofit().

.. collapse:: Auto-sklearn's logfiles eat up all my disk space. What can I do?

Auto-sklearn heavily uses the hard drive to store temporary data, models and log files which can be used to inspect the behavior of Auto-sklearn. By default, Auto-sklearn stores 50 models and their predictions on the validation data (which is a subset of the training data in case of holdout and the full training data in case of cross-validation) on the hard drive. Redundant models and their predictions (i.e. when we have more than 50 models) are removed everytime the ensemble builder finishes an iteration, which means that the number of models stored on disk can temporarily be higher if a model is output while the ensemble builder is running.

One can therefore change the number of models that will be stored on disk by passing an integer for the argumentmax_models_on_discto Auto-sklearn, for example reduce the number of models stored on disk if you have space issues.

As the number of models is only an indicator of the disk space used it is also possible to pass the memory in MB the models are allowed to use as afloat (also via the max_models_on_discarguments). As above, this is rather a guideline on how much memory is used as redundant models are only removed from disk when the ensemble builder finishes an iteration.

.. note::

Especially when running in parallel it can happen that multiple models are constructed during one run of the ensemble builder and thus Auto-sklearn can exceed the given limit.

.. note::

These limits do only apply to models and their predictions, but not to other files stored in the temporary directory such as the log files.

The Search Space ================

.. collapse:: How can I restrict the searchspace?

The following shows an example of how to exclude all preprocessing methods and restrict the configuration space to only random forests.

.. code:: python

import autosklearn.classification automl = autosklearn.classification.AutoSklearnClassifier( include = { 'classifier': ["random_forest"], 'feature_preprocessor': ["no_preprocessing"] }, exclude=None ) automl.fit(X_train, y_train) predictions = automl.predict(X_test)

Note: The strings used to identify estimators and preprocessors are the filenames without .py.

For a full list please have a look at the source code (in autosklearn/pipeline/components/):

* Classifiers <https://github.com/automl/auto-sklearn/tree/master/autosklearn/pipeline/components/classification>_ *Regressors <https://github.com/automl/auto-sklearn/tree/master/autosklearn/pipeline/components/regression>_ *Preprocessors <https://github.com/automl/auto-sklearn/tree/master/autosklearn/pipeline/components/feature_preprocessing>_

We do also provide an example on how to restrict the classifiers to search over :ref:sphx_glr_examples_40_advanced_example_interpretable_models.py.

.. collapse:: How can I turn off data preprocessing?

Data preprocessing includes One-Hot encoding of categorical features, imputation of missing values and the normalization of features or samples. These ensure that the data the gets to the sklearn models is well formed and can be used for training models.

While this is necessary in general, if you'd like to disable this step, please refer to this :ref:example <sphx_glr_examples_80_extending_example_extending_data_preprocessor.py>.

.. collapse:: How can I turn off feature preprocessing?

Feature preprocessing is a single transformer which implements for example feature selection or transformation of features into a different space (i.e. PCA).

This can be turned off by settinginclude={'feature_preprocessor'=["no_preprocessing"]} as shown in the example above.

.. collapse:: Will non-scikit-learn models be added to Auto-sklearn?

The short answer: no.

The long answer answer is a bit more nuanced: maintaining Auto-sklearn requires a lot of time and effort, which would grow even larger when depending on more libraries. Also, adding more libraries would require us to generate meta-data more often. Lastly, having more choices does not guarantee a better performance for most users as having more choices demands a longer search for good models and can lead to more overfitting.

Nevertheless, everyone can still add their favorite model to Auto-sklearn's search space by following theexamples on how to extend Auto-sklearn
<https://automl.github.io/auto-sklearn/master/examples/index.html#extension-examples>_.

If there is interest in creating a Auto-sklearn-contrib repository with 3rd-party models please open an issue for that.

.. collapse:: How can I only search for interpretable models

Auto-sklearn can be restricted to only use interpretable models and preprocessing algorithms. Please see the Section :ref:spaceto learn how to restrict the models which are searched over or see the Example :ref:sphx_glr_examples_40_advanced_example_interpretable_models.py.

We don't provide a judgement which of the models are interpretable as this is very much up to the specific use case, but would like to note that decision trees and linear models usually most interpretable.

Ensembling ==========

.. collapse:: What can I configure wrt the ensemble building process?

The following hyperparameters control how the ensemble is constructed:

* ensemble_class class object implementing :class:autosklearn.ensembles.AbstractEnsemble, will be instantiated by auto-sklearn's ensemble builder. *ensemble_kwargs are keyword arguments that are passed to the ensemble_classupon instantiation. See below for an example argument.ensemble_nbest allows the user to directly specify the number of models considered for the ensemble. This hyperparameter can be an integer n, such that only the best n* models are used in the final ensemble. If a float between 0.0 and 1.0 is provided, ensemble_nbest would be interpreted as a fraction suggesting the percentage of models to use in the ensemble building process (namely, if ensemble_nbest is a float, library pruning is implemented as described in Caruana et al. (2006) <https://dl.acm.org/doi/10.1109/ICDM.2006.76>_).max_models_on_disc defines the maximum number of models that are kept on the disc, as a mechanism to control the amount of disc space consumed by auto-sklearn*. Throughout the automl process, different individual models are optimized, and their predictions (and other metadata) is stored on disc. The user can set the upper bound on how many models are acceptable to keep on disc, yet this variable takes priority in the definition of the number of models used by the ensemble builder (that is, the minimum of ensemble_size, ensemble_nbest and max_models_on_disc determines the maximal amount of models used in the ensemble). If set to None, this feature is disabled.

The default method for Auto-sklearn is :class:autosklearn.ensembles.EnsembleSelection, which features the argumentensemble_size. that determines the maximal size of the ensemble. Models can be added repeatedly, so the number of different models is usually less than theensemble_size.

.. collapse:: Which models are in the final ensemble?

The results obtained from the final ensemble can be printed by calling show_models() or leaderboard(). The auto-sklearn ensemble is composed of scikit-learn models that can be inspected as exemplified in the Example :ref:sphx_glr_examples_40_advanced_example_get_pipeline_components.py.

.. collapse:: Can I fit an ensemble also only post-hoc?

It is possible to build ensembles post-hoc. An example on how to do this (first searching for individual models, and then building an ensemble from them) can be seen in :ref:sphx_glr_examples_60_search_example_sequential.py.

Configuring the Search Procedure ================================

.. collapse:: Can I change the resampling strategy?

Examples for using holdout and cross-validation can be found in :ref:example <sphx_glr_examples_40_advanced_example_resampling.py>

If using a custom resampling strategy with predefined splits, you may need to disable the subsampling performed with particularly large datasets or if using a smallmemory_limit. Please see the manual section on :ref:limits:class:AutoSklearnClassifier(dataset_compression=...) <autosklearn.classification.AutoSklearnClassifier>. for more details.

.. collapse:: Can I use a custom metric

Examples for using a custom metric can be found in :ref:example <sphx_glr_examples_40_advanced_example_metrics.py>

Meta-Learning =============

.. collapse:: Which datasets are used for meta-learning?

We updated the list of datasets used for meta-learning several times and this list now differs significantly from the original 140 datasets we used in 2015 when the paper and the package were released. An up-to-date list ofOpenML task IDs <https://docs.openml.org/#tasks>_ can be found ongithub <https://github.com/automl/auto-sklearn/blob/master/scripts/update_metadata_util.py>_.

.. collapse:: How can datasets from the meta-data be excluded?

For Auto-sklearn 1.0 one can pass the dataset name via the fit()function. If a dataset with the same name is within the meta-data, that datasets will not be used.

For Auto-sklearn 2.0 it is not possible to do so because of the method used to construct the meta-data.

.. collapse:: Which meta-features are used for meta-learning?

We do not have a user guide on meta-features but they are all pretty simple and can be foundin the source code <https://github.com/automl/auto-sklearn/blob/master/autosklearn/metalearning/metafeatures/metafeatures.py>_.

.. collapse:: How is the meta-data generated for Auto-sklearn 1.0?

We currently generate meta-data the following way. First, for each of the datasets mentioned above, we run Auto-sklearn without meta-learning for a total of two days on multiple metrics (for classification these are accuracy, balanced accuracy, log loss and the area under the curce). Second, for each run we then have a look at each models that improved the score, i.e. the trajectory of the best known model at a time, and refit it on the whole training data. Third, for each of these models we then compute all scores we're interested in, these also include other ones such F1 and precision. Finally, for each combination of dataset and metric we store the best model we know of.

.. collapse:: How is the meta-data generated for Auto-sklearn 2.0?

Please check our paper <https://arxiv.org/abs/2007.04074>_ for details.

Issues and Debugging ====================

.. collapse:: How can I limit the number of model evaluations for debugging?

In certain cases, for example for debugging, it can be helpful to limit the number of model evaluations. We do not provide this as an argument in the API as we believe that it should NOT be used in practice, but that the user should rather provide time limits. An example on how to add the number of models to try as an additional stopping condition can be foundin this github issue <https://github.com/automl/auto-sklearn/issues/451#issuecomment-376445607>_. Please note that Auto-sklearn will stop when either the time limit or the number of models termination condition is reached.

.. collapse:: Why does the final ensemble contains only a dummy model?

This is a symptom of the problem that all runs started by Auto-sklearn failed. Usually, the issue is that the runtime or memory limit were too tight. Please check the output ofsprint_statistics()to see the distribution of why runs failed. If there are mostly crashed runs, please check the log file for further details. If there are mostly runs that exceed the memory or time limit, please increase the respective limit and rerun the optimization.

.. collapse:: Auto-sklearn does not use the specified amount of resources?

Auto-sklearn wraps scikit-learn and therefore inherits its parallelism implementation. In short, scikit-learn uses two modes of parallelizing computations:

1. By using joblib to distribute independent function calls on multiple cores. 2. By using lower level libraries such as OpenMP and numpy to distribute more fine-grained computation.

This means that Auto-sklearn can use more resources than expected by the user. For technical reasons we can only control the 1st way of parallel execution, but not the 2nd. Thus, the user needs to make sure that the lower level parallelization libraries only use as many cores as allocated (on a laptop or workstation running a single copy of Auto-sklearn it can be fine to not adjust this, but when using a compute cluster it is necessary to align the parallelism setting with the number of requested CPUs). This can be done by setting the following environment variables:MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, BLIS_NUM_THREADSandOMP_NUM_THREADS.

More details can be found in the scikit-learn docs <https://scikit-learn.org/stable/computing/parallelism.html?highlight=joblib#parallelism>_.

Other =====

.. collapse:: Model persistence

auto-sklearn is mostly a wrapper around scikit-learn. Therefore, it is possible to follow thepersistence Example <https://scikit-learn.org/stable/modules/model_persistence.html>_ from scikit-learn.

.. collapse:: Vanilla auto-sklearn

In order to obtain vanilla auto-sklearn as used in Efficient and Robust Automated Machine Learning
<https://papers.neurips.cc/paper/5872-efficient-and-robust-automated-machine-learning>_ setensemble_class=autosklearn.ensembles.SingleBest and initial_configurations_via_metalearning=0:

.. code:: python

import autosklearn.classification import autosklearn.ensembles automl = autosklearn.classification.AutoSklearnClassifier( ensemble_class=autosklearn.ensembles.SingleBest, initial_configurations_via_metalearning=0 )

This will always choose the best model according to the validation set. Setting the initial configurations found by meta-learning to zero makes auto-sklearn use the regular SMAC algorithm for suggesting new hyperparameter configurations.

---

`Doc/Index`

auto-sklearn

.. role:: bash(code) :language: bash

.. role:: python(code) :language: python

auto-sklearn is an automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator:

.. code:: python

import autosklearn.classification cls = autosklearn.classification.AutoSklearnClassifier() cls.fit(X_train, y_train) predictions = cls.predict(X_test)

auto-sklearn frees a machine learning user from algorithm selection and hyperparameter tuning. It leverages recent advantages in *Bayesian optimization, meta-learning and ensemble construction*. Learn more about the technology behind auto-sklearn by reading our paper published atNeurIPS 2015 <https://papers.neurips.cc/paper/5872-efficient-and-robust-automated-machine-learning.pdf>_ .

.. topic:: NEW: Text feature support

Auto-sklearn now supports text features, check our new example: :ref:sphx_glr_examples_40_advanced_example_text_preprocessing.py

Example *

.. code:: python

import autosklearn.classification import sklearn.model_selection import sklearn.datasets import sklearn.metrics

if __name__ == "__main__": X, y = sklearn.datasets.load_digits(return_X_y=True) X_train, X_test, y_train, y_test = \ sklearn.model_selection.train_test_split(X, y, random_state=1) automl = autosklearn.classification.AutoSklearnClassifier() automl.fit(X_train, y_train) y_hat = automl.predict(X_test) print("Accuracy score", sklearn.metrics.accuracy_score(y_test, y_hat))

This will run for one hour and should result in an accuracy above 0.98.

Manual

* :ref:installation* :ref:manual* :ref:api* :ref:extending* :ref:faq

Additional Material *

We provide slides and notebooks from talks and tutorials here:auto-sklearn-talks <https://github.com/automl/auto-sklearn-talks>_

License * auto-sklearn is licensed the same way as scikit-learn, namely the 3-clause BSD license.

Citing auto-sklearn *

If you use auto-sklearn in a scientific publication, we would appreciate a reference to the following paper:

Efficient and Robust Automated Machine Learning
<https://papers.neurips.cc/paper/5872-efficient-and-robust-automated-machine-learning>_, Feurer et al., Advances in Neural Information Processing Systems 28 (NIPS 2015).

Bibtex entry::

@inproceedings{feurer-neurips15a, title = {Efficient and Robust Automated Machine Learning}, author = {Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina and Springenberg, Jost and Blum, Manuel and Hutter, Frank}, booktitle = {Advances in Neural Information Processing Systems 28 (2015)}, pages = {2962--2970}, year = {2015} }

If you are using Auto-sklearn 2.0, please also cite

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning <https://arxiv.org/abs/2007.04074>_, Feurer et al., (arXiv, 2020).

Bibtex entry::

@article{feurer-arxiv20a, title = {Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning}, author = {Feurer, Matthias and Eggensperger, Katharina and Falkner, Stefan and Lindauer, Marius and Hutter, Frank}, journal = {arXiv:2007.04074 [cs.LG]}, year = {2020}, }

Contributing

We appreciate all contribution to auto-sklearn, from bug reports and documentation to new features. If you want to contribute to the code, you can pick an issue from theissue tracker <https://github.com/automl/auto-sklearn/issues>_.

Check out our contribution guide on github <https://github.com/automl/auto-sklearn/blob/master/CONTRIBUTING.md>_ if you want to know more! We've catered it for both new and experienced contributers.

.. note::

To avoid spending time on duplicate work or features that are unlikely to get merged, it is highly advised that you contact the developers by opening agithub issue <https://github
.com/automl/auto-sklearn/issues>_ before starting to work.

---

`Doc/Installation`

:orphan:

.. _installation:

============ Installation ============

System requirements ===================

auto-sklearn has the following system requirements:

* Linux operating system (for example Ubuntu) (get Linux here <https://www.wikihow.com/Install-Linux>_) * Python (>=3.7) (get Python here <https://www.python.org/downloads/>_), * C++ compiler (with C++11 supports) (get GCC here <https://www.tutorialspoint.com/How-to-Install-Cplusplus-Compiler-on-Linux>_).

In case you try to install Auto-sklearn on a system where no wheel files for the pyrfr package are provided (seehere <https://pypi.org/project/pyrfr/#files>_ for available wheels) you also need:

* SWIG (get SWIG here <http://www.swig.org/survey.html>_).

For an explanation of missing Microsoft Windows and macOS support please check the SectionWindows/macOS compatibility_.

Installing auto-sklearn =======================

You can install auto-sklearn with pip in the usual manner:

.. code:: bash

pip3 install auto-sklearn

We recommend installing auto-sklearn into avirtual environment <https://docs.python-guide.org/dev/virtualenvs/>_ or anAnaconda environment <https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>_.

If the pip3 installation command fails, make sure you have the System requirements_ installed correctly.

Ubuntu installation ===================

To provide Python 3, a C++11 building environment and the latest SWIG version on Ubuntu, run:

.. code:: bash

sudo apt-get install build-essential swig python3-dev

Anaconda installation =====================

You need to enable conda-forge to install auto-sklearn via anaconda. This section explains how to enable conda-forge so installation can be done with the commandconda install auto-sklearn. Optionally, you can also install auto-sklearn withpip as detailed in the Section Installing auto-sklearn_.

A common installation problem under recent Linux distribution is the incompatibility of the compiler version used to compile the Python binary shipped by AnaConda and the compiler installed by the distribution. This can be solved by installing the gcc compiler shipped with AnaConda (as well as swig):

.. code:: bash

conda install gxx_linux-64 gcc_linux-64 swig

Conda-forge ~~~~~~~~~~~

Installing auto-sklearn from the conda-forge channel can be achieved by adding conda-forge to your channels with:

.. code:: bash

conda config --add channels conda-forge

conda config --set channel_priority strict

You must haveconda >=4.9. To update conda or check your current conda version, please follow the instructions from the official anaconda documentation <https://docs.anaconda.com/anaconda/install/update-version/>_ . Once the conda-forge channel has been enabled, auto-sklearn can be installed with:

.. code:: bash

conda install auto-sklearn

It is possible to list all of the versions ofauto-sklearn available on your platform with:

.. code:: bash

conda search auto-sklearn --channel conda-forge

to read in more details checkauto sklearn feedstock <https://github.com/conda-forge/auto-sklearn-feedstock>_.

for more information about Conda forge checkconda-forge documentations <https://conda-forge.org/docs/>_.

Source Installation ===================

You can install auto-sklearn directly form source by following the below:

.. code:: bash

git clone --recurse-submodules [email protected]:automl/auto-sklearn.git cd auto-sklearn

# Install it in editable mode with all optional dependencies pip install -e ".[test,doc,examples]"

We use submodules so you will have to make sure the submodule is initialized if you missed the--recurse-submodules option.

.. code:: bash

git clone [email protected]:automl/auto-sklearn.git cd auto-sklearn git submodule update --init --recursive

pip install -e ".[test,doc,examples]"

Windows/macOS compatibility ===========================

Windows ~~~~~~~

auto-sklearn relies heavily on the Python module resource. resourceis part of Python'sUnix Specific Services <https://docs.python.org/3/library/unix.html>_ and not available on a Windows machine. Therefore, it is not possible to run auto-sklearn on a Windows machine.

Possible solutions:

* Windows 10 bash shell (see 431 <https://github.com/automl/auto-sklearn/issues/431>_ and860 <https://github.com/automl/auto-sklearn/issues/860>_ for suggestions) * virtual machine * docker image

macOS ~~~~~

We currently do not know if auto-sklearn works on macOS. There are at least two issues holding us back from actively supporting macOS:

* The resourcemodule cannot enforce a memory limit on a Python process (seeSMAC3/issues/115 <https://github.com/automl/SMAC3/issues/115>_). * Not all dependencies we are using are set up to work on macOS.

In case you're having issues installing the pyrfr package <https://github.com/automl/random_forest_run>_, check outthis installation suggestion on github <https://github.com/automl/auto-sklearn/issues/360#issuecomment-335150470>_.

Possible other:

* virtual machine * docker image

Docker Image ============ A Docker image is also provided on dockerhub. To download from dockerhub, use:

.. code:: bash

docker pull mfeurer/auto-sklearn:master

You can also verify that the image was downloaded via:

.. code:: bash

docker images # Verify that the image was downloaded

This image can be used to start an interactive session as follows:

.. code:: bash

docker run -it mfeurer/auto-sklearn:master

To start a Jupyter notebook, you could instead run e.g.:

.. code:: bash

docker run -it -v ${PWD}:/opt/nb -p 8888:8888 mfeurer/auto-sklearn:master /bin/bash -c "mkdir -p /opt/nb && jupyter notebook --notebook-dir=/opt/nb --ip='0.0.0.0' --port=8888 --no-browser --allow-root"

Alternatively, it is possible to use the development version of auto-sklearn by replacing all occurences ofmaster by development.

---

`Doc/Manual`

:orphan:

.. _manual:

====== Manual ======

This manual gives an overview of different aspects of auto-sklearn. For each section, we either references examples or give short explanations (click the title to expand text), e.g.

.. collapse:: Code examples

.. collapse:: Material from talks and presentations

We provide resources for talks, tutorials and presentations on auto-sklearn under auto-sklearn-talks <https://github.com/automl/auto-sklearn-talks>_

.. _askl2:

Auto-sklearn 2.0 ================

Auto-sklearn 2.0 includes latest research on automatically configuring the AutoML system itself and contains a multitude of improvements which speed up the fitting the AutoML system. Concretely, Auto-sklearn 2.0 automatically sets the :ref:bestmodel, decides whether it can use the efficient bandit strategy Successive Halving and uses meta-feature free Portfolios for efficient meta-learning.

auto-sklearn 2.0 has the same interface as regular auto-sklearn and you can use it via

.. code:: python

from autosklearn.experimental.askl2 import AutoSklearn2Classifier

A paper describing our advances is available on arXiv <https://arxiv.org/abs/2007.04074>_.

.. _limits:

Resource limits ===============

A crucial feature of auto-sklearn is limiting the resources (memory and time) which the scikit-learn algorithms are allowed to use. Especially for large datasets, on which algorithms can take several hours and make the machine swap, it is important to stop the evaluations after some time in order to make progress in a reasonable amount of time. Setting the resource limits is therefore a tradeoff between optimization time and the number of models that can be tested.

.. collapse:: Time and memory limits

Further guidelines can be found inauto-sklearn/issues/142 <https://github.com/automl/auto-sklearn/issues/142>_.

.. collapse:: CPU cores

By default, auto-sklearn uses one core. See also :ref:parallel on how to configure this.

.. collapse:: Managing data compression

.. _manual_managing_data_compression:

Auto-sklearn will attempt to fit the dataset into 1/10th of the memory_limit. This won't happen unless your dataset is quite large or you have small amemory_limit. This is done using two methods, reducing precision and to subsample. One reason you may want to control this is if you require high precision or you rely on predefined splits for which subsampling does not account for.

To turn off data preprocessing:

.. code:: python

AutoSklearnClassifier( dataset_compression = False )

You can specify which of the methods are performed using:

.. code:: python

AutoSklearnClassifier( dataset_compression = { "methods": ["precision", "subsample"] }, )

You can change the memory allocation for the dataset to a percentage of memory_limitor an absolute amount using:

.. code:: python

AutoSklearnClassifier( dataset_compression = { "memory_allocation": 0.2 }, )

The default arguments are used when dataset_compression = True are:

.. code:: python

{ "memory_allocation": 0.1, "methods": ["precision", "subsample"] }

The full description is given at :class:AutoSklearnClassifier(dataset_compression=...) <autosklearn.classification.AutoSklearnClassifier>.

.. _space:

The search space ================

Auto-sklearn by default searches a large space to find a well performing configuration. However, it is also possible to restrict the searchspace:

.. collapse:: Restricting the searchspace

The following shows an example of how to exclude all preprocessing methods and restrict the configuration space to only random forests.

.. code:: python

Note: The strings used to identify estimators and preprocessors are the filenames without .py.

For a full list please have a look at the source code (in autosklearn/pipeline/components/):

We do also provide an example on how to restrict the classifiers to search over :ref:sphx_glr_examples_40_advanced_example_interpretable_models.py.

.. collapse:: Turn off data preprocessing

While this is necessary in general, if you'd like to disable this step, please refer to this :ref:example <sphx_glr_examples_80_extending_example_extending_data_preprocessor.py>.

.. collapse:: Turn off feature preprocessing

Feature preprocessing is a single transformer which implements for example feature selection or transformation of features into a different space (i.e. PCA).

This can be turned off by settinginclude={'feature_preprocessor'=["no_preprocessing"]} as shown in the example above.

.. _bestmodel:

Model selection ===============

Auto-sklearn implements different strategies to identify the best performing model. For some use cases it might be necessary to adapt the resampling strategy or define a custom metric:

.. collapse:: Use different resampling strategies

Examples for using holdout and cross-validation can be found in :ref:example <sphx_glr_examples_40_advanced_example_resampling.py>

.. collapse:: Use a custom metric

Examples for using a custom metric can be found in :ref:example <sphx_glr_examples_40_advanced_example_metrics.py>

.. _ensembles:

Ensembling ==========

To get the best performance out of the evaluated models, auto-sklearn uses ensemble selection by Caruana et al. (2004) <https://dl.acm.org/doi/pdf/10.1145/1015330.1015432>_ to build an ensemble based on the models’ prediction for the validation set.

.. collapse:: Configure the ensemble building process

The following hyperparameters control how the ensemble is constructed:

* ensemble_sizedetermines the maximal size of the ensemble. If it is set to zero, no ensemble will be constructed.ensemble_nbest allows the user to directly specify the number of models considered for the ensemble. This hyperparameter can be an integer n, such that only the best n* models are used in the final ensemble. If a float between 0.0 and 1.0 is provided, ensemble_nbest would be interpreted as a fraction suggesting the percentage of models to use in the ensemble building process (namely, if ensemble_nbest is a float, library pruning is implemented as described in Caruana et al. (2006) <https://dl.acm.org/doi/10.1109/ICDM.2006.76>_).max_models_on_disc defines the maximum number of models that are kept on the disc, as a mechanism to control the amount of disc space consumed by auto-sklearn*. Throughout the automl process, different individual models are optimized, and their predictions (and other metadata) is stored on disc. The user can set the upper bound on how many models are acceptable to keep on disc, yet this variable takes priority in the definition of the number of models used by the ensemble builder (that is, the minimum of ensemble_size, ensemble_nbest and max_models_on_disc determines the maximal amount of models used in the ensemble). If set to None, this feature is disabled.

.. collapse:: Inspect the final ensemble

The results obtained from the final ensemble can be printed by calling show_models(). The auto-sklearn ensemble is composed of scikit-learn models that can be inspected as exemplified in the Example :ref:sphx_glr_examples_40_advanced_example_get_pipeline_components.py.

.. collapse:: Fit ensemble post-hoc

To use a single core only, it is possible to build ensembles post-hoc. An example on how to do this (first searching for individual models, and then building an ensemble from them) can be seen in :ref:sphx_glr_examples_60_search_example_sequential.py.

.. _inspect:

Inspecting the results ======================

auto-sklearn allows users to inspect the training results and statistics. Assume we have a fitted estimator:

.. code:: python

import autosklearn.classification automl = autosklearn.classification.AutoSklearnClassifier() automl.fit(X_train, y_train)

auto-sklearn offers the following ways to inspect the results

.. collapse:: Basic statistics

.. collapse:: Performance over Time

.. code:: python

automl.performance_over_time_.plot( x='Timestamp', kind='line', legend=True, title='Auto-sklearn accuracy over time', grid=True, ) plt.show()

.. collapse:: Evaluated models

The results obtained from the final ensemble can be printed by calling show_models().

.. collapse:: Leaderboard

automl.leaderboard() shows the ensemble members, check the :meth:docs <autosklearn.classification.AutoSklearnClassifier.leaderboard> for using leaderboard for getting information on all runs.

.. collapse:: Other

cv_results_ returns a dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame.

.. _parallel:

Parallel computation ====================

In it's default mode, auto-sklearn uses one core and interleaves ensemble building with evaluating new configurations.

.. collapse:: Parallelization with Dask

Note: auto-sklearn requires all workers to have access to a shared file system for storing training data and models.

.. code-block:: shell-session

$ export OPENBLAS_NUM_THREADS=1 $ export MKL_NUM_THREADS=1 $ export OMP_NUM_THREADS=1

For further information about how scikit-learn handles multiprocessing, please check theParallelism, resource management, and configuration <https://scikit-learn.org/stable/computing/parallelism.html>_ documentation from the library.

.. _othermanual:

Other =====

.. collapse:: Supported input types

auto-sklearn can accept targets for the following tasks (more details on Sklearn algorithms <https://scikit-learn.org/stable/modules/multiclass.html>_):

* Binary Classification * Multiclass Classification * Multilabel Classification * Regression * Multioutput Regression

Regarding the features, there are multiple things to consider:

.. code:: python

import autosklearn.classification automl = autosklearn.classification.AutoSklearnClassifier(allow_string_features=False) automl.fit(X_train, y_train)

.. collapse:: Model persistence

.. collapse:: Vanilla auto-sklearn

In order to obtain vanilla auto-sklearn as used in Efficient and Robust Automated Machine Learning
<https://papers.nips.cc/paper/5872-efficient-and-robust-automated-machine -learning>_ setensemble_size=1, initial_configurations_via_metalearning=0 and allow_string_features=False:

.. code:: python

import autosklearn.classification automl = autosklearn.classification.AutoSklearnClassifier( ensemble_size=1, initial_configurations_via_metalearning=0, allow_string_features=False, )

An ensemble of size one will result in always choosing the current best model according to its performance on the validation set. Setting the initial configurations found by meta-learning to zero makes auto-sklearn use the regular SMAC algorithm for suggesting new hyperparameter configurations.

.. collapse:: Early stopping and Callbacks

By using the parameter get_trials_callback, we can get access to the results of runs as they occur. See this example :ref:Early Stopping And Callbacks <sphx_glr_examples_40_advanced_example_early_stopping_and_callbacks.py> for more!

---

`Doc/Releases`

:orphan:

.. The following command allows to retrieve all commiters since a specified commit. From https://stackoverflow.com/questions/6482436/list-of-authors-in-git-since-a-given-commit git log 6cc8bb179fcb023d1c341cf33d2958a16a6935be.. --format="%aN <%aE>" --reverse | perl -e 'my %dedupe; while (<STDIN>) { print unless $dedupe{$_}++}'

======== Releases ========

Version 0.15.0 ==============

* ADD #1317, #1455, #1485, #1501, #1518, #1523: Initial support for multi-objective Auto-sklearn. * ADD #1300, #1410, #1414, #1415, #1420, #1468, #1500: Intial support for text features Auto-sklearn. You can now pass in columns identified as"string"columns which will be tokenized using pure sklearn methods. * ADD #1475: Support for passingX data to metrics, as required by fairlearnmetrics * ADD #1341, #1250: Expose interface to interact with how auto-sklearn performs dataset compression when required

* DOC #1304: This adds documentation for SMAC callbacks that can be used by Auto-sklearn. * DOC #1476: Example on how to interupt autosklearn with a callback, implementing a very naive early stopping

* MAINT #1364, #1473: Improve import time of Auto-sklearn 2 by moving the construction of the selector model from import time to construction time. * MAINT #1425: UpdateStopWatchto be context manager. * MAINT #1454: Rename interal bool parameterscategorical to feat_typeto reflect the use of different feature types * MAINT #1474: remove left-overs of a "public test set" from the code. This has no influence on any user-facing code. * MAINT #1487: Replace deprecated ofDataFrame.append* MAINT #1504: Renamerval to return_value or run_valueto remove ambiguity * MAINT #1506: Increase the time given to meta-learning-related unit tests to decrease the amount of timeouts on github. * MAINT #1527: Relax MLPRegressor unit tests precision. * MAINT #1545: Add explicit lower bound subsample check in the train evaluator * MAINT #1551: Fix issue with updated scipy skew see here. * MAINT #1434: Refactor the ensemble building process * MAINT #1464: Improve testing, with caching (#1464), modularity (#1417) * MAINT #1358: Add tooling Mypy, Flake8, isort, black

* FIX #741: Disable hyperparameters for a special data modality if it is not present, for example disable one hot encoding if no categorical features are present. * FIX #1365, #1369: Fix an issue withensemble_size == 0. * FIX #1374: Pass random state to all components of a pipeline. * FIX #1432: Fixes an issue in which theAutoSklearnClassifier.leaderboard()orAutoSklearnRegressor.leaderboard()could fail to display results. * FIX #1480: Properly terminate Auto-sklearn on an exception or a keyboard interrupt. * FIX #1532: Removes exception printing at shutdown for latest dask versions. The printed exceptions did not impact performance at all and were only confusing as they suggested failures of Auto-sklearn. * FIX #1547: Fixes a bug in Auto-sklearn 2 that could silently break it when passing in pandas DataFrames. * FIX #1550: Fix recent bug when performing evaluations with pandas Y.

Contributors v0.15.0

* Matthias Feurer * Eddie Bergman * Katharina Eggensperger * Sagar Kaushik * partev * Lukas Strack * Basavasagar K Patil * Eric Pedley * Aseem Kannal * SkBlaz

Version 0.14.7 ==============

* HOTFIX #1445: Locks ConfigSpace to <0.5.0 and smac to <1.3. Adds upper bounds on automl packages to help prevent further issues.

Contributors v0.14.7 * Eddie Bergman

Version 0.14.6 ==============

* HOTFIX #1407: Catches keyword arguments in SingleThreadedClient so they don't get passed to it's executing func.

Contributors v0.14.6 * Eddie Bergman

Version 0.14.5 ==============

* HOTFIX: Release PyPi package with automl_common included

Contributors v0.14.5 * Eddie Bergman

Version 0.14.4 ==============

* Fix #1356: SVR degree hyperparameter now only active with "poly" kernel. * Add #1311: Black format checking (non-strict). * Maint #1306: Run history is now saved every iteration * Doc #1309: Updated the doc faqs to include many use cases and the manual for early introductions * Doc #1322: Fix typo in contribution guide * Maint #1326: Add isort checker (non-strict) * Maint #1238, #1346, #1368, #1370: Update warnings in tests * Maint #1325: Test workflow can now be manually triggered * Maint #1332: Update docstring and typing ofinclude and excludeparams * Add #1260: Support for Python 3.10 * Add #1318: First update to use the shared backend in a new submoduleautoml_common <https://github.com/automl/automl_common>_ * Fix #1339: Resolve dependancy issues withsphinx_toolbox* Fix #1335: Fix issue where some regression algorithm gave incorrect output dimensions as raised in #1297 * Doc #1340: Update example for predefined splits * Fix #1329: Fix random state not being passed to the ConfigurationSpace * Maint #1348: Stop double triggering of github workflows * Doc #1349: Rename OSX to macOS in docs * Add #1321: Changeshow_models() to produce actual pipeline objects and not a str* Maint #1361: Removeflakydependency * Maint #1366: MakeSimpleClassificationPipelinetests more deterministic * Maint #1367: Update test values forMLPRegressor with newer numpy

Contributors v0.14.4

* Eddie Bergman * Matthias Feurer * Katharina Eggensperger * UserFindingSelf * partev

Version 0.14.3 ==============

* HOTFIX #1356: Updates dask to dask.distributed >=2012.12.

Contributors v0.14.3

* Eddie Bergman

Version 0.14.2 ==============

* FIX #1290: Fixes a bug where it was not possible to extend Auto-sklearn and run it in parallel.

Contributors v0.14.2

* Matthias Feurer

Version 0.14.1 ==============

* FIX #1248: Allow for sparse y_test. * FIX #1259: Fix an issue that could result insetup.pynot working due to relative paths being chosen. * MAINT #1261: Include a CITATION.cff file * MAINT #1263: Make unit test deterministic. * DOC #1269: Fix example on extending data preprocessing. * DOC #1270: Remove>>>from code examples in the documentation. * DOC #1271: Fix a typo in an example in the documentation. * DOC #1282: Add a contribution guide.

Contributors v0.14.1

* Eddie Bergman * Michael Becker * Katharina Eggensperger

Version 0.14.0 ==============

* ADD #900: Make data preprocessing more configurable, for example allow to completely disable it. * ADD #1128: Adds new functionality to retrieve data for an accuracy over time plot from Auto-sklearn without additional code. * FIX #1149: Stops Auto-sklearn from printing weird warnings (Exception ignored in [...]) at shutdown. * FIX #1169: Fixes a bug which made cross-validation and multi-output regression incompatible. * FIX #1170: Make all preprocessing techniques deterministic. * FIX #1190: Fixes a bug which could make predictive probabilities contain too few classes in case one class was only present a single time. * FIX #1209: Pass random states to pipeline objects. * FIX #1204: Add support for sparse data in Auto-sklearn 2.0. * FIX #1210: Add support for sparseylabels. * FIX #1245: Fixes a bug which could result in Auto-sklearn crashing in case a class was present only once. * DOC #532,#1242: Simplify installation instructions. * DOC #1144: Document installation viaconda* DOC #1195,#1201,#1214: Fix a few typos and links. Make some http links https links. * DOC #1200: Fixes variable name in an example. * DOC #1229: Improve code formatting in the documentation. * DOC #1235: Improve docker startup command so it also work on Windows. * MAINT #1198: Use latest Ubuntu LTS (20:04) for github actions. * MAINT #1231: The commandmake linkcheckno longer builds the documentation, speeding up link-checking. * MAINT #1233: Enable regression testing with 3 classification and 3 regression datasets on github actions. * MAINT #1239: Increase the timeout for github actions to 60 minutes.

Contributors v0.14.0

* Pieter Gijsbers * Taneli Mielikäinen * Rohit Agarwal * hnishi * Francisco Rivera Valverde * Eddie Bergman * Satyam Jha * Joel Jose * Oli * Matthias Feurer

Version 0.13.0 ==============

* ADD #1100: Provide access to the callbacks of SMAC. * ADD #1185: New leaderboard functionality to visualize models * FIX #1133: Refer to the correct attribute in an error message. * FIX #1154: Allow running Auto-sklearn on a 32-bit system. * MAINT #924: Instead of passing classes for the resampling strategy one has now to pass objects. * MAINT #1108: Limit the number of threads used by numpy and/or scikit-learn viathreadpoolctl. * MAINT #1135: Simplify internal workflow of pandas handling. This results in pandas being passed directly passed to scikit-learn models instead of being internally converted into a numpy array. However, this should neither impact the behavior nor the performance of Auto-sklearn. * MAINT #1157: Drop support for Python 3.6, enable support for Python 3.9. * MAINT #1159: Remove the output directory argument to the classifier and regressor. Despite the name, the output directory was not used and was a leftover from participating in the AutoML challenges. * MAINT #1187: Bump requires SMAC version to at least 0.14. * DOC #1109: Add an FAQ. * DOC #1126: Add new examples on how to use scikit-learn's inspect module. * DOC #1136: Add a new example on how to perform multi-output regression. * DOC #1152: Enable link checking when building the documentation. * DOC #1158: New example on how to configure the logger for Auto-sklearn. * DOC #1165: Improve the readme page.

Contributors v0.13.0

* Matthias Feurer * Eddie Bergman * bitsbuffer * Francisco Rivera Valverde

Version 0.12.8 ==============

* MAINT #1183: Introduce an upper bound on the dask version to retain compatibility with SMAC3.

Contributors v0.12.8

* Eddie Bergman

Version 0.12.7 ==============

* ADD #1178: Reduce precision if dataset is too large for given memory limit. * ADD #1179: Improve Auto-sklearn 2.0 meta-data by providing new meta-data for the metricsroc_auc and logloss. * DOC: Fix reference to arXiv paper * MAINT #1134,#1142,#1143: Improvements to the stale bot - the stale bot now marks issues labeled withfeedback requiredas stale if there is nothing happening for 30 days. After another 7 days it then closes the issue. * MAINT: Added a new issue template for questions. * MAINT #1168: Upper-bound scipy to1.6.3 as 1.7.0 is incompatible with SMAC. * MAINT #1173: Update the license files to be recognized by github.

Contributors v0.12.7

* Francisco Rivera Valverde * Matthias Feurer * JJ Ben-Joseph * Isaac Chung * Katharina Eggensperger * bitsbuffer * Eddie Bergman * olehb007

Version 0.12.6 ==============

* ADD #886: Provide new function which allows fitting only a single configuration. * DOC #1070: Clarify example on how successive halving and Bayesian optimization play together. * DOC #1112: Fix type. * DOC #1122: Add Python 3 to the installation command for Ubuntu. * FIX #1114: Fix a bug which made printing dummy models fail. * FIX #1117: Fix a bug previously madememory_limit=Nonefail. * FIX #1121: Fix an edge case which could decrease performance in Auto-sklearn 2.0 when using cross-validation with iterative fitting. * FIX #1123: Fix a bugautosklearn.metrics.calculate_scorefor metrics/scores which need to be minimized where the function previously returned the loss and not the score. * FIX #1115/#1124: Fix a bug which would prevent Auto-sklearn from computing meta-features in the multiprocessing case.

Contributors v0.12.6

* Francisco Rivera Valverde * stock90975 * Lucas Nildaimon dos Santos Silva * Matthias Feurer * Rohit Agarwal

Version 0.12.5 ==============

* MAINT: Remove Cython and numpy as installation requirements.

Contributors v0.12.5

* Matthias Feurer

Version 0.12.4 ==============

* ADD #660: Enable scikit-learn's power transformation for input features. * MAINT: Bump thepyrfrminimum dependency to 0.8.1 to automatically download wheels from pypi if possible. * FIX #732: Add a missing size check into the GMEANS clustering used for the NeurIPS 2015 paper. * FIX #1050: Add missing arguments to theAutoSklearn2Classifiersignature. * FIX #1072: Fixes a bug where theAutoSklearn2Classifiercould not be created due to trying to cache to the wrong directory.

Contributors v0.12.4

* Matthias Feurer * Francisco Rivera * Maximilian Greil * Pepe Berba

Version 0.12.3 ==============

* FIX #1061: Fixes a bug where the model could not be printed in a jupyter notebook. * FIX #1075: Fixes a bug where the ensemble builder would wrongly prune good models for loss functions (i.e. functions that need to be minimized such aslogloss or mean_squared_error. * FIX #1079: Fixes a bug whereAutoMLClassifier.cv_results and AutoMLRegressor.cv_resultscould rank results in opposite order for loss functions (i.e. functions that need to be minimized such aslogloss or mean_squared_error. * FIX: Fixes a bug in offline meta-data generation that could lead to a deadlock. * MAINT #1076: Uses the correct multiprocessing context for computing meta-features * MAINT: Cleanup readme and main directory

Contributors v0.12.3

* Matthias Feurer * ROHIT AGARWAL * Francisco Rivera

Version 0.12.2 ==============

* ADD #1045: New example demonstrating how to log multiple metrics during a run of Auto-sklearn. * DOC #1052: Add links to mybinder * DOC #1059: Improved the example on manually starting workers for Auto-sklearn. * FIX #1046: Add the final result of the ensemble builder to the ensemble builder trajectory. * MAINT: Two log outputs of level warning about metadata were turned reduced to the info loglevel as they are not actionable for the user. * MAINT #1062: Use threads for local dask workers and forkserver to start subprocesses to reduce overhead. * MAINT #1053: Remove the restriction to guard single-core Auto-sklearn by__main__ == "__name__" again.

Contributors v0.12.2

* Matthias Feurer * ROHIT AGARWAL * Francisco Rivera * Katharina Eggensperger

Version 0.12.1 ==============

* ADD: A new heuristic which gives a warning and subsamples the data if it is too large for the givenmemory_limit. * ADD #1024: Tune scikit-learn'sMLPClassifier and MLPRegressor. * MAINT #1017: Improve the logging server introduced in release 0.12.0. * MAINT #1024: Move to scikit-learn 0.24.X. * MAINT #1038: Use new datasets for regression and classification and also update the metadata used for Auto-sklearn 1.0. * MAINT #1040: Minor speed improvements in the ensemble selection algorithm.

Contributors v0.12.1

* Matthias Feurer * Katharina Eggensperger * Francisco Rivera

Version 0.12.0 ==============

* BREAKING: Auto-sklearn must now be guarded by __name__ == "__main__"due to the use of thespawnmultiprocessing context. * ADD #1026: Adds improved meta-data for Auto-sklearn 2.0 which results in strong improved performance. * MAINT #984 and #1008: Move to scikit-learn 0.23.X * MAINT #1004: Move from travis-ci to github actions. * MAINT 8b67af6: drop the requirement to the lockfile package. * FIX #990: Fixes a bug that made Auto-sklearn fail if there are missing values in a pandas DataFrame. * FIX #1007, #1012 and #1014: Log multiprocessing output via a new log server. Remove several potential deadlocks related to the joint use of multi-processing, multi-threading and logging.

Contributors v0.12.0

* Matthias Feurer * ROHIT AGARWAL * Francisco Rivera

Version 0.11.1 ==============

* FIX #989: Fixes a bug where ywas not passed to all data preprocessors which made 3rd party category encoders fail. * FIX #1001: Fixes a bug which could make Auto-sklearn fail at random. * MAINT #1000: Introduce a minimal version fordask.distributed.

Contributors v0.11.1

* Matthias Feurer

Version 0.11.0 ==============

* ADD #992: Move ensemble building from being a separate process to a job submitted to the dask cluster. This allows for better control of the memory used in multiprocessing settings. * FIX #905: MakeAutoSklearn2Classifierpicklable. * FIX #970: Fix a bug where Auto-sklearn would fail if categorical features are passed as a Pandas Dataframe. * MAINT #772: Improve error message in case of dummy prediction failure. * MAINT #948: Finally use Pandas >= 1.0. * MAINT #973: Improve meta-data by running meta-data generation for more time and separately for important metrics. * MAINT #997: Improve memory handling in the ensemble building process. This allows building ensembles for larger datasets.

Contributors v0.11.0

* Matthias Feurer * Francisco Rivera * Karl Leswing * ROHIT AGARWAL

Version 0.10.0 ==============

* ADD #325: Allow to separately optimize metrics for metadata generation. * ADD #946: New dask backend for parallel Auto-sklearn. * BREAKING #947: Drop Python3.5 support. * BREAKING #946: Remove shared model mode for parallel Auto-sklearn. * FIX #351: No longer pass un-picklable logger instances to the target function. * FIX #840: Fixes a bug which prevented computing metadata for regression datasets. Also adds a unit test for regression metadata computation. * FIX #897: Allow custom splitters to be used with multi-ouput regression. * FIX #951: Fixes a lot of bugs in the regression pipeline that caused bad performance for regression datasets. * FIX #953: Re-addliac-arffas a dependency. * FIX #956: Fixes a bug which could cause Auto-sklearn not to find a model on disk which is part of the ensemble. * FIX #961: Fixes a bug which caused Auto-sklearn to load bad meta-data for metrics which cannot be computed on multiclass datasets (especially ROC_AUC). * DOC #498: Improve the example on resampling strategies by showing how to pass scikit-learn's splitter objects to Auto-sklearn. * DOC #670: Demonstrate how to give access to training accuracy. * DOC #872: Improve an example on how obtain the best model. * DOC #940: Improve documentation of the docker image. * MAINT: Improve the docker file by setting environment variable that restrict BLAS and OMP to only use a single core. * MAINT #949: Replacepip by pip3in the installation guidelines. * MAINT #280, #535, #956: Update meta-data and include regression meta-data again.

Contributors v0.10.0

* Francisco Rivera * Matthias Feurer * felixleungsc * Chu-Cheng Fu * Francois Berenger

Version 0.9.0 =============

* ADD #157,#889: Improve handling of pandas dataframes, including the possibility to use pandas' categorical column type. * ADD #375: NewSelectRatesfeature preprocessing component for regression. * ADD #891: Improve the robustness of Auto-sklearn by using the single best model if no ensemble is found. * ADD #902: Track performance of the ensemble over time. * ADD #914: Add an example on using pandas dataframes as input to Auto-sklearn. * ADD #919: Add an example for multilabel classification. * MAINT #909: Fix broken links in the documentation. * MAINT #907,#911: Add initial support for mypy. * MAINT #881,#927: Automatically build docker images on pushes to the master and development branch and also push them to dockerhub and the github docker registry. * MAINT #918: Remove old dependencies from requirements.txt. * MAINT #931: Add information about the host system and installed packages to the log file. * MAINT #933: Reduce the number of warnings raised when building the documentation by sphinx. * MAINT #936: Completely restructure the examples section. * FIX #558: Provide better error message when the ensemble process fails due to a memory issue. * FIX #901: Allow custom resampling strategies again (was broken due to an upgrade of SMAC). * FIX #916: Fixes a bug where the data preprocessing configurations were ignored. * FIX #925: make internal data preprocessing objects clonable.

Contributors v0.9.0 *

* Francisco Rivera * Matthias Feurer * felixleungsc * Vladislav Skripniuk

Version 0.8 ===========

* ADD #803: multi-output regression * ADD #893: new Auto-sklearn mode Auto-sklearn 2.0

Contributors v0.8.0 *

* Chu-Cheng Fu * Matthias Feurer

Version 0.7.1 =============

* ADD #764: support for automatic per_run_time_limit selection * ADD #864: add the possibility to predict with cross-validation * ADD #874: support to limit the disk space consumption * MAINT #862: improved documentation and render examples in web page * MAINT #869: removal of competition data manager support * MAINT #870: memory improvements when building ensemble * MAINT #882: memory improvements when performing ensemble selection * FIX #701: scaling factors for metafeatures should not be learned using test data * FIX #715: allow unlimited ML memory * FIX #771: improved worst possible result calculation * FIX #843: default value for SelectPercentileRegression * FIX #852: clip probabilities within [0-1] * FIX #854: improved tmp file naming * FIX #863: SMAC exceptions also registered in log file * FIX #876: allow Auto-sklearn model to be cloned * FIX #879: allow 1-D binary predictions

Contributors v0.7.1 *

* Matthias Feurer * Xiaodong DENG * Francisco Rivera

Version 0.7.0 =============

* ADD #785: user control to reduce the hard drive memory required to store ensembles * ADD #794: iterative fit for gradient boosting * ADD #795: add successive halving evaluation strategy * ADD #814: new sklearn.metrics.balanced_accuracy_score instead of custom metric * ADD #815: new experimental evaluation mode called iterative_cv * MAINT #774: move from scikit-learn 0.21.X to 0.22.X * MAINT #791: move from smac 0.8 to 0.12 * MAINT #822: make autosklearn modules PEP8 compliant * FIX #733: fix for n_jobs=-1 * FIX #739: remove unnecessary warning * FIX ##769: fixed error in calculation of meta features * FIX #778: support for python 3.8 * FIX #781: support for pandas 1.x

Contributors v0.7.0 *

* Andrew Nader * Gui Miotto * Julian Berman * Katharina Eggensperger * Matthias Feurer * Maximilian Peters * Rong-Inspur * Valentin Geffrier * Francisco Rivera

Version 0.6.0 =============

* MAINT: move from scikit-learn 0.19.X to 0.21.X * MAINT #688: allow for pyrfr version 0.8.X * FIX #680: Remove unnecessary print statement * FIX #600: Remove unnecessary warning

Contributors v0.6.0 *

* Guilherme Miotto * Matthias Feurer * Jin Woo Ahn

Version 0.5.2 =============

* FIX #669: Correctly handle arguments to the AutoMLRegressor* FIX #667: Auto-sklearn works with numpy 1.16.3 again. * ADD #676: Allow brackets [ ] inside the temporary and output directory paths. * ADD #424: (Experimental) scripts to reproduce the results from the original Auto-sklearn paper.

Contributors v0.5.2 *

* Jin Woo Ahn * Herilalaina Rakotoarison * Matthias Feurer * yazanobeidi

Version 0.5.1 =============

* ADD #650: Auto-sklearn will immediately stop if prediction using scikit-learn's dummy predictor fail. * ADD #537: Auto-sklearn will no longer start for time limits less than 30 seconds. * FIX #655: Fixes an issue where predictions using models from parallel Auto-sklearn runs could be wrong. * FIX #648: Fixes an issue with custom meta-data directories. * FIX #626: Fixes an issue where losses were not minimized, but maximized. * MAINT #646: Do no longer restrict the numpy version to be less than 1.14.5.

Contributors v0.5.1 *

* Jin Woo Ahn * Taneli Mielikäinen * Matthias Feurer * jianswang

Version 0.5.0 =============

* ADD #593: Auto-sklearn supports the n_jobsargument for parallel computing on a single machine. * DOC #618: Added links to several system requirements. * Fixes #611: Improved installation from pip. * TEST #614: Test installation with clean Ubuntu on travis-ci. * MAINT: Fixed broken link and typo in the documentation.

Contributors v0.5.0 *

* Mohd Shahril * Adrian * Matthias Feurer * Jirka Borovec * Pradeep Reddy Raamana

Version 0.4.2 =============

* Fixes #538: Remove rounding errors when giving a training set fraction for holdout. * Fixes #558: Ensemble script now uses less memory and the memory limit can be given to Auto-sklearn. * Fixes #585: Auto-sklearn's ensemble script produced wrong results when called directly (and not via one of Auto-sklearn's estimator classes). * Fixes an error in the ensemble script which made it non-deterministic. * MAINT #569: Rename hyperparameter to have a different name than a scikit-learn hyperparameter with different meaning. * MAINT #592: backwards compatible requirements.txt * MAINT #588: Fix SMAC version to 0.8.0 * MAINT: remove dependency on the six package * MAINT: upgrade to XGBoost 0.80

Contributors v0.4.2 *

* Taneli Mielikäinen * Matthias Feurer * Diogo Bastos * Zeyi Wen * Teresa Conceição * Jin Woo Ahn

Version 0.4.1 =============

* Added documentation on how to extend Auto-sklearn <https://github.com/automl/auto-sklearn/pull/510>_ with custom classifier, regressor, and preprocessor. * Auto-sklearn now requires numpy version between 1.9.0 and 1.14.5, due to higher versions causing travis failure. * Examples now usesklearn.datasets.load_breast_cancer() instead of sklearn.datasets.load_digits()to reduce memory usage for travis build. * Fixes future warnings on non-tuple sequence for indexing. * Fixes#500 <https://github.com/automl/auto-sklearn/issues/500>_: fixes ensemble builder to correctly evaluate model score with any metrics. See thisPR <https://github.com/automl/auto-sklearn/pull/522>_. * Fixes#482 <https://github.com/automl/auto-sklearn/issues/482>_ and#491 <https://github.com/automl/auto-sklearn/issues/491>_: Users can now set up custom logger configuration by passing a dictionary created by a yaml file tologging_config. * Fixes#566 <https://github.com/automl/auto-sklearn/issues/566>_: ensembles are now sorted correctly. * Fixes#293 <https://github.com/automl/auto-sklearn/issues/293>_: Auto-sklearn checks if appropriate target type was given for classification and regression before call tofit(). * Travis-ci now runs flake8 to enforce pep8 style guide, and uses travis-ci instead of circle-ci for deployment.

Contributors v0.4.1 *

* Matthias Feurer * Manuel Streuhofer * Taneli Mielikäinen * Katharina Eggensperger * Jin Woo Ahn

Version 0.4.0 =============

* Fixes #409 <https://github.com/automl/auto-sklearn/issues/409>_: fixespredict_proba to no longer raise an AttributeError. * Improved documentation of the parallel example. * Classifiers are now tested to be idempotent asrequired by scikit-learn
<https://scikit-learn.org/stable/developers/develop.html#fitting>_. * Fixes the usage of the shrinkage parameter in LDA. * Fixes#410 <https://github.com/automl/auto-sklearn/issues/410>_ and changes the SGD hyperparameters * Fixes#425 <https://github.com/automl/auto-sklearn/issues/425>_ which caused the non-linear support vector machine to always crash on OSX. * Implements#149 <https://github.com/automl/auto-sklearn/issues/149>_: it is now possible to pass a custom cross-validation split following scikit-learn'smodel_selectionmodule. * It is now possible to decide whether or not to shuffle the data in Auto-sklearn by passing a boolshufflein the dictionary ofresampling_strategy_arguments. * Added functionality to track the test performance over time. * Re-factored the ensemble building to be faster, read less data from the hard drive and perform random tie breaking in case of equally well-performing models. * Implements#438 <https://github.com/automl/auto-sklearn/issues/438>_: To be consistent with the output of SMAC (which minimizes the loss of a target function), the output of the ensemble builder is now also the output of a minimization problem. * Implements#271 <https://github.com/automl/auto-sklearn/issues/271>_: XGBoost is available again, even configuring the new dropout functionality. * New documentation section :ref:inspect. * Fixes#444 <https://github.com/automl/auto-sklearn/issues/444>_: Auto-sklearn now only loads models for refit which are actually relevant for the ensemble. * Adds an operating system check at import and installation time to make sure to not accidentaly run on a Windows machine. * New examples gallery using sphinx gallery: :ref:examples* Safeguard Auto-sklearn against deleting directories it did not create (Issue#317 <https://github.com/automl/auto-sklearn/issues/317>_.

Contributors v0.4.0 *

* Matthias Feurer * kaa * Josh Mabry * Katharina Eggensperger * Vladimir Glazachev * Jesper van Engelen * Jin Woo Ahn * Enrico Testa * Marius Lindauer * Yassine Morakakam

Version 0.3.0 =============

* Upgrade to scikit-learn 0.19.1. * Do not use theDummyClassifier or DummyRegressoras part of an ensemble. Fixes#140 <https://github.com/automl/auto-sklearn/issues/140>_. * Fixes #295 by loading the data in the subprocess instead of the main process. * Fixes #326: refitting could result in a type error. This is now fixed by better type checking in the classification components. * Updated search space forRandomForestClassifier, ExtraTreesClassifierandGradientBoostingClassifier(fixes #358). * Removal of constant features is now a part of the pipeline. * Allow passing an SMBO object into theAutoSklearnClassifierandAutoSklearnRegressor.

Contributors v0.3.0 *

* Matthias Feurer * Jesper van Engelen

Version 0.2.1 =============

* Allows the usage of scikit-learn 0.18.2. * Upgrade to latest SMAC version (0.6.0) and latest random forest version (0.6.1). * Added a Dockerfile. * Added the possibility to change the size of the holdout set when using holdout resampling strategy. * Fixed a bug in QDA's hyperparameters. * Typo fixes in print statements. * New method to retrieve the models used in the final ensemble.

Contributors v0.2.1 *

* Matthias Feurer * Katharina Eggensperger * Felix Leung * caoyi0905 * Young Ryul Bae * Vicente Alencar * Lukas Großberger

Version 0.2.0 =============

*auto-sklearn supports custom metrics and all metrics included in scikit-learn. Different metrics can now be passed to thefit()-method estimator objects, for exampleAutoSklearnClassifier.fit(metric='roc_auc'). * Upgrade to scikit-learn 0.18.1. * Drop XGBoost as the latest release (0.6a2) does not work when spawned by the pyninsher. auto-sklearn* can use multiprocessing in calls topredict()andpredict_proba. By Laurent Sorber <https://github.com/lsorber>_.

Contributors v0.2.0 *

* Matthias Feurer * Katharina Eggensperger * Laurent Sorber * Rafael Calsaverini

Version 0.1.x =============

There are no release notes for auto-sklearn prior to version 0.2.0.

Contributors v0.1.x *

* Matthias Feurer * Katharina Eggensperger * Aaron Klein * Jost Tobias Springenberg * Anatolii Domashnev * Stefan Falkner * Alexander Sapronov * Manuel Blum * Diego Kobylkin * Jaidev Deshpande * Jongheon Jeong * Hector Mendoza * Timothy J Laurent * Marius Lindauer * _329_ * Iver Jordal

---

`CONTRIBUTING`

`Contributing to auto-sklearn`


Thanks for checking out the contribution guide!
We included a quick overview at the end for anyone familiar with open-source contribution or familiar with auto-sklearn and simply wants to see our workflow.
If you're new to contributing then hopefully this guide helps with contributing to open-source and auto-sklearn, whether it be a simple doc fix, a small bug fix or even new features that everyone can get use out of.
If you're looking for a particular project to work on, check out the Issues for things you might be interested in!
For experienced contributors, you can skip the overview and find the quick walk-through here!
This guide is only aimed towards Unix command line users as that's what we know but the same principles apply.
Contributing

There are many kinds of contributions you can make to auto-sklearn but we'll focus on three main ones Documentation, Bug Fixes and Features, each of which require a little bit of a different flow.
We need to perform several checks to make sure it meets code standards and won't cause any issues later on.
We tend to follow a development cycle which could be called _Gitflow_ which you are new to git or git-based projects, you can see a nice summary here.
First we'll go over the general flow, what each step does and then later look at making more specific kinds of changes, what we'd like to see and how you might create a workflow.
Following that we'll tell you about how you can test your changes locally and then how to submit your pull request!
General steps 

*   The first thing to do is create your own fork.
    This is to give you a nice place to work on your changes without impacting any code from the original repository.

To do this, navigate to automl/auto-sklearn and hit the fork button in the top-right corner. This will copy the repository to your own account, including all of its different branches. You'll be able to access this athttps://github.com/{your-username}/auto-sklearn.

* The next steps are to download your own fork and to create a new branch where all your changes will go. It's important to work off the latest changes on the development branch.

bash

# With https
    # Note the --recurse-submodules args, we use a submodule autosklearn/automl_common
    # so it needs to be downloaded too
    git clone --recurse-submodules https://github.com/your-username/auto-sklearn
    # ... or with ssh
    git clone --recurse-submodules [email protected]:your-username/auto-sklearn.git
    # Navigate into the cloned repo
    cd auto-sklearn
    # Create a new branch based off the development one
    git checkout -b my_new_branch development
    # If you missed the --recurse-submodules arg during clone or need to install the
    # submodule manually, then execute the following line:
    #
    # git submodule udate --init --recursive
    # ... Alternatively, if you would prefer a more manual method
    # Show all the available branches with a * beside your current one
    git branch
    # Switch to the development branch
    git checkout development
    # Create a new branch based on the currently active branch
    git checkout -b my_new_branch    # If you missed the --recurse-submodules arg during clone or need to install the
    # submodule manually, then execute the following line:
    #
    # git submodule update --init --recursive

The reason to create a new branch is two fold:

* One, it keeps the commit history for your changes much cleaner once we merge them in. * If you have to perform a rebase or a merge later on, this will be much easier.

* You'll need a virtual environment to work in. If you've never used them before, now is definitely the time to start as a virtual environment lets you keep packages for a project separate.

bash

# Create a virtual environment in a folder called my-virtual-env
    python -m venv my-virtual-env    # Activate the virtual environment
    source my-virtual-env/bin/activate


    *   A popular alternative to managing Python projects is conda.
    *   The folder

my-virtual-env

 is where dependency packages that are required for auto-sklearn will go.
    *   In general, once you have activated the virtual environment with

source my-virtual-env/bin/activate, anything you install with pip install

 will now go into the virtual environment.
        While this environment is active, any python you run will have access to the packages here.
        If at any time you want to deactivate it simply type

deactivate

 in the shell or just close the shell and open a new one.
    *   If you use Python 3.6 or lower on your machine then unfortunately auto-sklearn doesn't support this.
        Fortunately, you can check out pyenv which lets you switch between Python versions on the fly!

* Now that we have a virtual environment, it's time to install all the dependencies into it. We've provided a simplemakecommand to help do this.

bash

make install-dev
    # Manually
    pip install -e .[test,examples,doc]    # If you're using shells other than bash you'll need to use
    pip install -e ".[test,examples,doc]"


    *   If your only exposure to using pip is

pip install package_name

 then this might be a bit confusing.
    *   If we type

pip install -e . (notice the 'dot'), this tells pip to install a package located here, in this directory, .

.
        The

-e flag indicates that it should be editable, meaning you will not have to run pip install .

 every time you make a change and want to try it.
    *   Finally the

[test,examples,doc] tells pip

 that there's some extra optional dependencies that we want to install.
        These are dependencies used in development but ones that are not required to actually run auto-sklearn itself.
        You can check out what these are in the

setup.py

 file.
    *   If you're new to virtual environments, after performing all this, it's a great time to check out what actually exists in the

my-virtual-env folder.

* You can check out some functionality we have captured in a Makefile by running make help

* Now it's time to make some changes, whether it be for documentation, a bug fix or a new features.

`Making Changes`


We'll go over three main categories of contributions but don't feel limited by these headers, adding to our tests, improving the typing of functions and methods or even some compliance changes are also super useful!
#### Bug Fixes
Auto-sklearn has been through quite a few iterations, been used for many purposes and is constantly used in ways we didn't even think of.
Like any maturing software, there will be bugs, old and new.
While we try to create unit tests to catch as many of these as we can, some slip through and new bugs get introduced as dependencies are updated and new features are introduced.
If you're looking to help by fixing some bugs, or you've encountered your own bugs you'd like fixed in the official version of auto-sklearn, we would greatly appreciate any help!
We'd be happy to guide you through what we think may be the underlying cause or at least point you in the right direction if it's something you wish to work on.
Some core things to consider in fixing a bug:
*   What's the minimal working example that reproduces this bug?
    *   This is usually the first step.
        The process of creating a minimal piece of code that reproduces a bug often illuminates what needs to be fixed.
*   What's the quick fix and what's the long term fix?
    *   Sometimes it's a code typo, a quick correction and problem solved.
        Other times, the bug is an artifact of some larger underlying issue that has gone unnoticed and might require some restructuring.
        If this is the case, let us know as you work on it!
        If it requires breaking, such as changing default behaviour or public API, sometimes a quick patch and fix will do and larger restricting fixes can be tackled in a timely manner.
        As a rule of thumb, if a bug requires modifying more then 50-100 lines of code it's probably something we would like to talk through on how best to tackle it.
*   How can we create a test for this bug in the future?
    *   Of course, once a bug is squashed, we'd like it to not show again and having a test to catch it for the future will future-proof against any changes down the line.
        Thankfully, most of this is usually captured in the minimal working example and all that is left is to turn it into a comprehensive test!
What's important once fixing a bug is writing a good PR that let's us now how you identified the bug, what the problem was and how it was fixed.
This lets use review your code with all this in mind and follow the same thought process that lead you to fix it in the first place!
#### Documentation
Anything to contribute to better documentation is always appreciated and the main way users can get to know about auto-sklearn.
Whether it's a typo fix, something you didn't find clear or something you think we didn't explain properly, we'd love to improve it!

All of our documentation is done with sphinxwith some various plugins that you can see indoc/conf.py.

All of your changes can be viewed by first building the docs and then opening doc/build/html/index.html in a browser.

* If you're simply fixing a typo, there shouldn't be much to do except make the PR and we'll accept it without much issue. * If you want to fix a link you should know how linking withsphinxworks * For links to internal documentation, you can create a label with`.rst .. _mylabel:

text

*   Later on you can reference this label by

.rst
    I am reference to :ref:

the above label<mylabel>


    text
*   Now if you want to link some external documentation you'll need
        to do something like
.rst
    Here's a

link<https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html>

_ to the external documentation on linking for sphinx
    text
Notice the trailing _ which is important
*   If you want to make some more detailed documentation about some feature that you introduced or you think is not well documented, you'll have to think about a few things.
    *   Can you include a code snippet to illustrate what you mean?
    *   Are there other relevant parts of the documentation or code that should be linked to?
    *   How much other parts of auto-sklearn are you relying on readers to know before hand, maybe link to those sections if you do.*   If you want to contribute an example, it's a great way to really illustrate an entire flow of some feature.
    sphinx-gallery will run any python file example_*.py in one of the example folders.
    This allows you to have both ReStructured Markdown (rst) and python code with it's output into a single html page!
    You'll want to check out some of the other examples to see how to embed rst into a .py file.
#### Features
While auto-sklearn has many features we're proud of, there's always room for more and better functionality.
Features don't have to be performance driven, in fact, most of the new features we'd love to see are to improve a users ability to interact with auto-sklearn, whether it be usability or an ability to inspect the inner workings in an intuitive manner!
However, features are usually a bigger project and for this we'd really advise getting in touch with us first about the feature in mind.
There are some things we believe best left for external libraries or integrations that we don't wish to consider at this time.
Another reason to get in touch is to nail exactly what this feature will look like beforehand, the more direct and concise the feature is, the better.
Some things to keep in mind with new features:
*   A new feature is great, but will this change any existing default behaviours?
    Unexpected changes for existing users can be detrimental and unexpected.
    Sometimes it has to be done but often these new features can be presented as an option the user can enable.
*   If you're introducing some new API:
    *   Creating some code samples of how you'd like your feature to be used is a great start.
    *   What current way is there to do the same thing, can any functionality already present be used to help with this new API?
    *   Are you going to deprecate any current API? This is an important fact to consider and something that will definitely have to be discussed.
Testing Features - Writing features are great but new features means new bugs, but thankfully that's what we can write tests for.
How to test your feature is always tricky, especially if the feature is big in scope.
Unfortunately there's no secret or rule of thumb other than try to cover every case you can think of.
The more it's tested the better!
Bugs will still get through, that's okay, we will have done what we can and we can fix those in the future but as long as the usual use-cases are covered, this shouldn't be too much of a problem.
Documenting Features - Now how are people going to know about your new feature you've introduced?
This is what documentation is so great for and it's how almost all software functionality is expressed.
If the feature is enabled by a parameter, great, almost all the documentation is already present in the code docstring, automatically being rendered in the online docs ... that docstring that was updated when you made changes ... right?
Sometimes, the new functionality isn't so clear from a simple parameter description and so maybe something needs to be added to the manual.rst, a short paragraph suffices and much appreciated.
Lastly, if the feature really is a game changer or you're very proud of it, consider making an example_*.py that will be run and rendered in the online docs!
Testing 

* Let's assume you've made some changes, now we have to make sure they work.
    Begin by simply running all the tests.
    If there's any errors, they'll pop up once it's complete.
bash
    pytest
    text
* Note that these may take a while so check out pytest --help to see how you can run tests so that only previous failures run or only certain tests are run.
        This can help you try changes and get results faster.
        Do however run one last full pytest once you are finished and happy!
    * Here are some we find particularly useful

        # Run tests in specific file like 'test_estimators.py'
        pytest "test/test_automl/test_estimators.py"
        # Run an entire directory of tests such as 'pipeline'
        pytest "test/test_pipeline"
        # Run a specific test 'test_mytest' in a specific directory 'test_automl'
        pytest -k "test_mytest" "test/test_automl"

# Rerun all the tests that failed in the last pytestcommand pytest --last-failed

# Rerun all tests but run the failed ones first pytest --failed-first

# Exit on the first test failure pytest -x

text

* More advanced editors like PyCharm may have built in integrations which could be good to check out!
    * Running all unittests will take a while, here's how you can run them in parallel


        export OPENBLAS_NUM_THREADS=1
        export MKL_NUM_THREADS=1
        export OMP_NUM_THREADS=1
      
        pytest -n 4
        text
* Now we are going to use sphinx to generate all the documentation and make sure there are no issues.
bash
    make doc
    text
*   If you're unfamiliar with sphinx, it's a documentation generator which can read comments and docstrings from within the code and generate html documentation.
    *   If you've added documentation, we also has a command linkcheck for making sure all the links correctly go to some destination.
        This helps tests for dead links or accidental typos.
bash
    make linkcheck
    text
*   We also use sphinx-gallery which can take python files (such as those in the examples folder) and run them, creating html which shows the code and the output it generates.
bash
    make examples
    text
*   To view the documentation itself, make sure it is built with the above commands and then open doc/build/html/index.html with your favourite browser:
bash
    # Firefox
    firefox ./doc/build/html/index.html
    # Using your default browser
    xdg-open ./doc/build/html/index.html
    
text
* Once you've made all your changes and all the tests pass successfully, we need to make sure that the code fits a certain format and that the typing is correct.
    * Formatting and import sorting can helps keep things uniform across all coding styles. We use black and isort to do this for us. To automatically run these formatters across the code base, just run the following command:
bash
    make format
    text
* To then check for issues using black, isort, mypy, flake8 and pydocstyle, run
bash
    make check
    text
* To do this checking automatically, we use pre-commit which if you already installed everything with make install-dev then this has been done for you.
    This will happen every time you make a commit and warn you of any issues.
    Otherwise you can run the following to install pre-commit.
bash
    pre-commit install
    text
* To run pre-commit manually:
bash
    pre-commit run --all-files
    text
*   The reason we use tools like flake8, mypy, black, isort and pydocstyle is to make sure that when we review code:
        *   There are no extra blank spaces and blank lines. (flake8, black)
        *   Lines don't end up too long. (flake8, black)
        *   Code from multiple source keeps a similar appearance. (black)
        *   Importing things is consistently ordered. (isort)
        *   Functions are type annotated and correct with static type checking. (mypy)
        * Function and classes have docstrings. (pydocstyle)
        If you are new to Python types, or stuck with how something should be 'typed', please feel free to push the pull request in the following steps and we should be able to help you out.
    * If interested, the configuration for pre-commit can be found in .pre-commit-config.yaml with the other tools mainly being configured in pyproject.toml and .flake8.

Creating the PR 

*   We've made sure all the changes work, we've maybe added a test for them and run all the tests locally.
    It's time to commit the changes, push them up to your fork and create a pull request!
bash
    # Get an overview of all the files changes
    git status
    # Add your changed files
    git add {changed files}
    git commit -m "Something as meaningful as possible"

# This will push my_new_branch to your fork located at origingit push --set-upstream origin my_new_branch

text

*   At this point, we need to create a pull request (PR) to the automl/auto-sklearn repository with our new changes.
    This can be done simply by going to your own forked repo, clicking 'Contribute', and selecting
    the development branch of automl/auto-sklearn.    *   automl/auto-sklearn | development <- your-username/auto-sklearn | my_new_branch
        The reason we don't want to directly merge new PR's into master is to make
        sure we always have a stable version. With a development branch, we can safely
        accumulate certain changes and makes sure they all work together before creating
        a new master version.
*   Now you've got to describe what you've changed.
    You'll likely want to check out this blogpost which we believe to give a good overview of what a good PR looks like and will help us get your changes in sooner rather than later!
    Some key things to include here are:
    *   A high level overview of what you've done such as fixing a problem or
        introducing a new feature.
        If it's a simple doc fix, don't worry too much about this.
    *   Have you introduced any breaking changes to the API?
        We may not realise it while reviewing the changes but if you are aware,
        it definitely helps to tell us!
    *   Do you think this might have any implications in the future or how would
        further work on this look like?
    *   If you've introduced some new feature, write about who might use it,
        give a brief code sample to show it and let us know how you tested it!
    *   If you've fixed a bug, write about why this bug exists in the first place,
        how you solved it and how a test makes sure it won't pop up again!
*   Once you've submitted a PR, we have it set up so github will automatically schedule some unit tests and documentation building to run.
    This will make sure all the tests run smoothly, make sure the documentation builds correctly and do some quick check on code quality.
    You'll be able to see these run in the Checks tab or at the bottom of the PR.
    If you see a red x, that means somethings probably gone wrong which should have been caught by running the tests locally but we also do some checks in environments you were not developing on.
    Sometimes it is the case that there is a bug unrelated to your changes that cause the tests to fail but we are aware of these.
    If you see one of these failures, feel free to ask!
*   Meanwhile, we'll also review your code.
    Some common review points are:
    *   This seems odd, why was this done?
    *   Could you see if you can use the functionality from place X?
    *   Could you create a test or documentation for this?
    *   This could be a nicer way of doing this, what do you think?
    Occasionally there will be some major point which will require more discussion but those are more on a case-by-case basis.
*   This process of review, fix and testing may go on a few times.
    The simplest way to reduce the time needed and to help us too is to run the tests, code formatting check and doc building locally.
    If they all pass locally they will very often have no issues in the automated tests.
    Once everyone is happy, it's time for us to hit (squash) merge, get it into the development branch and have one more contribution to auto-sklearn!
Pull Request Overview

* Create a fork of the automl/auto-sklearn git repo
* Check out what's available by running make help.
* Clone your own fork and create a new branch from the branch to work on

bash
    git clone [email protected]:your-username/auto-sklearn.git
    cd auto-sklearn
    git checkout -b my_new_branch development
    # Initialize autosklearn/automl_common submodule
    git submodule update --init --recursive
    # Create a virtual environment and activate it so there are no package
    # conflicts
    python -m venv my-virtual-env
    source my-virtual-env/bin/activate
    make install-dev
    # pip install -e ".[test,docs,examples]" # To manually install things
    # Edit files...
    # Format code
    make format
    # Check for any issues
    make check
    # ... fix any issues
    # If you changed documentation:
    # This will generate all documentation and check links
    make doc
    make linkcheck
    make examples  # mainly needed if you modified some examples
    # ... fix any issues
    # If you edited any code
    # Check out pytest --help if you want to only run specific tests
    pytest
    # ... fix any issues
    # If you want to run pre-commit, the formatting checks we run on github
    pre-commit install
    pre-commit run --all-files
    # ... fix any issues
    # Check the changed files
    git status
    # Add the changes
    git add {changed files}
    git commit -m "Meaningful as you can make it message"
    # Push back to your fork
    git push --set-upstream origin my_new_branch
    
text
* Go to github, go to your fork and then make a pull request using the Contribute button.
    * automl/auto-sklearn | development <- your-username/auto-sklearn | my_new_branch
* Write a PR with a description of the changes, why you implemented them and any implications.
    * Check out this blog post for some inspiration!
* Once we see this, we will run some automated tests on the pull request. These
tests are the same as the ones you can run manually and are mentioned in the
test section.
* We'll review the code and perhaps ask for some changes
* Once we're happy with the result, we'll merge it in!
General FAQ
I've finished my pull request or made new changes and want it reviewed, now what? 

*   We'll actively monitor what pull requests are already in progress.
    Once you believe you pull request to be ready or you need some feedback, feel free to comment on your pull request, tagging @eddiebergman or @mfeurer and we'll provide feedback as soon as we can!
I've been asked to rebase my pull request, why and what do I do? 

*   It can often be the case that while you were working on some changes, we may have merged something new into the development branch.    This can be a problem because the branch you created was based off the old development branch.
    This means there are new changes in the automl/auto-sklearn code base you don't have in your forked repo or locally.
    This is not always an issue, generally if different parts of the code were touched they can be merged safely.
    Either way, if we ask you to rebase, it is because it won't merge or we think there may have been overlapping changes between what you were working on and the new code put into the development branch.
*   First, update the development branch on your fork
       The easiest way to do this is go to https://github.com/your-username/auto-sklearn*, navigate to the development branch and hit fetch upstream.
    *   This will make it so your fork of auto-sklearn is now up to date
*   Second, we need to go to our clone and pull in these new changes to development
bash
    git checkout development
    git pull
    text
* Lastly, we need to rebase the my_new_branch on top of the new development
bash
    git checkout my_new_branch
    git rebase development
    text
*   Now if there were no conflicts, that's it, you can continue as normal.
    If there are conflicts, you'll have to sort these out which you can find out how to do here and here.
---
README 
auto-sklearn
auto-sklearn is an automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator.
Find the documentation here. Quick links:
  * Installation Guide
  * Releases
  * Manual
  * Examples
  * API
auto-sklearn in one image 
auto-sklearn in four lines of code 
python
import autosklearn.classification
cls = autosklearn.classification.AutoSklearnClassifier()
cls.fit(X_train, y_train)
predictions = cls.predict(X_test)
text
Relevant publications 
If you use auto-sklearn in scientific publications, we would appreciate citations.
Efficient and Robust Automated Machine Learning
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum and Frank Hutter
Advances in Neural Information Processing Systems 28 (2015)Link to publication.

@inproceedings{feurer-neurips15a,
    title     = {Efficient and Robust Automated Machine Learning},
    author    = {Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina and Springenberg, Jost and Blum, Manuel and Hutter, Frank},
    booktitle = {Advances in Neural Information Processing Systems 28 (2015)},
    pages     = {2962--2970},
    year      = {2015}
}
text
----------------------------------------
Auto-Sklearn 2.0: The Next Generation
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer and Frank Hutter*
arXiv:2007.04074 [cs.LG], 2020Link to publication.

@article{feurer-arxiv20a,
    title     = {Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning},
    author    = {Feurer, Matthias and Eggensperger, Katharina and Falkner, Stefan and Lindauer, Marius and Hutter, Frank},
    booktitle = {arXiv:2007.04074 [cs.LG]},
    year      = {2020}
}

----------------------------------------

Also, have a look at the blog on automl.org where we regularly release blogposts.

---

auto-sklearn

Doc/ Templates/Class

Doc/ Templates/Class Without Init

Doc/ Templates/Function

Doc/Api

`Doc/Extending`

`Doc/Faq`

`Doc/Index`

`Doc/Installation`

`Doc/Manual`

`Doc/Releases`

`CONTRIBUTING`

`Contributing to auto-sklearn`

Contributing

General steps

`Making Changes`

`Testing`

Creating the PR

`Pull Request Overview`

General FAQ

I've finished my pull request or made new changes and want it reviewed, now what?

I've been asked to rebase my pull request, why and what do I do?

README

auto-sklearn

auto-sklearn in one image

auto-sklearn in four lines of code

Relevant publications

Table of Contents

Doc/ Templates/Class Copy Section

Doc/ Templates/Class Without Init Copy Section

Doc/ Templates/Function Copy Section

Doc/Api Copy Section

Doc/Extending Copy Section

Doc/Faq Copy Section

Doc/Index Copy Section

Doc/Installation Copy Section

Doc/Manual Copy Section

Doc/Releases Copy Section

CONTRIBUTING Copy Section

Contributing to auto-sklearn

Contributing

General steps Copy Chapter

Making Changes Copy Chapter

Testing Copy Chapter

Creating the PR Copy Chapter

Pull Request Overview

General FAQ

I've finished my pull request or made new changes and want it reviewed, now what? Copy Section

I've been asked to rebase my pull request, why and what do I do? Copy Section

README Copy Section

auto-sklearn

auto-sklearn in one image Copy Chapter

auto-sklearn in four lines of code Copy Chapter

Relevant publications Copy Chapter

Explore Related Technologies & Topics

Doc/ Templates/Class

Doc/ Templates/Class Without Init

Doc/ Templates/Function

Doc/Api

`Doc/Extending`

`Doc/Faq`

`Doc/Index`

`Doc/Installation`

`Doc/Manual`

`Doc/Releases`

`CONTRIBUTING`

`Contributing to auto-sklearn`

General steps

`Making Changes`

`Testing`

Creating the PR

`Pull Request Overview`

I've finished my pull request or made new changes and want it reviewed, now what?

I've been asked to rebase my pull request, why and what do I do?

README

auto-sklearn in one image

auto-sklearn in four lines of code

Relevant publications