LightGBM Documentation

mkdir lightgbm-docker
cd lightgbm-docker
wget https://raw.githubusercontent.com/lightgbm-org/LightGBM/main/docker/gpu/dockerfile.gpu
docker build -f dockerfile.gpu -t lightgbm-gpu .

nvidia-docker run --rm -d --name lightgbm-gpu -p 8888:8888 -v /home:/home lightgbm-gpu

docker exec -it lightgbm-gpu bash

localhost:8888

mkdir lightgbm-docker
cd lightgbm-docker
wget https://raw.githubusercontent.com/lightgbm-org/LightGBM/main/docker/dockerfile-cli
docker build \
    -t lightgbm-cli \
    -f dockerfile-cli \
    .

# configure the CLI
cat << EOF > train.conf
task = train
objective = binary
data = binary.train
num_trees = 10
output_model = LightGBM-CLI-model.txt
EOF

# get training data
curl -O https://raw.githubusercontent.com/lightgbm-org/LightGBM/main/examples/binary_classification/binary.train

# train, and save model to a text file
docker run \
  --rm \
  --volume "${PWD}":/opt/training \
  --workdir /opt/training \
  lightgbm-cli \
  config=train.conf

mkdir lightgbm-docker
cd lightgbm-docker
wget https://raw.githubusercontent.com/lightgbm-org/LightGBM/main/docker/dockerfile-python
docker build \
    -t lightgbm-python \
    -f dockerfile-python \
    .

# get training data
curl -O https://raw.githubusercontent.com/lightgbm-org/LightGBM/main/examples/binary_classification/binary.train

# create training script
cat << EOF > train.py
import lightgbm as lgb
import numpy as np
params = {
    "objective": "binary",
    "num_trees": 10
}

bst = lgb.train(
    train_set=lgb.Dataset("binary.train"),
    params=params
)
bst.save_model("LightGBM-python-model.txt")
EOF

# run training in a container
docker run \
    --rm \
    --volume "${PWD}":/opt/training \
    --workdir /opt/training \
    lightgbm-python \
    python train.py

docker run \
    --rm \
    --volume "${PWD}":/opt/training \
    --workdir /opt/training \
    -it lightgbm-python \
    python

mkdir lightgbm-docker
cd lightgbm-docker
wget https://raw.githubusercontent.com/lightgbm-org/LightGBM/main/docker/dockerfile-r

docker build \
    -t lightgbm-r \
    -f dockerfile-r \
    .

# get training data
curl -O https://raw.githubusercontent.com/lightgbm-org/LightGBM/main/examples/binary_classification/binary.train

# create training script
cat << EOF > train.R
library(lightgbm)
params <- list(
    objective = "binary"
    , num_trees = 10L
)

bst <- lgb.train(
    data = lgb.Dataset("binary.train"),
    params = params
)
lgb.save(bst, "LightGBM-R-model.txt")
EOF

# run training in a container
docker run \
    --rm \
    --volume "${PWD}":/opt/training \
    --workdir /opt/training \
    lightgbm-r \
    Rscript train.R

docker run \
    --rm \
    -it lightgbm-r \
    R

docker run \
    --rm \
    --env PASSWORD="lightgbm" \
    -p 8787:8787 \
    lightgbm-r

docker build \
    -t lightgbm-r-45 \
    -f dockerfile-r \
    --build-arg R_VERSION=4.5 \
    .

0
0
0
1
1
0
0
0
1
...

+-------------------------+----------------------------------------------------------------------------------------+
| Class                   | Description                                                                            |
+=========================+========================================================================================+
| ``Application``         | The entrance of application, including training and prediction logic                   |
+-------------------------+----------------------------------------------------------------------------------------+
| ``Bin``                 | Data structure used for storing feature discrete values (converted from float values)  |
+-------------------------+----------------------------------------------------------------------------------------+
| ``Boosting``            | Boosting interface (GBDT, DART, etc.)                                                  |
+-------------------------+----------------------------------------------------------------------------------------+
| ``Config``              | Stores parameters and configurations                                                   |
+-------------------------+----------------------------------------------------------------------------------------+
| ``Dataset``             | Stores information of dataset                                                          |
+-------------------------+----------------------------------------------------------------------------------------+
| ``DatasetLoader``       | Used to construct dataset                                                              |
+-------------------------+----------------------------------------------------------------------------------------+
| ``FeatureGroup``        | Stores the data of feature, could be multiple features                                 |
+-------------------------+----------------------------------------------------------------------------------------+
| ``Metric``              | Evaluation metrics                                                                     |
+-------------------------+----------------------------------------------------------------------------------------+
| ``Network``             | Network interfaces and communication algorithms                                        |
+-------------------------+----------------------------------------------------------------------------------------+
| ``ObjectiveFunction``   | Objective functions used to train                                                      |
+-------------------------+----------------------------------------------------------------------------------------+
| ``Tree``                | Stores information of tree model                                                       |
+-------------------------+----------------------------------------------------------------------------------------+
| ``TreeLearner``         | Used to learn trees                                                                    |
+-------------------------+----------------------------------------------------------------------------------------+

Code Structure
~~~~~~~~~~~~~~

+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| Path                | Description                                                                                                                        |
+=====================+====================================================================================================================================+
| ./include           | Header files                                                                                                                       |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./include/utils     | Some common functions                                                                                                              |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./src/application   | Implementations of training and prediction logic                                                                                   |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./src/boosting      | Implementations of Boosting                                                                                                        |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./src/io            | Implementations of IO related classes, including ``Bin``, ``Config``, ``Dataset``, ``DatasetLoader``, ``Feature`` and ``Tree``     |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./src/metric        | Implementations of metrics                                                                                                         |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./src/network       | Implementations of network functions                                                                                               |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./src/objective     | Implementations of objective functions                                                                                             |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+
| ./src/treelearner   | Implementations of tree learners                                                                                                   |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------+

Documents API
-------------

Refer to `docs README <./README.rst>`__.

C API
-----

Refer to `C API <./C-API.rst>`__ or the comments in `c\_api.h <https://github.com/lightgbm-org/LightGBM/blob/main/include/LightGBM/c_api.h>`__ file, from which the documentation is generated.

Tests
-----

C++ unit tests are located in the ``./tests/cpp_tests`` folder and written with the help of Google Test framework.
To run tests locally first refer to the `Installation Guide <./Installation-Guide.rst#build-c-unit-tests>`__ for how to build tests and then simply run compiled executable file.
It is highly recommended to build tests with `sanitizers <./Installation-Guide.rst#sanitizers>`__.

High Level Language Package
---------------------------

See the implementations at `Python-package <https://github.com/lightgbm-org/LightGBM/tree/main/python-package>`__ and `R-package <https://github.com/lightgbm-org/LightGBM/tree/main/R-package>`__.

Questions
---------

Refer to `FAQ <./FAQ.rst>`__.

Also feel free to open `issues <https://github.com/lightgbm-org/LightGBM/issues>`__ if you met problems.

---

## File: docs/Experiments.rst

Experiments
===========

Comparison Experiment
---------------------

For the detailed experiment scripts and output logs, please refer to this `repo`_.

History
^^^^^^^

08 Mar, 2020: update according to the latest master branch (`1b97eaf <https://github.com/dmlc/xgboost/commit/1b97eaf7a74315bfa2c132d59f937a35408bcfd1>`__ for XGBoost, `bcad692 <https://github.com/lightgbm-org/LightGBM/commit/bcad692e263e0317cab11032dd017c78f9e58e5f>`__ for LightGBM). (``xgboost_exact`` is not updated for it is too slow.)

27 Feb, 2017: first version.

Data
^^^^

We used 5 datasets to conduct our comparison experiments. Details of data are listed in the following table:

+-----------+-----------------------+---------------------------------------------------------------------------------+-------------+----------+----------------------------------------------+
| Data      | Task                  | Link                                                                            | #Train\_Set | #Feature | Comments                                     |
+===========+=======================+=================================================================================+=============+==========+==============================================+
| Higgs     | Binary classification | `link <https://archive.ics.uci.edu/dataset/280/higgs>`__                        | 10,500,000  | 28       | last 500,000 samples were used as test set   |
+-----------+-----------------------+---------------------------------------------------------------------------------+-------------+----------+----------------------------------------------+
| Yahoo LTR | Learning to rank      | `link <https://proceedings.mlr.press/v14/chapelle11a.html>`__                   | 473,134     | 700      | set1.train as train, set1.test as test       |
+-----------+-----------------------+---------------------------------------------------------------------------------+-------------+----------+----------------------------------------------+
| MS LTR    | Learning to rank      | `link <https://www.microsoft.com/en-us/research/project/mslr/>`__               | 2,270,296   | 137      | {S1,S2,S3} as train set, {S5} as test set    |
+-----------+-----------------------+---------------------------------------------------------------------------------+-------------+----------+----------------------------------------------+
| Expo      | Binary classification | `link <https://community.amstat.org/jointscsg-section/dataexpo/dataexpo2009>`__ | 11,000,000  | 700      | last 1,000,000 samples were used as test set |
+-----------+-----------------------+---------------------------------------------------------------------------------+-------------+----------+----------------------------------------------+
| Allstate  | Binary classification | `link <https://www.kaggle.com/c/ClaimPredictionChallenge>`__                    | 13,184,290  | 4228     | last 1,000,000 samples were used as test set |
+-----------+-----------------------+---------------------------------------------------------------------------------+-------------+----------+----------------------------------------------+

Environment
^^^^^^^^^^^

We ran all experiments on a single Linux server (Azure ND24s) with the following specifications:

+------------------+-----------------+---------------------+
| OS               | CPU             | Memory              |
+==================+=================+=====================+
| Ubuntu 16.04 LTS | 2 \* E5-2690 v4 | 448GB               |
+------------------+-----------------+---------------------+

Baseline
^^^^^^^^

We used `xgboost`_ as a baseline.

Both xgboost and LightGBM were built with OpenMP support.

Settings
^^^^^^^^

We set up total 3 settings for experiments. The parameters of these settings are:

1. xgboost:

   .. code:: text

       eta = 0.1
       max_depth = 8
       num_round = 500
       nthread = 16
       tree_method = exact
       min_child_weight = 100

2. xgboost\_hist (using histogram based algorithm):

   .. code:: text

       eta = 0.1
       num_round = 500
       nthread = 16
       min_child_weight = 100
       tree_method = hist
       grow_policy = lossguide
       max_depth = 0
       max_leaves = 255

3. LightGBM:

   .. code:: text

       learning_rate = 0.1
       num_leaves = 255
       num_trees = 500
       num_threads = 16
       min_data_in_leaf = 0
       min_sum_hessian_in_leaf = 100

xgboost grows trees depth-wise and controls model complexity by ``max_depth``.
LightGBM uses a leaf-wise algorithm instead and controls model complexity by ``num_leaves``.
So we cannot compare them in the exact same model setting. For the tradeoff, we use xgboost with ``max_depth=8``, which will have max number leaves to 255, to compare with LightGBM with ``num_leaves=255``.

Other parameters are default values.

Result
^^^^^^

Speed
'''''

We compared speed using only the training task without any test or metric output. We didn't count the time for IO.
For the ranking tasks, since XGBoost and LightGBM implement different ranking objective functions, we used ``regression`` objective for speed benchmark, for the fair comparison.

The following table is the comparison of time cost:

+-----------+-----------+---------------+---------------+
| Data      | xgboost   | xgboost\_hist | LightGBM      |
+===========+===========+===============+===============+
| Higgs     | 3794.34 s | 165.575 s     | **130.094 s** |
+-----------+-----------+---------------+---------------+
| Yahoo LTR | 674.322 s | 131.462 s     | **76.229 s**  |
+-----------+-----------+---------------+---------------+
| MS LTR    | 1251.27 s | 98.386 s      | **70.417 s**  |
+-----------+-----------+---------------+---------------+
| Expo      | 1607.35 s | 137.65 s      | **62.607 s**  |
+-----------+-----------+---------------+---------------+
| Allstate  | 2867.22 s | 315.256 s     | **148.231 s** |
+-----------+-----------+---------------+---------------+

LightGBM ran faster than xgboost on all experiment data sets.

Accuracy
''''''''

We computed all accuracy metrics only on the test data set.

+-----------+-----------------+----------+-------------------+--------------+
| Data      | Metric          | xgboost  | xgboost\_hist     | LightGBM     |
+===========+=================+==========+===================+==============+
| Higgs     | AUC             | 0.839593 | 0.845314          | **0.845724** |
+-----------+-----------------+----------+-------------------+--------------+
| Yahoo LTR | NDCG\ :sub:`1`  | 0.719748 | 0.720049          | **0.732981** |
|           +-----------------+----------+-------------------+--------------+
|           | NDCG\ :sub:`3`  | 0.717813 | 0.722573          | **0.735689** |
|           +-----------------+----------+-------------------+--------------+
|           | NDCG\ :sub:`5`  | 0.737849 | 0.740899          | **0.75352**  |
|           +-----------------+----------+-------------------+--------------+
|           | NDCG\ :sub:`10` | 0.78089  | 0.782957          | **0.793498** |
+-----------+-----------------+----------+-------------------+--------------+
| MS LTR    | NDCG\ :sub:`1`  | 0.483956 | 0.485115          | **0.517767** |
|           +-----------------+----------+-------------------+--------------+
|           | NDCG\ :sub:`3`  | 0.467951 | 0.47313           | **0.501063** |
|           +-----------------+----------+-------------------+--------------+
|           | NDCG\ :sub:`5`  | 0.472476 | 0.476375          | **0.504648** |
|           +-----------------+----------+-------------------+--------------+
|           | NDCG\ :sub:`10` | 0.492429 | 0.496553          | **0.524252** |
+-----------+-----------------+----------+-------------------+--------------+
| Expo      | AUC             | 0.756713 | 0.776224          | **0.776935** |
+-----------+-----------------+----------+-------------------+--------------+
| Allstate  | AUC             | 0.607201 | **0.609465**      |  0.609072    |
+-----------+-----------------+----------+-------------------+--------------+

Memory Consumption
''''''''''''''''''

We monitored RES while running training task. And we set ``two_round=true`` (this will increase data-loading time and
reduce peak memory usage but not affect training speed or accuracy) in LightGBM to reduce peak memory usage.

+-----------+---------+---------------+--------------------+--------------------+
| Data      | xgboost | xgboost\_hist | LightGBM (col-wise)|LightGBM (row-wise) |
+===========+=========+===============+====================+====================+
| Higgs     | 4.853GB | 7.335GB       | **0.897GB**        |     1.401GB        |
+-----------+---------+---------------+--------------------+--------------------+
| Yahoo LTR | 1.907GB | 4.023GB       | **1.741GB**        |     2.161GB        |
+-----------+---------+---------------+--------------------+--------------------+
| MS LTR    | 5.469GB | 7.491GB       | **0.940GB**        |     1.296GB        |
+-----------+---------+---------------+--------------------+--------------------+
| Expo      | 1.553GB | 2.606GB       | **0.555GB**        |     0.711GB        |
+-----------+---------+---------------+--------------------+--------------------+
| Allstate  | 6.237GB | 12.090GB      | **1.116GB**        |     1.755GB        |
+-----------+---------+---------------+--------------------+--------------------+

Parallel Experiment
-------------------

History
^^^^^^^

27 Feb, 2017: first version.

Data
^^^^

We used a terabyte click log dataset to conduct parallel experiments. Details are listed in following table:

+--------+-----------------------+---------+---------------+----------+
| Data   | Task                  | Link    | #Data         | #Feature |
+========+=======================+=========+===============+==========+
| Criteo | Binary classification | `link`_ | 1,700,000,000 | 67       |
+--------+-----------------------+---------+---------------+----------+

This data contains 13 integer features and 26 categorical features for 24 days of click logs.
We statisticized the click-through rate (CTR) and count for these 26 categorical features from the first ten days.
Then we used next ten days' data, after replacing the categorical features by the corresponding CTR and count, as training data.
The processed training data have a total of 1.7 billions records and 67 features.

Environment
^^^^^^^^^^^

We ran our experiments on 16 Windows servers with the following specifications:

+---------------------+-----------------+---------------------+-------------------------------------------+
| OS                  | CPU             | Memory              | Network Adapter                           |
+=====================+=================+=====================+===========================================+
| Windows Server 2012 | 2 \* E5-2670 v2 | DDR3 1600Mhz, 256GB | Mellanox ConnectX-3, 54Gbps, RDMA support |
+---------------------+-----------------+---------------------+-------------------------------------------+

Settings
^^^^^^^^

.. code:: text

    learning_rate = 0.1
    num_leaves = 255
    num_trees = 100
    num_thread = 16
    tree_learner = data

We used data parallel here because this data is large in ``#data`` but small in ``#feature``. Other parameters were default values.

Results
^^^^^^^

+----------+---------------+---------------------------+
| #Machine | Time per Tree | Memory Usage(per Machine) |
+==========+===============+===========================+
| 1        | 627.8 s       | 176GB                     |
+----------+---------------+---------------------------+
| 2        | 311 s         | 87GB                      |
+----------+---------------+---------------------------+
| 4        | 156 s         | 43GB                      |
+----------+---------------+---------------------------+
| 8        | 80 s          | 22GB                      |
+----------+---------------+---------------------------+
| 16       | 42 s          | 11GB                      |
+----------+---------------+---------------------------+

The results show that LightGBM achieves a linear speedup with distributed learning.

GPU Experiments
---------------

Refer to `GPU Performance <./GPU-Performance.rst>`__.

.. _repo: https://github.com/guolinke/boosting_tree_benchmarks

.. _xgboost: https://github.com/dmlc/xgboost

.. _link: https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/

---

## File: docs/FAQ.rst

.. role:: raw-html(raw)
    :format: html

LightGBM FAQ
############

.. contents:: LightGBM Frequently Asked Questions
    :depth: 1
    :local:
    :backlinks: none

------

Please post questions, feature requests, and bug reports at https://github.com/lightgbm-org/LightGBM/issues.

This project is mostly maintained by volunteers, so please be patient.
If your request is time-sensitive or more than a month goes by without a response, please tag the maintainers below for help.

-  `@guolinke <https://github.com/guolinke>`__ **Guolin Ke**
-  `@shiyu1994 <https://github.com/shiyu1994>`__ **Yu Shi**
-  `@jameslamb <https://github.com/jameslamb>`__ **James Lamb**
-  `@jmoralez <https://github.com/jmoralez>`__ **José Morales**
-  `@borchero <https://github.com/borchero>`__ **Oliver Borchert**
-  `@mayer79 <https://github.com/mayer79>`__ **Michael Mayer**

--------------

General LightGBM Questions
==========================

.. contents::
    :local:
    :backlinks: none

1. Where do I find more details about LightGBM parameters?
----------------------------------------------------------

Take a look at `Parameters <./Parameters.rst>`__.

2. On datasets with millions of features, training does not start (or starts after a very long time).
-----------------------------------------------------------------------------------------------------

Use a smaller value for ``bin_construct_sample_cnt`` and a larger value for ``min_data``.

3. When running LightGBM on a large dataset, my computer runs out of RAM.
-------------------------------------------------------------------------

**Multiple Solutions**: set the ``histogram_pool_size`` parameter to the MB you want to use for LightGBM (histogram\_pool\_size + dataset size = approximately RAM used),
lower ``num_leaves`` or lower ``max_bin`` (see `lightgbm-org/LightGBM#562 <https://github.com/lightgbm-org/LightGBM/issues/562>`__).

4. I am using Windows. Should I use Visual Studio or MinGW for compiling LightGBM?
----------------------------------------------------------------------------------

Visual Studio `performs best for LightGBM <https://github.com/lightgbm-org/LightGBM/issues/542>`__.

5. When using LightGBM GPU, I cannot reproduce results over several runs.
-------------------------------------------------------------------------

This is normal and expected behaviour, but you may try to use ``gpu_use_dp = true`` for reproducibility
(see `lightgbm-org/LightGBM#560 <https://github.com/lightgbm-org/LightGBM/pull/560#issuecomment-304561654>`__).
You may also use the CPU version.

6. Bagging is not reproducible when changing the number of threads.
-------------------------------------------------------------------

:raw-html:`<strike>`
LightGBM bagging is multithreaded, so its output depends on the number of threads used.
There is `no workaround currently <https://github.com/lightgbm-org/LightGBM/issues/632>`__.
:raw-html:`</strike>`

Starting from `#2804 <https://github.com/lightgbm-org/LightGBM/pull/2804>`__ bagging result doesn't depend on the number of threads.
So this issue should be solved in the latest version.

7. I tried to use Random Forest mode, and LightGBM crashes!
-----------------------------------------------------------

This is expected behaviour for arbitrary parameters. To enable Random Forest,
you must use ``bagging_fraction`` and ``feature_fraction`` different from 1, along with a ``bagging_freq``.
`This thread <https://github.com/lightgbm-org/LightGBM/issues/691>`__ includes an example.

8. CPU usage is low (like 10%) in Windows when using LightGBM on very large datasets with many-core systems.
------------------------------------------------------------------------------------------------------------

Please use `Visual Studio <https://visualstudio.microsoft.com/downloads/>`__
as it may be `10x faster than MinGW <https://github.com/lightgbm-org/LightGBM/issues/749>`__ especially for very large trees.

9. When I'm trying to specify a categorical column with the ``categorical_feature`` parameter, I get the following sequence of warnings, but there are no negative values in the column.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

.. code-block:: console

   [LightGBM] [Warning] Met negative value in categorical features, will convert it to NaN
   [LightGBM] [Warning] There are no meaningful features, as all feature values are constant.

The column you're trying to pass via ``categorical_feature`` likely contains very large values.
Categorical features in LightGBM are limited by int32 range,
so you cannot pass values that are greater than ``Int32.MaxValue`` (2147483647) as categorical features (see `lightgbm-org/LightGBM#1359 <https://github.com/lightgbm-org/LightGBM/issues/1359>`__).
You should convert them to integers ranging from zero to the number of categories first.

10. LightGBM crashes randomly with the error like: ``Initializing libiomp5.dylib, but found libomp.dylib already initialized.``
-------------------------------------------------------------------------------------------------------------------------------

.. code-block:: console

   OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized.
   OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

**Possible Cause**: This error means that you have multiple OpenMP libraries installed on your machine and they conflict with each other.
(File extensions in the error message may differ depending on the operating system).

If you are using Python distributed by Conda, then it is highly likely that the error is caused by the ``numpy`` package from Conda which includes the ``mkl`` package which in turn conflicts with the system-wide library.
In this case you can update the ``numpy`` package in Conda or replace the Conda's OpenMP library instance with system-wide one by creating a symlink to it in Conda environment folder ``$CONDA_PREFIX/lib``.

**Solution**: Assuming you are using macOS with Homebrew, the command which overwrites OpenMP library files in the current active Conda environment with symlinks to the system-wide library ones installed by Homebrew:

.. code-block:: bash

   ln -sf `ls -d "$(brew --cellar libomp)"/*/lib`/* $CONDA_PREFIX/lib

The described above fix worked fine before the release of OpenMP 8.0.0 version.
Starting from 8.0.0 version, Homebrew formula for OpenMP includes ``-DLIBOMP_INSTALL_ALIASES=OFF`` option which leads to that the fix doesn't work anymore.
However, you can create symlinks to library aliases manually:

.. code-block:: bash

   for LIBOMP_ALIAS in libgomp.dylib libiomp5.dylib libomp.dylib; do sudo ln -sf "$(brew --cellar libomp)"/*/lib/libomp.dylib $CONDA_PREFIX/lib/$LIBOMP_ALIAS; done

Another workaround would be removing MKL optimizations from Conda's packages completely:

.. code-block:: bash

    conda install nomkl

If this is not your case, then you should find conflicting OpenMP library installations on your own and leave only one of them.

11. LightGBM hangs when multithreading (OpenMP) and using forking in Linux at the same time.
--------------------------------------------------------------------------------------------

Use ``nthreads=1`` to disable multithreading of LightGBM. There is a bug with OpenMP which hangs forked sessions
with multithreading activated. A more expensive solution is to use new processes instead of using fork, however,
keep in mind it is creating new processes where you have to copy memory and load libraries (example: if you want to
fork 16 times your current process, then you will require to make 16 copies of your dataset in memory)
(see `lightgbm-org/LightGBM#1789 <https://github.com/lightgbm-org/LightGBM/issues/1789#issuecomment-433713383>`__).

An alternative, if multithreading is really necessary inside the forked sessions, would be to compile LightGBM with
Intel toolchain. Intel compilers are unaffected by this bug.

For C/C++ users, any OpenMP feature cannot be used before the fork happens. If an OpenMP feature is used before the
fork happens (example: using OpenMP for forking), OpenMP will hang inside the forked sessions. Use new processes instead
and copy memory as required by creating new processes instead of forking (or, use Intel compilers).

Cloud platform container services may cause LightGBM to hang, if they use Linux fork to run multiple containers on a
single instance. For example, LightGBM hangs in AWS Batch array jobs, which `use the ECS agent
<https://aws.amazon.com/batch/faqs/>`__ to manage multiple running jobs. Setting ``nthreads=1`` mitigates the issue.

12. Why is early stopping not enabled by default in LightGBM?
-------------------------------------------------------------

Early stopping involves choosing a validation set, a special type of holdout which is used to evaluate the current state of the model after each iteration to see if training can stop.

In ``LightGBM``, `we have decided to require that users specify this set directly <./Parameters.rst#valid>`_. Many options exist for splitting training data into training, test, and validation sets.

The appropriate splitting strategy depends on the task and domain of the data, information that a modeler has but which ``LightGBM`` as a general-purpose tool does not.

13. Does LightGBM support direct loading data from zero-based or one-based LibSVM format file?
----------------------------------------------------------------------------------------------

LightGBM supports loading data from zero-based LibSVM format file directly.

14. Why CMake cannot find the compiler when compiling LightGBM with MinGW?
--------------------------------------------------------------------------

.. code-block:: bash

    CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
    CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage

This is a known issue of CMake when using MinGW. The easiest solution is to run again your ``cmake`` command to bypass the one time stopper from CMake. Or you can upgrade your version of CMake to at least version 3.17.0.

See `lightgbm-org/LightGBM#3060 <https://github.com/lightgbm-org/LightGBM/issues/3060#issuecomment-626338538>`__ for more details.

15. Where can I find LightGBM's logo to use it in my presentation?
------------------------------------------------------------------

You can find LightGBM's logo in different file formats and resolutions `here <https://github.com/lightgbm-org/LightGBM/tree/main/docs/logo>`__.

16. LightGBM crashes randomly or operating system hangs during or after running LightGBM.
-----------------------------------------------------------------------------------------

**Possible Cause**: This behavior may indicate that you have multiple OpenMP libraries installed on your machine and they conflict with each other, similarly to the ``FAQ #10``.

If you are using any Python-package that depends on ``threadpoolctl``, you also may see the following warning in your logs in this case:

.. code-block:: console

    /root/miniconda/envs/test-env/lib/python3.8/site-packages/threadpoolctl.py:546: RuntimeWarning:
    Found Intel OpenMP ('libiomp') and LLVM OpenMP ('libomp') loaded at
    the same time. Both libraries are known to be incompatible and this
    can cause random crashes or deadlocks on Linux when loaded in the
    same Python program.
    Using threadpoolctl may cause crashes or deadlocks. For more
    information and possible workarounds, please see
        https://github.com/joblib/threadpoolctl/blob/master/multiple_openmp.md

Detailed description of conflicts between multiple OpenMP instances is provided in the `following document <https://github.com/joblib/threadpoolctl/blob/master/multiple_openmp.md>`__.

**Solution**: Assuming you are using LightGBM Python-package and conda as a package manager, we strongly recommend using ``conda-forge`` channel as the only source of all your Python package installations because it contains built-in patches to workaround OpenMP conflicts. Some other workarounds are listed `here <https://github.com/joblib/threadpoolctl/blob/master/multiple_openmp.md>`__ under the "Workarounds for Intel OpenMP and LLVM OpenMP case" section.

If this is not your case, then you should find conflicting OpenMP library installations on your own and leave only one of them.

17. Loading LightGBM fails like: ``cannot allocate memory in static TLS block``
-------------------------------------------------------------------------------

When loading LightGBM, you may encounter errors like the following.

.. code-block:: console

   lib/libgomp.so.1: cannot allocate memory in static TLS block

This most commonly happens on aarch64 Linux systems.

``gcc``'s OpenMP library (``libgomp.so``) tries to allocate a small amount of static thread-local storage ("TLS")
when it's dynamically loaded.

That error can happen when the loader isn't able to find a large enough block of memory.

On aarch64 Linux, processes and loaded libraries share the same pool of static TLS,
which makes such failures more likely. See these discussions:

* https://bugzilla.redhat.com/show_bug.cgi?id=1722181#c6
* https://github.com/pytorch/pytorch/issues/2575

If you are experiencing this issue when using the ``lightgbm`` Python-package, try upgrading
to at least ``v4.6.0``.

For older versions of the Python-package, or for other LightGBM APIs, this issue can
often be avoided by loading ``libgomp.so.1``. That can be done directly by setting environment
variable ``LD_PRELOAD``, like this:

.. code-block:: console

    export LD_PRELOAD=/root/miniconda3/envs/test-env/lib/libgomp.so.1

It can also be done indirectly by changing the order that other libraries are loaded
into processes, which varies by programming language and application type.

For more details, see these discussions:

* https://github.com/lightgbm-org/LightGBM/pull/6654#issuecomment-2352014275
* https://github.com/lightgbm-org/LightGBM/issues/6509
* https://maskray.me/blog/2021-02-14-all-about-thread-local-storage
* https://bugzilla.redhat.com/show_bug.cgi?id=1722181#c6

------

R-package
=========

.. contents::
    :local:
    :backlinks: none

1. Any training command using LightGBM does not work after an error occurred during the training of a previous LightGBM model.
------------------------------------------------------------------------------------------------------------------------------

In older versions of the R-package (prior to ``v3.3.0``), this could happen occasionally and the solution was to run ``lgb.unloader(wipe = TRUE)`` to remove all LightGBM-related objects. Some conversation about this could be found in `lightgbm-org/LightGBM#698 <https://github.com/lightgbm-org/LightGBM/issues/698>`__.

That is no longer necessary as of ``v3.3.0``, and function ``lgb.unloader()`` has since been removed from the R-package.

2. I used ``setinfo()``, tried to print my ``lgb.Dataset``, and now the R console froze!
----------------------------------------------------------------------------------------

As of at least LightGBM v3.3.0, this issue has been resolved and printing a ``Dataset`` object does not cause the console to freeze.

In older versions, avoid printing the ``Dataset`` after calling ``setinfo()``.

As of LightGBM v4.0.0, ``setinfo()`` has been replaced by a new method, ``set_field()``.

3. ``error in data.table::data.table()...argument 2 is NULL``.
--------------------------------------------------------------

If you are experiencing this error when running ``lightgbm``, you may be facing the same issue reported in `#2715 <https://github.com/lightgbm-org/LightGBM/issues/2715>`_ and later in `#2989 <https://github.com/lightgbm-org/LightGBM/pull/2989#issuecomment-614374151>`_. We have seen that in some situations, using ``data.table`` 1.11.x results in this error. To get around this, you can upgrade your version of ``data.table`` to at least version 1.12.0.

4. ``package/dependency ‘Matrix’ is not available ...``
-------------------------------------------------------

In April 2024, ``Matrix==1.7-0`` was published to CRAN.
That version had a floor of ``R (>=4.4.0)``.
``{Matrix}`` is a hard runtime dependency of ``{lightgbm}``, so on any version of R older than ``4.4.0``, running ``install.packages("lightgbm")`` results in something like the following.

.. code-block:: text

    package ‘Matrix’ is not available for this version of R

To fix that without upgrading to R 4.4.0 or greater, manually install an older version of ``{Matrix}``.

.. code-block:: R

    install.packages('https://cran.r-project.org/src/contrib/Archive/Matrix/Matrix_1.6-5.tar.gz', repos = NULL)

------

Python-package
==============

.. contents::
    :local:
    :backlinks: none

1. ``Error: setup script specifies an absolute path`` when installing from GitHub using ``python setup.py install``.
--------------------------------------------------------------------------------------------------------------------

.. note::
    As of v4.0.0, ``lightgbm`` does not support directly invoking ``setup.py``.
    This answer refers only to versions of ``lightgbm`` prior to v4.0.0.

.. code-block:: console

   error: Error: setup script specifies an absolute path:
   /Users/Microsoft/LightGBM/python-package/lightgbm/../../lib_lightgbm.so
   setup() arguments must *always* be /-separated paths relative to the setup.py directory, *never* absolute paths.

This error should be solved in latest version.
If you still meet this error, try to remove ``lightgbm.egg-info`` folder in your Python-package and reinstall,
or check `this thread on stackoverflow <https://stackoverflow.com/questions/18085571/pip-install-error-setup-script-specifies-an-absolute-path>`__.

2. Error messages: ``Cannot ... before construct dataset``.
-----------------------------------------------------------

I see error messages like...

.. code-block:: console

   Cannot get/set label/weight/init_score/group/num_data/num_feature before construct dataset

but I've already constructed a dataset by some code like:

.. code-block:: python

    train = lightgbm.Dataset(X_train, y_train)

or error messages like

.. code-block:: console

    Cannot set predictor/reference/categorical feature after freed raw data, set free_raw_data=False when construct Dataset to avoid this.

**Solution**: Because LightGBM constructs bin mappers to build trees, and train and valid Datasets within one Booster share the same bin mappers,
categorical features and feature names etc., the Dataset objects are constructed when constructing a Booster.
If you set ``free_raw_data=True`` (default), the raw data (with Python data struct) will be freed.
So, if you want to:

-  get label (or weight/init\_score/group/data) before constructing a dataset, it's same as get ``self.label``;

-  set label (or weight/init\_score/group) before constructing a dataset, it's same as ``self.label=some_label_array``;

-  get num\_data (or num\_feature) before constructing a dataset, you can get data with ``self.data``.
   Then, if your data is ``numpy.ndarray``, use some code like ``self.data.shape``. But do not do this after subsetting the Dataset, because you'll get always ``None``;

-  set predictor (or reference/categorical feature) after constructing a dataset,
   you should set ``free_raw_data=False`` or init a Dataset object with the same raw data.

3. I encounter segmentation faults (segfaults) randomly after installing LightGBM from PyPI using ``pip install lightgbm``.
---------------------------------------------------------------------------------------------------------------------------

We are doing our best to provide universal wheels which have high running speed and are compatible with any hardware, OS, compiler, etc. at the same time.
However, sometimes it's just impossible to guarantee the possibility of usage of LightGBM in any specific environment (see `lightgbm-org/LightGBM#1743 <https://github.com/lightgbm-org/LightGBM/issues/1743>`__).

Therefore, the first thing you should try in case of segfaults is **compiling from the source** using ``pip install --no-binary lightgbm lightgbm``.
For the OS-specific prerequisites see https://github.com/lightgbm-org/LightGBM/blob/main/python-package/README.rst.

Also, feel free to post a new issue in our GitHub repository. We always look at each case individually and try to find a root cause.

4. I would like to install LightGBM from conda. What channel should I choose?
-----------------------------------------------------------------------------

We strongly recommend installation from the ``conda-forge`` channel and not from the ``default`` one.

For some specific examples, see `this comment <https://github.com/lightgbm-org/LightGBM/issues/4948#issuecomment-1013766397>`__.

In addition, as of ``lightgbm==4.4.0``, the ``conda-forge`` package automatically supports CUDA-based GPU acceleration.

5. How do I subclass ``scikit-learn`` estimators?
-------------------------------------------------

For ``lightgbm <= 4.5.0``, copy all of the constructor arguments from the corresponding
``lightgbm`` class into the constructor of your custom estimator.

For later versions, just ensure that the constructor of your custom estimator calls ``super().__init__()``.

Consider the example below, which implements a regressor that allows creation of truncated predictions.
This pattern will work with ``lightgbm > 4.5.0``.

.. code-block:: python

    import numpy as np
    from lightgbm import LGBMRegressor
    from sklearn.datasets import make_regression

    class TruncatedRegressor(LGBMRegressor):

        def __init__(self, **kwargs):
            super().__init__(**kwargs)

        def predict(self, X, max_score: float = np.inf):
            preds = super().predict(X)
            np.clip(preds, a_min=None, a_max=max_score, out=preds)
            return preds

    X, y = make_regression(n_samples=1_000, n_features=4)

    reg_trunc = TruncatedRegressor().fit(X, y)

    preds = reg_trunc.predict(X)
    print(f"mean: {preds.mean():.2f}, max: {preds.max():.2f}")
    # mean: -6.81, max: 345.10

    preds_trunc = reg_trunc.predict(X, max_score=preds.mean())
    print(f"mean: {preds_trunc.mean():.2f}, max: {preds_trunc.max():.2f}")
    # mean: -56.50, max: -6.81

---

## File: docs/Features.rst

Features
========

This is a conceptual overview of how LightGBM works\ `[1] <#references>`__. We assume familiarity with decision tree boosting algorithms to focus instead on aspects of LightGBM that may differ from other boosting packages. For detailed algorithms, please refer to the citations or source code.

Optimization in Speed and Memory Usage
--------------------------------------

Many boosting tools use pre-sort-based algorithms\ `[2, 3] <#references>`__ (e.g. default algorithm in xgboost) for decision tree learning. It is a simple solution, but not easy to optimize.

LightGBM uses histogram-based algorithms\ `[4, 5, 6] <#references>`__, which bucket continuous feature (attribute) values into discrete bins. This speeds up training and reduces memory usage. Advantages of histogram-based algorithms include the following:

-  **Reduced cost of calculating the gain for each split**

   -  Pre-sort-based algorithms have time complexity ``O(#data)``

   -  Computing the histogram has time complexity ``O(#data)``, but this involves only a fast sum-up operation. Once the histogram is constructed, a histogram-based algorithm has time complexity ``O(#bins)``, and ``#bins`` is far smaller than ``#data``.

-  **Use histogram subtraction for further speedup**

   -  To get one leaf's histograms in a binary tree, use the histogram subtraction of its parent and its neighbor

   -  So it needs to construct histograms for only one leaf (with smaller ``#data`` than its neighbor). It then can get histograms of its neighbor by histogram subtraction with small cost (``O(#bins)``)

-  **Reduce memory usage**

   -  Replaces continuous values with discrete bins. If ``#bins`` is small, can use small data type, e.g. uint8\_t, to store training data

   -  No need to store additional information for pre-sorting feature values

-  **Reduce communication cost for distributed learning**

Sparse Optimization
-------------------

-  Need only ``O(2 * #non_zero_data)`` to construct histogram for sparse features

Optimization in Accuracy
------------------------

Leaf-wise (Best-first) Tree Growth

It is common to represent categorical features with one-hot encoding, but this approach is suboptimal for tree learners. Particularly for high-cardinality categorical features, a tree built on one-hot features tends to be unbalanced and needs to grow very deep to achieve good accuracy.

Instead of one-hot encoding, the optimal solution is to split on a categorical feature by partitioning its categories into 2 subsets. If the feature has ``k`` categories, there are ``2^(k-1) - 1`` possible partitions.
But there is an efficient solution for regression trees\ `[8] <#references>`__. It needs about ``O(k * log(k))`` to find the optimal partition.

The basic idea is to sort the categories according to the training objective at each split.
More specifically, LightGBM sorts the histogram (for a categorical feature) according to its accumulated values (``sum_gradient / sum_hessian``) and then finds the best split on the sorted histogram.

Optimization in Network Communication
-------------------------------------

It only needs to use some collective communication algorithms, like "All reduce", "All gather" and "Reduce scatter", in distributed learning of LightGBM.
LightGBM implements state-of-the-art algorithms\ `[9] <#references>`__.
These collective communication algorithms can provide much better performance than point-to-point communication.

.. _Optimization in Parallel Learning:

Optimization in Distributed Learning
------------------------------------

LightGBM provides the following distributed learning algorithms.

Feature Parallel
~~~~~~~~~~~~~~~~

Traditional Algorithm
^^^^^^^^^^^^^^^^^^^^^

Feature parallel aims to parallelize the "Find Best Split" in the decision tree. The procedure of traditional feature parallel is:

1. Partition data vertically (different machines have different feature set).

2. Workers find local best split point {feature, threshold} on local feature set.

3. Communicate local best splits with each other and get the best one.

4. Worker with best split to perform split, then send the split result of data to other workers.

5. Other workers split data according to received data.

The shortcomings of traditional feature parallel:

-  Has computation overhead, since it cannot speed up "split", whose time complexity is ``O(#data)``.
   Thus, feature parallel cannot speed up well when ``#data`` is large.

-  Need communication of split result, which costs about ``O(#data / 8)`` (one bit for one data).

Feature Parallel in LightGBM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Since feature parallel cannot speed up well when ``#data`` is large, we make a little change: instead of partitioning data vertically, every worker holds the full data.
Thus, LightGBM doesn't need to communicate for split result of data since every worker knows how to split data.
And ``#data`` won't be larger, so it is reasonable to hold the full data in every machine.

The procedure of feature parallel in LightGBM:

1. Workers find local best split point {feature, threshold} on local feature set.

2. Communicate local best splits with each other and get the best one.

3. Perform best split.

However, this feature parallel algorithm still suffers from computation overhead for "split" when ``#data`` is large.
So it will be better to use data parallel when ``#data`` is large.

Data Parallel
~~~~~~~~~~~~~

Traditional Algorithm
^^^^^^^^^^^^^^^^^^^^^

Data parallel aims to parallelize the whole decision learning. The procedure of data parallel is:

1. Partition data horizontally.

2. Workers use local data to construct local histograms.

3. Merge global histograms from all local histograms.

4. Find best split from merged global histograms, then perform splits.

The shortcomings of traditional data parallel:

-  High communication cost.
   If using point-to-point communication algorithm, communication cost for one machine is about ``O(#machine * #feature * #bin)``.
   If using collective communication algorithm (e.g. "All Reduce"), communication cost is about ``O(2 * #feature * #bin)`` (check cost of "All Reduce" in chapter 4.5 at `[9] <#references>`__).

Data Parallel in LightGBM
^^^^^^^^^^^^^^^^^^^^^^^^^

We reduce communication cost of data parallel in LightGBM:

1. Instead of "Merge global histograms from all local histograms", LightGBM uses "Reduce Scatter" to merge histograms of different (non-overlapping) features for different workers.
   Then workers find the local best split on local merged histograms and sync up the global best split.

2. As aforementioned, LightGBM uses histogram subtraction to speed up training.
   Based on this, we can communicate histograms only for one leaf, and get its neighbor's histograms by subtraction as well.

All things considered, data parallel in LightGBM has time complexity ``O(0.5 * #feature * #bin)``.

Voting Parallel
~~~~~~~~~~~~~~~

Voting parallel further reduces the communication cost in `Data Parallel <#data-parallel>`__ to constant cost.
It uses two-stage voting to reduce the communication cost of feature histograms\ `[10] <#references>`__.

GPU Support
-----------

Thanks `@huanzhang12 <https://github.com/huanzhang12>`__ for contributing this feature. Please read `[11] <#references>`__ to get more details.

- `GPU Installation <./Installation-Guide.rst#build-gpu-version>`__

- `GPU Tutorial <./GPU-Tutorial.rst>`__

Applications and Metrics
------------------------

LightGBM supports the following applications:

-  regression, the objective function is L2 loss

-  binary classification, the objective function is logloss

-  multi classification

-  cross-entropy, the objective function is logloss and supports training on non-binary labels

-  LambdaRank, the objective function is LambdaRank with NDCG

LightGBM supports the following metrics:

-  L1 loss

-  L2 loss

-  Log loss

-  Classification error rate

-  AUC

-  NDCG

-  MAP

-  Multi-class log loss

-  Multi-class error rate

-  AUC-mu ``(new in v3.0.0)``

-  Average precision ``(new in v3.1.0)``

-  Fair

-  Huber

-  Poisson

-  Quantile

-  MAPE

-  Kullback-Leibler

-  Gamma

-  Tweedie

For more details, please refer to `Parameters <./Parameters.rst#metric-parameters>`__.

Other Features
--------------

-  Limit ``max_depth`` of tree while grows tree leaf-wise

-  `DART <https://arxiv.org/abs/1505.01866>`__

-  L1/L2 regularization

-  Bagging

-  Column (feature) sub-sample

-  Continued train with input GBDT model

-  Continued train with the input score file

-  Weighted training

-  Validation metric output during training

-  Multiple validation data

-  Multiple metrics

-  Early stopping (both training and prediction)

-  Prediction for leaf index

For more details, please refer to `Parameters <./Parameters.rst>`__.

References
----------

[1] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu. "`LightGBM\: A Highly Efficient Gradient Boosting Decision Tree`_." Advances in Neural Information Processing Systems 30 (NIPS 2017), pp. 3149-3157.

[2] Mehta, Manish, Rakesh Agrawal, and Jorma Rissanen. "SLIQ: A fast scalable classifier for data mining." International Conference on Extending Database Technology. Springer Berlin Heidelberg, 1996.

[3] Shafer, John, Rakesh Agrawal, and Manish Mehta. "SPRINT: A scalable parallel classifier for data mining." Proc. 1996 Int. Conf. Very Large Data Bases. 1996.

[4] Ranka, Sanjay, and V. Singh. "CLOUDS: A decision tree classifier for large datasets." Proceedings of the 4th Knowledge Discovery and Data Mining Conference. 1998.

[5] Machado, F. P. "Communication and memory efficient parallel decision tree construction." (2003).

[6] Li, Ping, Qiang Wu, and Christopher J. Burges. "Mcrank: Learning to rank using multiple classification and gradient boosting." Advances in Neural Information Processing Systems 20 (NIPS 2007).

[7] Shi, Haijian. "Best-first decision tree learning." Diss. The University of Waikato, 2007.

[8] Walter D. Fisher. "`On Grouping for Maximum Homogeneity`_." Journal of the American Statistical Association. Vol. 53, No. 284 (Dec., 1958), pp. 789-798.

[9] Thakur, Rajeev, Rolf Rabenseifner, and William Gropp. "`Optimization of collective communication operations in MPICH`_." International Journal of High Performance Computing Applications 19.1 (2005), pp. 49-66.

[10] Qi Meng, Guolin Ke, Taifeng Wang, Wei Chen, Qiwei Ye, Zhi-Ming Ma, Tie-Yan Liu. "`A Communication-Efficient Parallel Algorithm for Decision Tree`_." Advances in Neural Information Processing Systems 29 (NIPS 2016), pp. 1279-1287.

[11] Huan Zhang, Si Si and Cho-Jui Hsieh. "`GPU Acceleration for Large-scale Tree Boosting`_." SysML Conference, 2018.

.. _LightGBM\: A Highly Efficient Gradient Boosting Decision Tree: https://proceedings.neurips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html

.. _On Grouping for Maximum Homogeneity: https://www.jstor.org/stable/2281952

.. _Optimization of collective communication operations in MPICH: https://www.mpich.org/2012/10/24/optimization-of-collective-communication-operations-in-mpich/

.. _A Communication-Efficient Parallel Algorithm for Decision Tree: https://proceedings.neurips.cc/paper/2016/hash/10a5ab2db37feedfdeaab192ead4ac0e-Abstract.html

.. _GPU Acceleration for Large-scale Tree Boosting: https://arxiv.org/abs/1706.08359

---

## File: docs/gcc-Tips.rst

The content of this document was very outdated and is no longer available to avoid any misleadings.