# Technical Documentation: lyuwenyu/docs

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/lyuwenyu/docs)

### Overview (README.md)

# Fluid Documentation Skeleton

## Build

To build documentation, you need have a linux machine and have python2, virtualenv, gmake installed.

### Preparation

You need to create a `virtualenv` instead of polute the global python library path

```bash
virtualenv .env
```

You can enter virtualenv by

```bash
source .env/bin/activate
```

You can exit virtualenv by

```bash
deactivate
```

### Install dependencies

```bash
# enter virtualenv
source .env/bin/activate
# install dependencies
pip install -r requirements.txt
```

### Make HTML

```bash
# make clean  # make clean to regenerate toctree. Just `make html` may have a cache.
make html
```
and the html files will be generated to `build/html`. You can open `build/html/index.html` with your browser to see the documentation.

## Edit

### Edit documentation

It is suggested to use `reStructuredText` because it is the only official markup language supportted by our documentation generating system, sphinx. `markdown` can also be used. However, since the `markdown` has so many dialects, there is no guarantee that the `markdown` source file can be rendered well.

The `reStructuredText` cheatsheet is [here](http://docutils.sourceforge.net/docs/user/rst/quickref.html).


### Edit structure

The `sphinx` (our documentation generating system) uses `toctree` to organize documentation. `toctree` means `table of content tree`. 

Please see the [sphinx documentation](http://www.sphinx-doc.org/en/master/), especially [`toctree` directives](http://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html)

---

### Design Memory README (`docs/design/memory/README.md`)

# Region-based Heterogeneous Memory Management
## Design

### Usage

To allocate 4KB CPU memory:

```cpp
p = memory::Alloc(platform::CPUPlace(), 4*1024);
```

To allocate 4KB memory on the 3rd GPU:

```cpp
p = memory::Alloc(platform::CUDAPlace(2), 4*1024);
```

To free memory and check the so-far used amount of memory on a place:

```cpp
auto pl = platform::CUDAPlace(0);
p = memory::Alloc(pl, 4*1024);
cout << memory::Used(pl);
memory::Free(pl, p);
```

### API

In `paddle/memory/memory.h` we have:

```cpp
namespace memory {
template <typename Place> void* Alloc(Place, size_t);
template <typename Place> void Free(Place, void*);
template <typename Place> size_t Used(Place);
}  // namespace memory
```

These function templates have specializations on either `platform::CPUPlace` or `platform::CUDAPlace`:

```cpp
template<>
void* Alloc<CPUPlace>(CPUPlace p, size_t size) {
  return GetCPUBuddyAllocator()->Alloc(size);
}
```

and

```cpp
template<>
void Alloc<CUDAPlace>(CUDAPlace p, size_t size) {
  return GetGPUBuddyAllocator(p.id)->Alloc(size);
}
```

Similar specializations exist for `Free` and `Used`.

### Implementation

`GetCPUBuddyAllocator` and `GetGPUBuddyAllocator` are singletions.

```cpp
BuddyAllocator* GetCPUBuddyAllocator() {
  static BuddyAllocator* a = NULL;
  if (a == NULL) {
    a = new BuddyAllocator(new CPUAllocator /*backup allocator*/, ...);
  }
  return a;
}

BuddyAllocator* GetGPUBuddyAllocator(int gpu_id) {
  static BuddyAllocator* as = NULL;
  if (as == NULL) {
    as = new BuddyAllocator*[platform::NumGPUs()];
    for (int gpu = 0; gpu < platform::NumGPUs(); gpu++) {
      as[gpu] = new BuddyAllocator(new GPUAllocator(gpu) /* backup allocator */, ...);
    }
  }
  return as[gpu_id);
```

#### `BuddyAllocator`

`BuddyAllocator` implements the buddy allocation algorithm.  Its constructor takes parameters only related with the algorithm:

```cpp
BuddyAllocator::BuddyAllocator(initial_pool_size, max_pool_size) {
  ...
}
```

Please be aware that **`BuddyAllocator` always allocate aligned memory**, aligned on 32-bytes, which can hold a `BuddyAllocator::Block` object:

```cpp
class BuddyAllocator {
 private:
  struct Block {
    size_t size;
    Block* left, right;
    size_t index; // allocator id
  };
  ...
};
```

Because BuddyAllocator has the meta-data of each block, it can trace the used memory -- record the amount returned by `Alloc` freed in `Free`.  Instead, `CPUAllocator` and `GPUAllocator` doesn't know the size of freed memory block and cannot do the trace.

#### System Allocators

The `GPUAllocator` and `CPUAllocator` are calls *system allocators*.  They work as the fallback allocators of `BuddyAllocator`.

## Justification

I got inspiration from Majel and Caffe2, though above design look different from both.

### Caffe2

In Caffe2, `Tensor<Context>::mutable_data()` allocates the memroy.  In particular, [`Tensor<Context>::mutable_data`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/tensor.h#L523) calls [`Tensor<Context>::raw_mutable_data`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/tensor.h#L459), which in turn calls [`Context::New`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/tensor.h#L479).

There are two implementations of `Context`:

1. [`CPUContext`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/context.h#L105), whose [`New` method](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/context.h#L131) calls [`g_cpu_allocator.get()->New(size_t)`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/context.cc#L15) to allocate the memory.

1. [`CUDAContext`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/context_gpu.h#L99), which has a data member [`int gpu_id_`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/context_gpu.h#L202).  This looks very similar to class `majel::CUDAPlace`, who also has an `int id_` data member.   `CUDAContext::New(size_t)` calls [`g_cub_allocator->DeviceAllocate(&ptr, nbytes)`](https://github.com/caffe2/caffe2/blob/v0.7.0/caffe2/core/context_gpu.cu#L355) to allocate the memory.

### Majel

In Majel, there are basically two allocator types:

1. `cpu::SystemAllocator`, which has similar functionality to `caffe2::CPUContext::New/Delete`.
1. `gpu::SystemAllocator`, which has similar functionality to `caffe2::CUDAContext::New/Delete`.

However, memory allocation is not via these two allocators.  Instead, these two allocators are defined in hidden namespaces.

In Majel there are hidden global variables like:

1. `cpu::SystemAllocator g_cpu_allocator`, and
1. `vector<gpu::SystemAllocator*> g_gpu_allocators(NUM_GPUS)`.

Programs allocate memory via a BuddyAllocator, which can take the `g_cpu_allocator` or a `g_gpu_allocators[gpu_id]` as its *fallback allocator*, so that if BuddyAllocator cannot find a block in its memory pool, it extends its memory pool by calling the fallback allocator's `New(size_t)`.

---

### User Guides Rec Case Recommender System README (`docs/user_guides/rec_case/recommender_system/README.md`)

# Recommender System

The source code of this tutorial is in [book/recommender_system](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system). For new users, please refer to [Running This Book](https://github.com/PaddlePaddle/book/blob/develop/README.md#running-the-book) .

## Background Introduction

With the continuous development of network technology and the ever-expanding scale of e-commerce, the number and variety of goods grow rapidly and users need to spend a lot of time to find the goods they want to buy. This is information overload. In order to solve this problem, recommendation system came into being.

The recommendation system is a subset of the Information Filtering System, which can be used in a range of areas such as movies, music, e-commerce, and Feed stream recommendations. The recommendation system discovers the user's personalized needs and interests by analyzing and mining user behaviors, and recommends information or products that may be of interest to the user. Unlike search engines, recommendation system do not require users to accurately describe their needs, but model their historical behavior to proactively provide information that meets user interests and needs.

The GroupLens system \[[1](#references)\] introduced by the University of Minnesota in 1994 is generally considered to be a relatively independent research direction for the recommendation system. The system first proposed the idea of completing recommendation task based on collaborative filtering. After that, the collaborative filtering recommendation based on the model led the development of recommendation system for more than ten years.

The traditional personalized recommendation system methods mainly include:

- Collaborative Filtering Recommendation: This method is one of the most widely used technologies which requires the collection and analysis of users' historical behaviors, activities and preferences. It can usually be divided into two sub-categories: User-Based Recommendation \[[1](#references)\] and Item-Based Recommendation \[[2](#references)\]. A key advantage of this method is that it does not rely on the machine to analyze the content characteristics of the item, so it does not need to understand the item itself to accurately recommend complex items such as movies. However, the disadvantage is that there is a cold start problem for new users without any behavior. At the same time, there is also a sparsity problem caused by insufficient interaction data between users and commodities. It is worth mentioning that social network \[[3](#references)\] or geographic location and other context information can be integrated into collaborative filtering.
- Content-Based Filtering Recommendation \[[4](#references)\] : This method uses the content description of the product to abstract meaningful features by calculating the similarity between the user's interest and the product description to make recommendations to users. The advantage is that it is simple and straightforward. It does not need to evaluate products based on the comments of users. Instead, it compares the product similarity by product attributes to recommend similar products to the users of interest. The disadvantage is that there is also a cold start problem for new users without any behavior.
- Hybrid Recommendation \[[5](#references)\]: Use different inputs and techniques to jointly recommend items to complement each single recommendation technique.

In recent years, deep learning has achieved great success in many fields. Both academia and industry are trying to apply deep learning to the field of recommendation systems. Deep learning has excellent ability to automatically extract features, can learn multi-level abstract feature representations, and learn heterogeneous or cross-domain content information, which can deal with the cold start problem \[[6](#references)\] of recommendation system to some extent. This tutorial focuses on the deep learning model of recommendation system and how to implement the model with PaddlePaddle.

## Result Demo

We use a dataset containing user information, movie information, and movie ratings as a recommendation system. When we train the model, we only need to input the corresponding user ID and movie ID, we can get a matching score (range [0, 5], the higher the score is regarded as the greater interest), and then according to the recommendation of all movies sort the scores and recommend them to movies that may be of interest to the user.

```
Input movie_id: 1962
Input user_id: 1
Prediction Score is 4.25
```

## Model Overview

In this chapter, we first introduce YouTube's video personalization recommendation system \[[7](#references)\], and then introduce the fusion recommendation model we implemented.

### YouTube's Deep Neural Network Personalized Recommendation System

YouTube is the world's largest video uploading, sharing and discovery site, and the YouTube Personalized Recommendation System recommends personalized content from a growing library to more than 1 billion users. The entire system consists of two neural networks: a candidate generation network and a ranking network. The candidate generation network generates hundreds of candidates from a million-level video library, and the ranking network sorts the candidates and outputs the highest ranked tens of results. The system structure is shown in Figure 1:

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/YouTube_Overview.png?raw=true" width="70%" ><br/>
Figure 1. YouTube personalized recommendation system structure
</p>

#### Candidate Generation Network

The candidate generation network models the recommendation problem as a multi-class classification problem with a large number of categories. For a Youtube user, using its watching history (video ID), search tokens, demographic information (such as geographic location, user login device), binary features (such as gender, whether to log in), and continuous features (such as user age), etc., multi-classify all videos in the video library to obtain the classification result of each category (ie, the recommendation probability of each video), eventually outputting hundreds of videos with high probability.

First, the historical information such as watching history and search token records are mapped to vectors and averaged to obtain a fixed length representation. At the same time, demographic characteristics are input to optimize the recommendation effect of new users, and the binary features and continuous features are normalized to the range [0, 1]. Next, put all the feature representations into a vector and input them to the non-linear multilayer perceptron (MLP, see [Identification Figures](https://github.com/PaddlePaddle/book/blob/develop/02.recognize_digits/README.md) tutorial). Finally, during training, the output of the MLP is classified by softmax. When predicting, the similarity of the user's comprehensive features (MLP output) to all videos' features is calculated, and the highest score of $k$ is obtained as the result of the candidate generation network. Figure 2 shows the candidate generation network structure.

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/Deep_candidate_generation_model_architecture.png?raw=true" width="70%" ><br/>
Figure 2. Candidate generation network structure
</p>

For a user $U$, the formula for predicting whether the video $\omega$ that the user wants to watch at the moment is video $i$ is:

$$P(\omega=i|u)=\frac{e^{v_{i}u}}{\sum_{j \in V}e^{v_{j}u}}$$

Where $u$ is the feature representation of the user $U$, $V$ is the video library collection, and $v_i$ is the feature representation of the $i$ video in the video library. $u$ and $v_i$ are vectors of equal length, and the dot product can be implemented by a fully connected layer.

Considering that the number of categories in the softmax classification is very large, in order to ensure a certain computational efficiency: 1) in the training phase, use negative sample category sampling to reduce the number of actually calculated categories to thousands; 2) in the recommendation (prediction) phase, ignore the normalized calculation of softmax (does not affect the result), and simplifies the category scoring problem into the nearest neighbor search problem in the dot product space, then takes the nearest $k$ video of $u$ as a candidate for generation.

#### Ranking Network
The structure of the ranking network is similar to the candidate generation network, but its goal is to perform finer ranking of the candidates. Similar to the feature extraction method in traditional advertisement ranking, a large number of related features (such as video ID, last watching time, etc.) for video sorting are also constructed here. These features are treated similarly to the candidate generation network, except that at the top of the ranking network is a weighted logistic regression that scores all candidate videos and sorts them from high to low. Then, return to the user.

### Fusion recommendation model
This section uses Convolutional Neural Networks to learn the representation of movie titles. The convolutional neural network for text and the fusion recommendation model are introduced in turn.

#### Convolutional Neural Network (CNN) for text

Convolutional neural networks are often used to deal with data of a grid-like topology. For example, an image can be viewed as a pixel of a two-dimensional grid, and a natural language can be viewed as a one-dimensional sequence of words. Convolutional neural networks can extract a variety of local features and combine them to obtain more advanced feature representations. Experiments show that convolutional neural networks can efficiently model image and text problems.

The convolutional neural network is mainly composed of convolution and pooling operations, and its application and combination methods are flexible and varied. In this section we will explain the network as shown in Figure 3:

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/text_cnn.png?raw=true" width = "80%" align="center"/><br />
Figure 3. Convolutional neural network text classification model
</p>

Suppose the length of the sentence to be processed is $n$, where the word vector of the $i$ word is $x_i\in\mathbb{R}^k$, and $k$ is the dimension size.

First, splicing the word vector: splicing each $h$ word to form a word window of size $h$, denoted as $x_{i:i+h-1}$, which represents the word sequence splicing of $x_{i}, x_{i+1}, \ldots, x_{i+h-1}$, where $i$ represents the position of the first word in the word window throughout the sentence, ranging from $1$ to $n-h+1$, $x_{i:i+h-1}\in\mathbb{R}^{hk}$.

Second, perform a convolution operation: apply the convolution kernel $w\in\mathbb{R}^{hk}$ to the window $x_{i:i+h-1}$ containing $h$ words. , get the feature $c_i=f(w\cdot x_{i:i+h-1}+b)$, where $b\in\mathbb{R}$ is the bias and $f$ is the non Linear activation function, such as $sigmoid$. Apply the convolution kernel to all word windows ${x_{1:h}, x_{2:h+1},\ldots,x_{n-h+1:n}}$ in the sentence, producing a feature map:

$$c=[c_1,c_2,\ldots,c_{n-h+1}], c \in \mathbb{R}^{n-h+1}$$

Next, using the max pooling over time for feature maps to obtain the feature $\hat c$, of the whole sentence corresponding to this convolution kernel, which is the maximum value of all elements in the feature map:

$$\hat c=max(c)$$

#### Fusion recommendation model overview

In the film personalized recommendation system that incorporates the recommendation model:

1. First, take user features and movie features as input to the neural network, where:

   - The user features incorporate four attribute information: user ID, gender, occupation, and age.

   - The movie feature incorporate three attribute information: movie ID, movie type ID, and movie name.

2. For the user feature, map the user ID to a vector representation with a dimension size of 256, enter the fully connected layer, and do similar processing for the other three attributes. Then the feature representations of the four attributes are fully connected and added separately.

3. For movie features, the movie ID is processed in a manner similar to the user ID. The movie type ID is directly input into the fully connected layer in the form of a vector, and the movie name is represented by a fixed-length vector using a text convolutional neural network. The feature representations of the three attributes are then fully connected and added separately.

4. After obtaining the vector representation of the user and the movie, calculate the cosine similarity of them as the score of the personalized recommendation system. Finally, the square of the difference between the similarity score and the user's true score is used as the loss function of the regression model.

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/rec_regression_network.png?raw=true" width="90%" ><br/>
Figure 4. Fusion recommendation model
</p>

## Data Preparation

### Data Introduction and Download

We take [MovieLens Million Dataset (ml-1m)](http://files.grouplens.org/datasets/movielens/ml-1m.zip) as an example. The ml-1m dataset contains 1,000,000 reviews of 4,000 movies by 6,000 users (scores ranging from 1 to 5, all integer), collected by the GroupLens Research lab.

Paddle provides modules for automatically loading data in the API. The data module is `paddle.dataset.movielens`


```python
from __future__ import print_function
import paddle
movie_info = paddle.dataset.movielens.movie_info()
print(list(movie_info.values())[0])
```


```python
# Run this block to show dataset's documentation
# help(paddle.dataset.movielens)
```

The original data includes feature data of the movie, user's feature data, and the user's rating of the movie.

For example, one of the movie features is:


```python
movie_info = paddle.dataset.movielens.movie_info()
print(list(movie_info.values())[0])
```

    <MovieInfo id(1), title(Toy Story ), categories(['Animation', "Children's", 'Comedy'])>


This means that the movie id is 1, and the title is 《Toy Story》, which is divided into three categories. These three categories are animation, children, and comedy.


```python
user_info = paddle.dataset.movielens.user_info()
print(list(user_info.values())[0])
```

    <UserInfo id(1), gender(F), age(1), job(10)>


This means that the user ID is 1, female, and younger than 18 years old. The occupation ID is 10.


Among them, the age uses the following distribution

*  1:  "Under 18"
* 18:  "18-24"
* 25:  "25-34"
* 35:  "35-44"
* 45:  "45-49"
* 50:  "50-55"
* 56:  "56+"

The occupation is selected from the following options:

*  0:  "other" or not specified
*  1:  "academic/educator"
*  2:  "artist"
*  3:  "clerical/admin"
*  4:  "college/grad student"
*  5:  "customer service"
*  6:  "doctor/health care"
*  7:  "executive/managerial"
*  8:  "farmer"
*  9:  "homemaker"
* 10:  "K-12 student"
* 11:  "lawyer"
* 12:  "programmer"
* 13:  "retired"
* 14:  "sales/marketing"
* 15:  "scientist"
* 16:  "self-employed"
* 17:  "technician/engineer"
* 18:  "tradesman/craftsman"
* 19:  "unemployed"
* 20:  "writer"

For each training or test data, it is <user features> + <movie feature> + rating.

For example, we get the first training data:


```python
train_set_creator = paddle.dataset.movielens.train()
train_sample = next(train_set_creator())
uid = train_sample[0]
mov_id = train_sample[len(user_info[uid].value())]
print("User %s rates Movie %s with Score %s"%(user_info[uid], movie_info[mov_id], train_sample[-1]))
```

    User <UserInfo id(1), gender(F), age(1), job(10)> rates Movie <MovieInfo id(1193), title(One Flew Over the Cuckoo's Nest ), categories(['Drama'])> with Score [5.0]


That is, the user 1 evaluates the movie 1193 as 5 points.

## Configuration Instruction

Below we begin to configure the model based on the form of the input data. First import the required library functions and define global variables.

- IS_SPARSE: whether to use sparse update in embedding
- PASS_NUM: number of epoch


```python
import math
import sys
import numpy as np
import paddle
import paddle.fluid as fluid
import paddle.fluid.layers as layers
import paddle.fluid.nets as nets

IS_SPARSE = True
BATCH_SIZE = 256
PASS_NUM = 20
```

Then define the model configuration for our user feature synthesis model

```
/* Detailed source-code truncated for AI context efficiency. */
```

As shown in the code above, for each user, we enter a 4-dimensional feature. This includes user_id, gender_id, age_id, job_id. These dimensional features are simple integer values. In order to facilitate the subsequent neural network processing of these features, we use the language model in NLP to transform these discrete integer values ​​into embedding. And form them into usr_emb, usr_gender_emb, usr_age_emb, usr_job_emb, respectively.

Then, we enter all the user features into a fully connected layer(fc). Combine all features into one 200-dimension feature.

Furthermore, we make a similar transformation for each movie feature, the network configuration is:


```
/* Detailed source-code truncated for AI context efficiency. */
```


The title of a movie is a sequence of integers, and the integer represents the subscript of the word in the index sequence. This sequence is sent to the `sequence_conv_pool` layer, which uses convolution and pooling on the time dimension. Because of this, the output will be fixed length, although the length of the input sequence will vary.

Finally, we define an `inference_program` to calculate the similarity between user features and movie features using cosine similarity.

```python
def inference_program():
    """the combined network"""

    usr_combined_features = get_usr_combined_features()
    mov_combined_features = get_mov_combined_features()

    inference = layers.cos_sim(X=usr_combined_features, Y=mov_combined_features)
    scale_infer = layers.scale(x=inference, scale=5.0)

    return scale_infer
```

Furthermore, we define a `train_program` to use the result computed by `inference_program`, and calculate the error with the help of the tag data. We also define an `optimizer_func` to define the optimizer.

```python
def train_program():
    """define the cost function"""

    scale_infer = inference_program()

    label = fluid.data(name='score', shape=[None, 1], dtype='float32')
    square_cost = layers.square_error_cost(input=scale_infer, label=label)
    avg_cost = layers.mean(square_cost)

    return [avg_cost, scale_infer]


def optimizer_func():
    return fluid.optimizer.SGD(learning_rate=0.2)
```


## Training Model

### Defining the training environment
Define your training environment and specify whether the training takes place on CPU or GPU.

```python
use_cuda = False
place = fluid.CUDAPlace(0) if use_cuda else fluid.CPUPlace()
```

### Defining the data provider
The next step is to define a data provider for training and testing. The provider reads in a data of size `BATCH_SIZE`. `paddle.dataset.movielens.train` will provide a data of size `BATCH_SIZE` after each scribbling, and the size of the out-of-order is the cache size `buf_size`.

```python
train_reader = fluid.io.batch(
    fluid.io.shuffle(
        paddle.dataset.movielens.train(), buf_size=8192),
    batch_size=BATCH_SIZE)

test_reader = fluid.io.batch(
    paddle.dataset.movielens.test(), batch_size=BATCH_SIZE)
```

### Constructing a training process (trainer)
We have constructed a training process here, including training optimization functions.

### Provide data

`feed_order` is used to define the mapping between each generated data and `paddle.layer.data`. For example, the data in the first column generated by `movielens.train` corresponds to the feature `user_id`.

```python
feed_order = [
    'user_id', 'gender_id', 'age_id', 'job_id', 'movie_id', 'category_id',
    'movie_title', 'score'
]
```

### Building training programs and testing programs
The training program and the test program are separately constructed, and the training optimizer is imported.

```python
main_program = fluid.default_main_program()
star_program = fluid.default_startup_program()
[avg_cost, scale_infer] = train_program()

test_program = main_program.clone(for_test=True)
sgd_optimizer = optimizer_func()
sgd_optimizer.minimize(avg_cost)
exe = fluid.Executor(place)

def train_test(program, reader):
    count = 0
    feed_var_list = [
        program.global_block().var(var_name) for var_name in feed_order
    ]
    feeder_test = fluid.DataFeeder(
    feed_list=feed_var_list, place=place)
    test_exe = fluid.Executor(place)
    accumulated = 0
    for test_data in reader():
        avg_cost_np = test_exe.run(program=program,
                                               feed=feeder_test.feed(test_data),
                                               fetch_list=[avg_cost])
        accumulated += avg_cost_np[0]
        count += 1
    return accumulated / count
```

### Build a training main loop and start training
We perform the training cycle according to the training cycle number (`PASS_NUM`) defined above and some other parameters, and perform a test every time. When the test result is good enough, we exit the training and save the trained parameters.

```
/* Detailed source-code truncated for AI context efficiency. */
```
Start training
```python
train_loop()
```

## Model Application

### Generate test data
Use the API of create_lod_tensor(data, lod, place) to generate the tensor of the detail level. `data` is a sequence, and each element is a sequence of index numbers. `lod` is the detail level's information, corresponding to `data`. For example, data = [[10, 2, 3], [2, 3]] means that it contains two sequences of lengths 3 and 2. Correspondingly lod = [[3, 2]], which indicates that it contains a layer of detail information, meaning that `data` has two sequences, lengths of 3 and 2.

In this prediction example, we try to predict the score given by user with ID1 for the movie 'Hunchback of Notre Dame'.

```python
infer_movie_id = 783
infer_movie_name = paddle.dataset.movielens.movie_info()[infer_movie_id].title
user_id = np.array([1]).astype("int64").reshape(-1)
gender_id = np.array([1]).astype("int64").reshape(-1)
age_id = np.array([0]).astype("int64").reshape(-1)
job_id = np.array([10]).astype("int64").reshape(-1)
movie_id = np.array([783]).astype("int64").reshape(-1) # Hunchback of Notre Dame
category_id = fluid.create_lod_tensor(np.array([10, 8, 9], dtype='int64'), [[3]], place) # Animation, Children's, Musical
movie_title = fluid.create_lod_tensor(np.array([1069, 4140, 2923, 710, 988], dtype='int64'), [[5]],
                                      place) # 'hunchback','of','notre','dame','the'
```

### Building the prediction process and testing
Similar to the training process, we need to build a prediction process, where `params_dirname` is the address used to store the various parameters in the training process.

```python
place = fluid.CUDAPlace(0) if use_cuda else fluid.CPUPlace()
exe = fluid.Executor(place)

inference_scope = fluid.core.Scope()
```

### Testing
Now we can make predictions. The `feed_order` we provide should be consistent with the training process.


```python
with fluid.scope_guard(inference_scope):
    [inferencer, feed_target_names,
    fetch_targets] = fluid.io.load_inference_model(params_dirname, exe)

    results = exe.run(inferencer,
                          feed={
                               'user_id': user_id,
                              'gender_id': gender_id,
                              'age_id': age_id,
                              'job_id': job_id,
                              'movie_id': movie_id,
                              'category_id': category_id,
                              'movie_title': movie_title
                          },
                          fetch_list=fetch_targets,
                          return_numpy=False)
    predict_rating = np.array(results[0])
    print("Predict Rating of user id 1 on movie \"" + infer_movie_name +
              "\" is " + str(predict_rating[0][0]))
    print("Actual Rating of user id 1 on movie \"" + infer_movie_name +
              "\" is 4.")
```

## Summary

This chapter introduced the traditional personalized recommendation system method and YouTube's deep neural network personalized recommendation system. It further took movie recommendation as an example, and used PaddlePaddle to train a personalized recommendation neural network model. The personalized recommendation system covers almost all aspects of e-commerce systems, social networks, advertising recommendations, search engines, etc. Deep learning technologies have played an important role in image processing, natural language processing, etc., and will also prevail in personalized recommendation systems.

<a name="references"></a>
## References

1. P. Resnick, N. Iacovou, etc. “[GroupLens: An Open Architecture for Collaborative Filtering of Netnews](http://ccs.mit.edu/papers/CCSWP165.html)”, Proceedings of ACM Conference on Computer Supported Cooperative Work, CSCW 1994. pp.175-186.
2. Sarwar, Badrul, et al. "[Item-based collaborative filtering recommendation algorithms.](http://files.grouplens.org/papers/www10_sarwar.pdf)" *Proceedings of the 10th international conference on World Wide Web*. ACM, 2001.
3. Kautz, Henry, Bart Selman, and Mehul Shah. "[Referral Web: combining social networks and collaborative filtering.](http://www.cs.cornell.edu/selman/papers/pdf/97.cacm.refweb.pdf)" Communications of the ACM 40.3 (1997): 63-65. APA
4. [Peter Brusilovsky](https://en.wikipedia.org/wiki/Peter_Brusilovsky) (2007). *The Adaptive Web*. p. 325.
5. Robin Burke , [Hybrid Web recommendation systems](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.435.7538&rep=rep1&type=pdf), pp. 377-408, The Adaptive Web, Peter Brusilovsky, Alfred Kobsa, Wolfgang Nejdl (Ed.), Lecture Notes in Computer Science, Springer-Verlag, Berlin, Germany, Lecture Notes in Computer Science, Vol. 4321, May 2007, 978-3-540-72078-2.
6. Yuan, Jianbo, et al. ["Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach."](https://arxiv.org/pdf/1611.05480v1.pdf) *arXiv preprint arXiv:1611.05480* (2016).
7. Covington P, Adams J, Sargin E. [Deep neural networks for youtube recommendations](https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/45530.pdf)[C]//Proceedings of the 10th ACM Conference on recommendation systems. ACM, 2016: 191-198.


<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://paddlepaddleimage.cdn.bcebos.com/bookimage/camo.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> is contributed by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

---

### User Guides Simple Case Fit A Line README (`docs/user_guides/simple_case/fit_a_line/README.md`)

# Linear Regression

Let's start this tutorial from the classic Linear Regression ([[1](#References)]) model.

In this chapter, you will build a model to predict house price with real datasets and learn about several important concepts about machine learning.

The source code of this tutorial is in [book/fit_a_line](https://github.com/PaddlePaddle/book/tree/develop/01.fit_a_line). For the new users, please refer to [Running This Book](https://github.com/PaddlePaddle/book/blob/develop/README.md#running-the-book) .



## Background
Given a $n$ dataset ${\{y_{i}, x_{i1}, ..., x_{id}\}}_{i=1}^{n}$, of which $ x_{i1}, \ldots, x_{id}$ are the values of the $d$th attribute of $i$ sample, and $y_i$ is the target to be predicted for this sample.

 The linear regression model assumes that the target $y_i$ can be described by a linear combination among attributes, i.e.

$$y_i = \omega_1x_{i1} + \omega_2x_{i2} + \ldots + \omega_dx_{id} + b,  i=1,\ldots,n$$

For example, in the problem of prediction of house price we are going to explore, $x_{ij}$ is a description of the various attributes of the house $i$ (such as the number of rooms, the number of schools and hospitals around, traffic conditions, etc.). $y_i$ is the price of the house.



At first glance, this assumption is too simple, and the true relationship among variables is unlikely to be linear. However, because the linear regression model has the advantages of simple form and easy to be modeled and analyzed, it has been widely applied in practical problems. Many classic statistical learning and machine learning books \[[2,3,4](#references)\] also focus on linear model in a chapter.

##  Result Demo
We used the Boston house price dataset obtained from [UCI Housing dataset](http://paddlemodels.bj.bcebos.com/uci_housing/housing.data) to train and predict the model. The scatter plot below shows the result of price prediction for parts of house with model. Each point on x-axis represents the median of the real price of the same type of house, and the y-axis represents the result of the linear regression model based on the feature prediction. When the two values are completely equal, they will fall on the dotted line. So the more accurate the model is predicted, the closer the point is to the dotted line.
<p align="center">
    <img src = "https://github.com/PaddlePaddle/book/blob/develop/01.fit_a_line/image/predictions.png?raw=true" width=400><br/>
    Figure One. Predict value V.S Ground-truth value
</p>

## Model Overview

### Model Definition

In the dataset of Boston house price, there are 14 values associated with the home: the first 13 are used to describe various information of house, that is $x_i$ in the model; the last value is the medium price of the house we want to predict, which is $y_i$ in the model.

Therefore, our model can be expressed as:

$$\hat{Y} = \omega_1X_{1} + \omega_2X_{2} + \ldots + \omega_{13}X_{13} + b$$

$\hat{Y}$ represents the predicted result of the model and is used to distinguish it from the real value $Y$. The parameters to be learned by the model are: $\omega_1, \ldots, \omega_{13}, b$.

After building the model, we need to give the model an optimization goal so that the learned parameters can make the predicted value $\hat{Y}$ get as close to the true value $Y$. Here we introduce the concept of loss function ([Loss Function](https://en.wikipedia.org/wiki/Loss_function), or Cost Function.  Input the target value $y_{i}$ of any data sample and the predicted value $\hat{y_{i}}$ given by a model. Then the loss function outputs a non-negative real number, which is usually used to represent model error.

For linear regression models, the most common loss function is the Mean Squared Error ([MSE](https://en.wikipedia.org/wiki/Mean_squared_error)), which is:

$$MSE=\frac{1}{n}\sum_{i=1}^{n}{(\hat{Y_i}-Y_i)}^2$$

That is, for a test set in size of $n$, $MSE$ is the mean of the squared error of the $n$ data prediction results.

The method used to optimize the loss function is generally the gradient descent method. The gradient descent method is a first-order optimization algorithm. If $f(x)$ is defined and divisible at point $x_n$, then $f(x)$ is considered to be the fastest in the negative direction of the gradient $-▽f(x_n)$ at point of $x_n$. Adjust $x$ repeatedly to make $f(x)$ close to the local or global minimum value. The adjustment is as follows:

$$x_n+1=x_n-λ▽f(x), n≧0$$

Where λ represents the learning rate. This method of adjustment is called the gradient descent method.

### Training Process

After defining the model structure, we will train the model through the following steps.

  1. Initialize parameters, including weights $\omega_i$ and bias $b$, to initialize them (eg. 0 as mean, 1 as variance).
  2. Forward propagation of network calculates network output and loss functions.
  3. Reverse error propagation according to the loss function ( [backpropagation](https://en.wikipedia.org/wiki/Backpropagation) ), passing forward the network error from the output layer and updating the parameters in the network.
  4. Repeat steps 2~3 until the network training error reaches the specified level or the training round reaches the set value.


## Dataset

### Dataset Introduction
The dataset consists of 506 lines, each containing information about a type of houses in a suburb of Boston and the median price of that type of house. The meaning of each dimensional attribute is as follows:

| Property Name | Explanation | Type |
| ------| ------ | ------ |
CRIM | Per capita crime rate in the town | Continuous value |
| ZN | Proportion of residential land with an area of over 25,000 square feet | Continuous value |
| INDUS | Proportion of non-retail commercial land | Continuous value |
CHAS | Whether it is adjacent to Charles River | Discrete value, 1=proximity; 0=not adjacent |
NOX | Nitric Oxide Concentration | Continuous value |
| RM | Average number of rooms per house | Continuous value |
| AGE | Proportion of self-use units built before 1940 | Continuous value |
| DIS | Weighted Distance to 5 Job Centers in Boston | Continuous value |
| RAD | Accessibility Index to Radial Highway | Continuous value |
| TAX | Tax Rate of Full-value Property | Continuous value |
| PTRATIO | Proportion of Student and Teacher | Continuous value |
| B | 1000(BK - 0.63)^2, where BK is black ratio | Continuous value |
LSTAT | Low-income population ratio | Continuous value |
| MEDV | Median price of a similar home | Continuous value |

### Data Pre-processing

#### Continuous value and discrete value
Analyzing the data, first we find that all 13-dimensional attributes exist 12-dimensional continuous value and 1-dimensional discrete values (CHAS). Discrete value is often represented by numbers like 0, 1, and 2, but its meaning is different from continuous value's because the difference of discrete value here has no meaning. For example, if we use 0, 1, and 2 to represent red, green, and blue, we cannot infer that the distance between blue and red is longer than that between green and red. So usually for a discrete property with $d$ possible values, we will convert them to $d$ binary properties with a value of 0 or 1 or map each possible value to a multidimensional vector. However, there is no this problem for CHAS, since CHAS itself is a binary attribute .

#### Normalization of attributes
Another fact that can be easily found is that the range of values of each dimensional attribute is largely different (as shown in Figure 2). For example, the value range of attribute B is [0.32, 396.90], and the value range of attribute NOX is [0.3850, 0.8170]. Here is a common operation - normalization. The goal of normalization is to scale the value of each attribute to a similar range, such as [-0.5, 0.5]. Here we use a very common operation method: subtract the mean and divide by the range of values.

There are at least three reasons for implementing normalization (or [Feature scaling](https://en.wikipedia.org/wiki/Feature_scaling)):

- A range of values that are too large or too small can cause floating value overflow or underflow during calculation.

- Different ranges of number result in different attributes being different for the model (at least in the initial period of training), and this implicit assumption is often unreasonable. This can make the optimization process difficult and the training time greatly longer.

- Many machine learning techniques/models (such as L1, L2 regular items, Vector Space Model) are based on the assumption that all attribute values are almost zero and their ranges of value are similar.



<p align="center">
    <img src = "https://github.com/PaddlePaddle/book/blob/develop/01.fit_a_line/image/ranges.png?raw=true" width=550><br/>
    Figure 2. Value range of attributes for all dimensions
</p>

#### Organizing training set and testing set

We split the dataset into two parts: one is used to adjust the parameters of the model, that is, to train the model, the error of the model on this dataset is called ** training error **; the other is used to test.The error of the model on this dataset is called the ** test error**. The goal of our training model is to predict unknown new data by finding the regulation from the training data, so the test error is an better indicator for the performance of the model. When it comes to the ratio of the segmentation data, we should take into account two factors: more training data will reduce the square error of estimated parameters, resulting in a more reliable model; and more test data will reduce the square error of the test error, resulting in more credible test error. The split ratio set in our example is $8:2$


In a more complex model training process, we often need more than one dataset: the validation set. Because complex models often have some hyperparameters ([Hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter_optimization)) that need to be adjusted, we will try a combination of multiple hyperparameters to train multiple models separately and then compare their performance on the validation set to select the relatively best set of hyperparameters, and finally use the model with this set of parameters to evaluate the test error on the test set. Since the model trained in this chapter is relatively simple, we won't talk about this process at present.

## Training

`fit_a_line/train.py` demonstrates the overall process of training.

### Configuring the Data feeder

First we import the libraries:

```python
from __future__ import print_function
import paddle
import paddle.fluid as fluid
import numpy
import math
import sys
```

We introduced the dataset [UCI Housing dataset](http://paddlemodels.bj.bcebos.com/uci_housing/housing.data) via the uci_housing module

It is encapsulated in the uci_housing module:

1. The process of data download. The download data is saved in ~/.cache/paddle/dataset/uci_housing/housing.data.
2. The process of [data preprocessing](#data preprocessing).

Next we define the data feeder for training. The data feeder reads a batch of data in the size of `BATCH_SIZE` each time. If the user wants the data to be random, it can define data in size of a batch and a cache. In this case, each time the data feeder randomly reads as same data as the batch size from the cache.

```python
BATCH_SIZE = 20

train_reader = fluid.io.batch(
    fluid.io.shuffle(
        paddle.dataset.uci_housing.train(), buf_size=500),
        batch_size=BATCH_SIZE)

test_reader = fluid.io.batch(
    fluid.io.shuffle(
        paddle.dataset.uci_housing.test(), buf_size=500),
        batch_size=BATCH_SIZE)
```

If you want to read data directly from \*.txt file, you can refer to the method as follows(need to prepare txt file by yourself).
```text
feature_names = [
    'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX',
    'PTRATIO', 'B', 'LSTAT', 'convert'
]
feature_num = len(feature_names)
data = numpy.fromfile(filename, sep=' ') # Read primary data from file
data = data.reshape(data.shape[0] // feature_num, feature_num)
maximums, minimums, avgs = data.max(axis=0), data.min(axis=0), data.sum(axis=0)/data.shape[0]

for i in six.moves.range(feature_num-1):
   data[:, i] = (data[:, i] - avgs[i]) / (maximums[i] - minimums[i]) # six.moves is compatible to python2 and python3

ratio = 0.8 # distribution ratio of train dataset and verification dataset
offset = int(data.shape[0]\*ratio)
train_data = data[:offset]
test_data = data[offset:]

def reader_creator(train_data):  
    def reader():  
        for d in train_data:  
            yield d[:-1], d[-1:]  
    return reader

train_reader = fluid.io.batch(
    fluid.io.shuffle(
        reader_creator(train_data), buf_size=500),
        batch_size=BATCH_SIZE)

test_reader = fluid.io.batch(
    fluid.io.shuffle(
        reader_creator(test_data), buf_size=500),
        batch_size=BATCH_SIZE)
```

### Configure Program for Training
The aim of the program for training is to define a network structure of a training model. For linear regression, it is a simple fully connected layer from input to output. More complex result, such as Convolutional Neural Network and Recurrent Neural Network, will be introduced in later chapters. It must return `mean error` as the first return value in program for training, for that `mean error` will be used for BackPropagation.

```python
x = fluid.data(name='x', shape=[None, 13], dtype='float32') # define shape and data type of input
y = fluid.data(name='y', shape=[None, 1], dtype='float32') # define shape and data type of output
y_predict = fluid.layers.fc(input=x, size=1, act=None) # fully connected layer connecting input and output

main_program = fluid.default_main_program() # get default/global main function
startup_program = fluid.default_startup_program() # get default/global launch program

cost = fluid.layers.square_error_cost(input=y_predict, label=y) # use label and output predicted data to estimate square error
avg_loss = fluid.layers.mean(cost) # compute mean value for square error and get mean loss
```
For details, please refer to:
[fluid.default_main_program](http://www.paddlepaddle.org/documentation/docs/zh/develop/api_cn/fluid_cn.html#default-main-program)
[fluid.default_startup_program](http://www.paddlepaddle.org/documentation/docs/zh/develop/api_cn/fluid_cn.html#default-startup-program)

### Optimizer Function Configuration

`SGD optimizer`, `learning_rate` below are learning rate, which is related to rate of convergence for train of network.

```python
#Clone main_program to get test_program
# operations of some operators are different between train and test. For example, batch_norm use parameter for_test to determine whether the program is for training or for testing.
#The api will not delete any operator, please apply it before backward and optimization.
test_program = main_program.clone(for_test=True)

sgd_optimizer = fluid.optimizer.SGD(learning_rate=0.001)
sgd_optimizer.minimize(avg_loss)

```

### Define Training Place

We can define whether an operation runs on the CPU or on the GPU.

```python
use_cuda = False
place = fluid.CUDAPlace(0) if use_cuda else fluid.CPUPlace() # define the execution space of executor

###executor can accept input program and add data input operator and result fetch operator based on feed map and fetch list. Use close() to close executor and call run(...) to run the program.
exe = fluid.Executor(place)

```
For details, please refer to:
[fluid.executor](http://www.paddlepaddle.org/documentation/docs/zh/develop/api_cn/fluid_cn.html#permalink-15-executor)

### Create Training Process
To train, it needs a train program and some parameters and creates a function to get test error in the process of train necessary parameters contain executor, program, reader, feeder, fetch_list,  executor represents executor created before. Program created before represents program executed by executor. If the parameter is undefined, then it is defined default_main_program by default. Reader represents data read. Feeder represents forward input variable and fetch_list represents variable user wants to get or name.

```python
num_epochs = 100

def train_test(executor, program, reader, feeder, fetch_list):
    accumulated = 1 * [0]
    count = 0
    for data_test in reader():
        outs = executor.run(program=program,
                            feed=feeder.feed(data_test),
                            fetch_list=fetch_list)
        accumulated = [x_c[0] + x_c[1][0] for x_c in zip(accumulated, outs)] # accumulate loss value in the process of test
        count += 1 # accumulate samples in test dataset
    return [x_d / count for x_d in accumulated] # compute mean loss

```

### Train Main Loop

give name of directory to be stored and initialize an executor

```python
%matplotlib inline
params_dirname = "fit_a_line.inference.model"
feeder = fluid.DataFeeder(place=place, feed_list=[x, y])
exe.run(startup_program)
train_prompt = "train cost"
test_prompt = "test cost"
from paddle.utils.plot import Ploter
plot_prompt = Ploter(train_prompt, test_prompt)
step = 0

exe_test = fluid.Executor(place)
```
Paddlepaddle provides reader mechanism to read training data. Reader provide multiple columns of data at one time. Therefore, we need a python list to read sequence. We create a loop to train until the result of train is good enough or time of loop is enough.
If the number of iterations for train is equal to the number of iterations for saving parameters, you can save train parameter into `params_dirname`.
Set main loop for training.
```
/* Detailed source-code truncated for AI context efficiency. */
```

## Predict
It needs to create trained parameters to run program for prediction. The trained parameters is in `params_dirname`.

### Prepare Environment for Prediction
Similar to the process of training, predictor needs a program for prediction. We can slightly modify our training program to include the prediction value.

```python
infer_exe = fluid.Executor(place)
inference_scope = fluid.core.Scope()
```

### Predict

Save pictures
```python
def save_result(points1, points2):
    import matplotlib
    matplotlib.use('Agg')
    import matplotlib.pyplot as plt
    x1 = [idx for idx in range(len(points1))]
    y1 = points1
    y2 = points2
    l1 = plt.plot(x1, y1, 'r--', label='predictions')
    l2 = plt.plot(x1, y2, 'g--', label='GT')
    plt.plot(x1, y1, 'ro-', x1, y2, 'g+-')
    plt.title('predictions VS GT')
    plt.legend()
    plt.savefig('./image/prediction_gt.png')
```

Via fluid.io.load_inference_model, predictor will read well-trained model from `params_dirname` to predict unknown data.

```python
with fluid.scope_guard(inference_scope):
    [inference_program, feed_target_names,
     fetch_targets] = fluid.io.load_inference_model(params_dirname, infer_exe) # load pre-predict model
    batch_size = 10

    infer_reader = fluid.io.batch(
        paddle.dataset.uci_housing.test(), batch_size=batch_size) # prepare test dataset

    infer_data = next(infer_reader())
    infer_feat = numpy.array(
        [data[0] for data in infer_data]).astype("float32") # extract data in test dataset
    infer_label = numpy.array(
        [data[1] for data in infer_data]).astype("float32") # extract label in test dataset

    assert feed_target_names[0] == 'x'
    results = infer_exe.run(inference_program,
                            feed={feed_target_names[0]: numpy.array(infer_feat)},
                            fetch_list=fetch_targets) # predict
    #print predict result and label and visualize the result
    print("infer results: (House Price)")
    for idx, val in enumerate(results[0]):
        print("%d: %.2f" % (idx, val)) # print predict result

    print("\nground truth:")
    for idx, val in enumerate(infer_label):
        print("%d: %.2f" % (idx, val)) # print label

    save_result(results[0], infer_label) # save picture
```



## Summary
In this chapter, we analyzed dataset of Boston House Price to introduce the basic concepts of linear regression model and how to use PaddlePaddle to implement training and testing. A number of models and theories are derived from linear regression model. Therefore, it is not unnecessary to figure out the principle and limitation of linear regression model.

<a name="References"></a>
## References
1. https://en.wikipedia.org/wiki/Linear_regression
2. Friedman J, Hastie T, Tibshirani R. The elements of statistical learning[M]. Springer, Berlin: Springer series in statistics, 2001.
3. Murphy K P. Machine learning: a probabilistic perspective[M]. MIT press, 2012.
4. Bishop C M. Pattern recognition[J]. Machine Learning, 2006, 128.

<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://paddlepaddleimage.cdn.bcebos.com/bookimage/camo.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> is contributed by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

---

### User Guides Simple Case Word2vec README (`docs/user_guides/simple_case/word2vec/README.md`)

# Word Vector

The source code of this tutorial is in [book/word2vec](https://github.com/PaddlePaddle/book/tree/develop/04.word2vec). For new users, please refer to [Running This Book](https://github.com/PaddlePaddle/book/blob/develop/README.md#running-the-book) .

## Background

In this chapter, we'll introduce the vector representation of words, also known as word embedding. Word vector is a common operation in natural language processing. It is a common technology underlying Internet services such as search engines, advertising systems, and recommendation systems.

In these Internet services, we often compare the correlation between two words or two paragraphs of text. In order to make such comparisons, we often have to express words in a way that is suitable for computer processing. The most natural way is probably the vector space model.In this way, each word is represented as a one-hot vector whose length is the dictionary size, and each dimension corresponds to each word in a dictionary, except that the value in the corresponding dimension of the word is 1, other elements are 0.

The One-hot vector is natural but has limitation. For example, in the internet advertising system, if the query entered by the user is "Mother's Day", the keyword of an advertisement is "Carnation". Although according to common sense, we know that there is a connection between these two words - Mother's Day should usually give the mother a bunch of carnations; but the distance between the two words corresponds to the one-hot vectors, whether it is Euclidean distance or cosine similarity, the two words are considered to be irrelevant due to their vector orthogonality. The root cause of this conclusion contradicting us is that the amount of information in each word itself is too small. Therefore, just giving two words is not enough for us to accurately determine whether they are relevant. To accurately calculate correlations, we need more information—knowledge from a large amount of data through machine learning methods.

In the field of machine learning, all kinds of "knowledge" are represented by various models, and the word embedding model is one of them. A one-hot vector can be mapped to a lower-dimensional embedding vector by the word embedding model, such as $embedding (Mother's day) = [0.3, 4.2, -1.5, ...], embedding (carnation) = [0.2, 5.6, -2.3, ...]$. In this representation of the embedding vector to which it is mapped, it is desirable that the word vectors corresponding to the similar words on the two semantics (or usages) are "more like", such that the cosine similarity of the corresponding word vectors of "Mother's Day" and "Carnation" is no longer zero.

The word embedding model can be a probability model, a co-occurrence matrix model, or a neural network model. Before implementing neural networks to calculate the embedding vector, the traditional method is to count the co-occurrence matrix $X$ of a word. $X$ is a matrix of $|V| \times |V|$ size, $X_{ij}$ means that in all corpora, The number of words appearing simultaneously with the i-th word and the j-th word in the vocabulary $V$(vocabulary), $|V|$ is the size of the vocabulary. Do matrix decomposition for $X$ (such as singular value decomposition, Singular Value Decomposition \[[5](#references)\]), and the result $U$ is treated as the embedding vector for all words:

$$X = USV^T$$

But such traditional method has many problems:

1) Since many words do not appear, the matrix is extremely sparse, so additional processing of the word frequency is needed to achieve a good matrix decomposition effect;

2) The matrix is very large and the dimensions are too high (usually up to $10^6 \times 10^6$);

3) You need to manually remove the stop words (such as although, a, ...), otherwise these frequently occurring words will also affect the effect of matrix decomposition.

The neural-network-based model does not need to calculate and store a large table that is statistically generated on the whole corpus, but obtains the word vector by learning the semantic information, so the problem above can be well solved. In this chapter, we will show the details of training word vectors based on neural networks and how to train a word embedding model with PaddlePaddle.


##  Result Demo

In this chapter, after the embedding vector is trained, we can use the data visualization algorithm t-SNE\[[4](#references)\] to draw the projection of the word features in two dimensions (as shown below). As can be seen from the figure, semantically related words (such as a, the, these; big, huge) are very close in projection, and semantic unrelated words (such as say, business; decision, japan) are far away from the projection.

<p align="center">
    <img src = "https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/2d_similarity.png?raw=true" width=400><br/>
    Figure 1. Two-dimensional projection of a word vector
</p>

On the other hand, we know that the cosine of two vectors is in the interval of $[-1,1]$: two identical vector cosines are 1, and the cosine value between two mutually perpendicular vectors is 0, The vector cosine of the opposite direction is -1, which the correlation is proportional to the magnitude of the cosine. So we can also calculate the cosine similarity of two word vectors:

```

please input two words: big huge
Similarity: 0.899180685161

please input two words: from company
Similarity: -0.0997506977351

```

The results above can be obtained by running `calculate_dis.py`, loading the words in the dictionary and the corresponding training feature results. We will describe the usage for details in [model application](#model application).


## Overview of Models

Here we introduce three models of training word vectors: N-gram model, CBOW model and Skip-gram model. Their central idea is to get the probability of a word appearing through the context. For the N-gram model, we will first introduce the concept of the language model. In the section [training model](#training model), we'll tutor you to implement it with PaddlePaddle. The latter two models are the most famous neuron word vector models in recent years, developed by Tomas Mikolov in Google \[[3](#references)\], although they are very simple, but the training effect is very good.

### Language Model

Before introducing the word embedding model, let us introduce a concept: the language model.
The language model is intended to model the joint probability function $P(w_1, ..., w_T)$ of a sentence, where $w_i$ represents the ith word in the sentence. The goal of the language model isn that the model gives a high probability to meaningful sentences and a small probability to meaningless sentences.Such models can be applied to many fields, such as machine translation, speech recognition, information retrieval, part-of-speech tagging, handwriting recognition, etc., All of which hope to obtain the probability of a continuous sequence. Take information retrieval as an example, when you search for "how long is a football bame" (bame is a medical term), the search engine will prompt you if you want to search for "how long is a football game", because the probability of calculating "how long is a football bame" is very low, and the word is similar to bame, which may cause errors, the game will maximize the probability of generating the sentence.

For the target probability of the language model $P(w_1, ..., w_T)$, if it is assumed that each word in the text is independent, the joint probability of the whole sentence can be expressed as the product of the conditional probabilities of all the words. which is:

$$P(w_1, ..., w_T) = \prod_{t=1}^TP(w_t)$$

However, we know that the probability of each word in the statement is closely related to the word in front of it, so in fact, the language model is usually represented by conditional probability:

$$P(w_1, ..., w_T) = \prod_{t=1}^TP(w_t | w_1, ... , w_{t-1})$$



### N-gram neural model

In computational linguistics, n-gram is an important text representation method that represents a continuous n items in a text. Each item can be a letter, word or syllable based on the specific application scenario. The n-gram model is also an important method in the statistical language model. When n-gram is used to train the language model, the nth word is generally predicted by the content of the n-1 words of each n-gram.

Scientists such as Yoshua Bengio introduced how to learn a word vector model of a neural network representation in the famous paper Neural Probabilistic Language Models \[[1](#references)\ in 2003. The Neural Network Language Model (NNLM) in this paper connects the linear model and a nonlinear hidden layer. It learns the language model and the word vector simultaneously, that is, by learning a large number of corpora to obtain the vector expression of the words, and the probability of the entire sentence is obtained by using these vectors. Since all words are represented by a low-dimensional vector, learning the language model in this way can overcome the curse of dimensionality.
Note: Because the "Neural Network Language Model" is more general, we do not use the real name of NNLM here, considering its specific practice, this model here is called N-gram neural model.

We have already mentioned above using the conditional probability language model, that is, the probability of the $t$ word in a sentence is related to the first $t-1$ words of the sentence. The farther the word actually has the smaller effect on the word, then if you consider an n-gram, each word is only affected by the preceding `n-1` words, then:

$$P(w_1, ..., w_T) = \prod_{t=n}^TP(w_t|w_{t-1}, w_{t-2}, ..., w_{t-n+1 })$$

Given some real corpora, these corpora are meaningful sentences, and the optimization goal of the N-gram model is to maximize the objective function:

$$\frac{1}{T}\sum_t f(w_t, w_{t-1}, ..., w_{t-n+1};\theta) + R(\theta)$$

Where $f(w_t, w_{t-1}, ..., w_{t-n+1})$ represents the conditional probability of getting the current word $w_t$ based on historical n-1 words, $R(\theta )$ represents a parameter regularization item.

<p align="center">
       <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/nnlm.png?raw=true" width=500><br/>
       Figure 2. N-gram neural network model
</p>

Figure 2 shows the N-gram neural network model. From the bottom up, the model is divided into the following parts:
 - For each sample, the model enters $w_{t-n+1},...w_{t-1}$, and outputs the probability distribution of the t-th word in the dictionary on the `|V|` words.

   Each input word $w_{t-n+1},...w_{t-1}$ first maps to the word vector $C(w_{t-n+1}),...W_{t-1})$ by the mapping matrix.

 - Then the word vectors of all words are spliced into a large vector, and a hidden layer representation of the historical words is obtained through a non-linear mapping:

    $$g=Utanh(\theta^Tx + b_1) + Wx + b_2$$

Among them, $x$ is a large vector of all words, representing text history features; $\theta$, $U$, $b_1$, $b_2$, and $W$ are respectively parameters for the word vector layer to the hidden layer connection. $g$ represents the probability of all output words that are not normalized, and $g_i$ represents the output probability of the $i$ word in the unnormalized dictionary.

 - According to the definition of softmax, by normalizing $g_i$, the probability of generating the target word $w_t$ is:

  $$P(w_t | w_1, ..., w_{t-n+1}) = \frac{e^{g_{w_t}}}{\sum_i^{|V|} e^{g_i}}$$

 - The loss value of the entire network is the multi-class classification cross entropy, which is expressed as

   $$J(\theta) = -\sum_{i=1}^N\sum_{k=1}^{|V|}y_k^{i}log(softmax(g_k^i))$$

   where $y_k^i$ represents the real label (0 or 1) of the $i$ sample of the $k$ class, and $softmax(g_k^i)$ represents the probability of the kth softmax output of the i-th sample.



### Continuous Bag-of-Words model(CBOW)

The CBOW model predicts the current word through the context of a word (each N words). When N=2, the model is shown below:

<p align="center">
    <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/cbow.png?raw=true" width=250><br/>
    Figure 3. CBOW model
</p>

Specifically, regardless of the contextual word input order, CBOW uses the mean of the word vectors of the context words to predict the current word. which is:

$$context = \frac{x_{t-1} + x_{t-2} + x_{t+1} + x_{t+2}}{4}$$

Where $x_t$ is the word vector of the $t$th word, the score vector (score) $z=U\*context$, the final classification $y$ uses softmax, and the loss function uses multi-class classification cross entropy.

### Skip-gram model

The benefit of CBOW is that the distribution of contextual words is smoothed over the word vector, removing noise.  Therefore it is very effective on small data sets. In the Skip-gram method, a word is used to predict its context, and many samples of the current word context are obtained, so it can be used for a larger data set.

<p align="center">
    <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/skipgram.png?raw=true" width=250><br/>
    Figure 4. Skip-gram model
</p>

As shown in the figure above, the specific method of the Skip-gram model is to map the word vector of a word to the word vector of $2n$ words ($2n$ represents the $n$ words before and after the input word), and then obtained the sum of the classification loss values of the $2n$ words by softmax.


## Data Preparation

### Data Introduction

This tutorial uses the Penn Treebank (PTB) (pre-processed version of Tomas Mikolov) dataset. The PTB data set is small and the training speed is fast. It is applied to Mikolov's open language model training tool \[[2](#references)\]. Its statistics are as follows:

<p align="center">
<table>
    <tr>
        <td>Training data</td>
        <td>Verify data</td>
        <td>Test data</td>
    </tr>
    <tr>
        <td>ptb.train.txt</td>
        <td>ptb.valid.txt</td>
        <td>ptb.test.txt</td>
    </tr>
    <tr>
        <td>42068 sentences</td>
        <td>3370 sentences</td>
        <td>3761 sentence</td>
    </tr>
</table>
</p>


### Data Preprocessing

This chapter trains the 5-gram model, which means that the first 4 words of each piece of data are used to predict the 5th word during PaddlePaddle training. PaddlePaddle provides the python package `paddle.dataset.imikolov` corresponding to the PTB dataset, which automatically downloads and preprocesses the data for your convenience.

Preprocessing adds the start symbol `<s>` and the end symbol `<e>` to each sentence in the data set. Then, depending on the window size (5 in this tutorial), slide the window to the right each time from start to end and generate a piece of data.

For example, "I have a dream that one day" provides 5 pieces of data:

```text
<s> I have a dream
I have a dream that
Have a dream that one
a dream that one day
Dream that one day <e>
```

Finally, based on the position of its word in the dictionary, each input is converted to an index sequence of integers as the input to PaddlePaddle.

<a name="training model"></a>
## Program the Model

The model structure of this configuration is shown below:

<p align="center">
    <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/ngram.png?raw=true" width=400><br/>
    Figure 5. N-gram neural network model in model configuration
</p>

First, load packages:

```python

from __future__ import print_function

import paddle
import paddle.fluid as fluid
import six
import numpy
import math

```

Then, define the parameters:

```python
EMBED_SIZE = 32   # embedding dimensions
HIDDEN_SIZE = 256 # hidden layer size
N = 5             # ngram size, here fixed 5
BATCH_SIZE = 100  # batch size
PASS_NUM = 100    # Training rounds

use_cuda = False  # Set to True if trained with GPU

word_dict = paddle.dataset.imikolov.build_dict()
dict_size = len(word_dict)
```

A larger `BATCH_SIZE` will make the training converge faster, but it will also consume more memory. Since the word vector calculation is large, if the environment allows, please turn on the GPU for training, and get results faster.
Unlike the previous PaddlePaddle v2 version, in the new Fluid version, we don't have to manually calculate the word vector. PaddlePaddle provides a built-in method `fluid.embedding`, which we can use directly to construct an N-gram neural network.

- Let's define our N-gram neural network structure. This structure is used in both training and predicting. Because the word vector is sparse, we pass the parameter `is_sparse == True` to speed up the update of the sparse matrix.

```python
def inference_program(words, is_sparse):

    embed_first = fluid.embedding(
        input=words[0],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')
    embed_second = fluid.embedding(
        input=words[1],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')
    embed_third = fluid.embedding(
        input=words[2],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')
    embed_fourth = fluid.embedding(
        input=words[3],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')

    concat_embed = fluid.layers.concat(
        input=[embed_first, embed_second, embed_third, embed_fourth], axis=1)
    hidden1 = fluid.layers.fc(input=concat_embed,
                              size=HIDDEN_SIZE,
                              act='sigmoid')
    predict_word = fluid.layers.fc(input=hidden1, size=dict_size, act='softmax')
    return predict_word
```

- Based on the neural network structure above, we can define our training method as follows:

```python
def train_program(predict_word):
    # The definition of'next_word' must be after the declaration of inference_program.
    # Otherwise the sequence of the train program input data becomes [next_word, firstw, secondw,
    #thirdw, fourthw], This is not true.
    next_word = fluid.data(name='nextw', shape=[None, 1], dtype='int64')
    cost = fluid.layers.cross_entropy(input=predict_word, label=next_word)
    avg_cost = fluid.layers.mean(cost)
    return avg_cost

def optimizer_func():
    return fluid.optimizer.AdagradOptimizer(
        learning_rate=3e-3,
        regularization=fluid.regularizer.L2DecayRegularizer(8e-4))

```

- Now we can start training. This version is much simpler than before. We have ready-made training and test sets: `paddle.dataset.imikolov.train()` and `paddle.dataset.imikolov.test()`. Both will return a reader. In PaddlePaddle, the reader is a Python function that reads the next piece of data when called each time . It is a Python generator.

`fluid.io.batch` will read in a reader and output a batched reader. We can also output the training of each step and batch during the training process.

```
/* Detailed source-code truncated for AI context efficiency. */
```

- `train_loop` will start training. The log of the training process during the period is as follows:

```text
Step 0: Average Cost 7.337213
Step 10: Average Cost 6.136128
Step 20: Average Cost 5.766995
...
```

<a name="model application"></a>
## Model Application
After the model is trained, we can use it to make some predictions.

### Predict the next word
We can use our trained model to predict the next word after learning the previous N-gram.

```
/* Detailed source-code truncated for AI context efficiency. */
```

Since the word vector matrix itself is relatively sparse, the training process takes a long time to reach a certain precision. In order to see the effect simply, the tutorial only sets up with a few rounds of training and ends with the following result. Our model predicts that the next word for `among a group of` is `the`. This is in line with the law of grammar. If we train for longer time, such as several hours, then the next predicted word we will get is `workers`. The format of the predicted output is as follows:

```text
[[0.03768077 0.03463154 0.00018074 ... 0.00022283 0.00029888 0.02967956]]
0
the
```
The first line represents the probability distribution of the predicted word in the dictionary, the second line represents the id corresponding to the word with the highest probability, and the third line represents the word with the highest probability.

The entrance to the entire program is simple:

```python
def main(use_cuda, is_sparse):
    if use_cuda and not fluid.core.is_compiled_with_cuda():
        return

    params_dirname = "word2vec.inference.model"

    train(
        if_use_cuda=use_cuda,
        params_dirname=params_dirname,
        is_sparse=is_sparse)

    infer(use_cuda=use_cuda, params_dirname=params_dirname)


main(use_cuda=use_cuda, is_sparse=True)
```

## Conclusion
In this chapter, we introduced word vectors, the relationship between language models and word vectors and how to obtain word vectors by training neural network models. In information retrieval, we can judge the correlation between query and document keywords based on the cosine value between vectors. In syntactic analysis and semantic analysis, trained word vectors can be used to initialize the model for better results. In the document classification, after the word vector, you can cluster to group synonyms in a document, or you can use N-gram to predict the next word. We hope that everyone can easily use the word vector to conduct research in related fields after reading this chapter.

<a name="references"></a>
## References
1. Bengio Y, Ducharme R, Vincent P, et al. [A neural probabilistic language model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)[J]. journal of machine learning Research, 2003, 3(Feb): 1137-1155.
2. Mikolov T, Kombrink S, Deoras A, et al. [Rnnlm-recurrent neural network language modeling toolkit](http://www.fit.vutbr.cz/~imikolov/rnnlm/rnnlm-demo.pdf)[C ]//Proc. of the 2011 ASRU Workshop. 2011: 196-201.
3. Mikolov T, Chen K, Corrado G, et al. [Efficient estimation of word representations in vector space](https://arxiv.org/pdf/1301.3781.pdf)[J]. arXiv preprint arXiv:1301.3781, 2013 .
4. Maaten L, Hinton G. [Visualizing data using t-SNE](https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf)[J]. Journal of Machine Learning Research, 2008, 9(Nov ): 2579-2605.
5. https://en.wikipedia.org/wiki/Singular_value_decomposition

<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://paddlepaddleimage.cdn.bcebos.com/bookimage/camo.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> is contributed by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

---

### Doc Fluid User Guides Nlp Case Understand Sentiment README (`doc/fluid/user_guides/nlp_case/understand_sentiment/README.md`)

# Sentiment Analysis

The source code of this tutorial is in [book/understand_sentiment](https://github.com/PaddlePaddle/book/tree/develop/06.understand_sentiment). For new users, please refer to [Running This Book](https://github.com/PaddlePaddle/book/blob/develop/README.md#running-the-book) .

## Background Introduction

In natural language processing, sentiment analysis generally refers to judging the emotion expressed by a piece of text. Among them, a piece of text can be a sentence, a paragraph or a document. Emotional state can be two categories, such as (positive, negative), (happy, sad); or three categories, such as (positive, negative, neutral) and so on.The application scenarios of understanding sentiment are very broad, such as dividing the comments posted by users on shopping websites (Amazon, Tmall, Taobao, etc.), travel websites, and movie review websites into positive comments and negative comments; or in order to analyze the user's overall experience with a product, grab user reviews of the product, and perform sentiment analysis. Table 1 shows an example of understanding sentiment of movie reviews:

| Movie Comments | Category |
| -------- | ----- |
|In Feng Xiaogang’s movies of the past few years, it is the best one | Positive |
|Very bad feat, like a local TV series | Negative |
|The round-lens lens is full of brilliance, and the tonal background is beautiful, but the plot is procrastinating, the accent is not good, and even though taking an effort but it is hard to focus on the show | Negative |
|The plot could be scored 4 stars. In addition, the angle of the round lens plusing the scenery of Wuyuan is very much like the feeling of Chinese landscape painting. It satisfied me. | Positive |

<p align="center">Form 1  Sentiment analysis of movie comments</p>

In natural language processing, sentiment is a typical problem of **text categorization**, which divides the text that needs to be sentiment analysis into its category. Text categorization involves two issues: text representation and classification methods. Before the emergence of the deep learning, the mainstream text representation methods are BOW (bag of words), topic models, etc.; the classification methods are SVM (support vector machine), LR (logistic regression) and so on.

For a piece of text, BOW means that its word order, grammar and syntax are ignored, and this text is only treated as a collection of words, so the BOW method does not adequately represent the semantic information of the text. For example, the sentence "This movie is awful" and "a boring, empty, non-connotative work" have a high semantic similarity in sentiment analysis, but their BOW representation has a similarity of zero. Another example is that the BOW is very similar to the sentence "an empty, work without connotations" and "a work that is not empty and has connotations", but in fact they mean differently.

The deep learning we are going to introduce in this chapter overcomes the above shortcomings of BOW representation. It maps text to low-dimensional semantic space based on word order, and performs text representation and classification in end-to-end mode. Its performance is significantly improved compared to the traditional method \[[1](#References)\].

## Model Overview
The text representation models used in this chapter are Convolutional Neural Networks and Recurrent Neural Networks and their extensions. These models are described below.

### Introduction of Text Convolutional Neural Networks (CNN)

We introduced the calculation process of the CNN model applied to text data in the [Recommended System](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system) section. Here is a simple review.

For a CNN, first convolute input word vector sequence to generate a feature map, and then obtain the features of the whole sentence corresponding to the kernel by using a max pooling over time on the feature map. Finally, the splicing of all the features obtained is the fixed-length vector representation of the text. For the text classification problem, connecting it via softmax to construct a complete model. In actual applications, we use multiple convolution kernels to process sentences, and convolution kernels with the same window size are stacked to form a matrix, which can complete the operation more efficiently. In addition, we can also use the convolution kernel with different window sizes to process the sentence. Figure 3 in the [Recommend System](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system) section shows four convolution kernels, namely Figure 1 below, with different colors representing convolution kernel operations of different sizes.

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/text_cnn.png?raw=true" width = "80%" align="center"/><br/>
Figure 1. CNN text classification model
</p>

For the general short text classification problem, the simple text convolution network described above can achieve a high accuracy rate \[[1](#References)\]. If you want a more abstract and advanced text feature representation, you can construct a deep text convolutional neural network\[[2](#References), [3](#References)\].

### Recurrent Neural Network (RNN)

RNN is a powerful tool for accurately modeling sequence data. In fact, the theoretical computational power of the RNN is perfected by Turing' \[[4](#References)\]. Natural language is a typical sequence data (word sequence). In recent years, RNN and its derivation (such as long short term memory\[[5](#References)\]) have been applied in many natural language fields, such as in language models, syntactic parsing, semantic role labeling (or general sequence labeling), semantic representation, graphic generation, dialogue, machine translation, etc., all perform well and even become the best at present.

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/06.understand_sentiment/image/rnn.png?raw=true" width = "60%" align="center"/><br />
Figure 2. Schematic diagram of the RNN expanded by time
</p>

The RNN expands as time is shown in Figure 2: at the time of $t$, the network reads the $t$th input $x_t$ (vector representation) and the state value of the hidden layer at the previous moment $h_{t- 1}$ (vector representation, $h_0$ is normally initialized to $0$ vector), and calculate the state value $h_t$ of the hidden layer at this moment. Repeat this step until all the inputs have been read. If the function is recorded as $f$, its formula can be expressed as:

$$h_t=f(x_t,h_{t-1})=\sigma(W_{xh}x_t+W_{hh}h_{t-1}+b_h)$$

Where $W_{xh}$ is the matrix parameter of the input to the hidden layer, $W_{hh}$ is the matrix parameter of the hidden layer to the hidden layer, and $b_h$ is the bias vector parameter of the hidden layer, $\sigma $ is the $sigmoid$ function.

When dealing with natural language, the word (one-hot representation) is usually mapped to its word vector representation, and then used as the input $x_t$ for each moment of the recurrent neural network. In addition, other layers may be connected to the hidden layer of the RNN depending on actual needs. For example, you can connect the hidden layer output of a RNN to the input of the next RNN to build a deep or stacked RNN, or extract the hidden layer state at the last moment as a sentence representation and then implement a classification model, etc.

### Long and Short Term Memory Network (LSTM)

For longer sequence data, the gradient disappearance or explosion phenomenon is likely to occur during training RNN\[[6](#References)\]. To solve this problem, Hochreiter S, Schmidhuber J. (1997) proposed LSTM (long short term memory\[[5](#References)\]).

Compared to a simple RNN, LSTM adds memory unit $c$, input gate $i$, forget gate $f$, and output gate $o$. The combination of these gates and memory units greatly enhances the ability of the recurrent neural network to process long sequence data. If the function \is denoted as $F$, the formula is:

$$ h_t=F(x_t,h_{t-1})$$

$F$ It is a combination of the following formulas\[[7](#References)\]:
$$ i_t = \sigma{(W_{xi}x_t+W_{hi}h_{t-1}+W_{ci}c_{t-1}+b_i)} $$
$$ f_t = \sigma(W_{xf}x_t+W_{hf}h_{t-1}+W_{cf}c_{t-1}+b_f) $$
$$ c_t = f_t\odot c_{t-1}+i_t\odot tanh(W_{xc}x_t+W_{hc}h_{t-1}+b_c) $$
$$ o_t = \sigma(W_{xo}x_t+W_{ho}h_{t-1}+W_{co}c_{t}+b_o) $$
$$ h_t = o_t\odot tanh(c_t) $$
Where $i_t, f_t, c_t, o_t$ respectively represent the vector representation of the input gate, the forget gate, the memory unit and the output gate, the $W$ and $b$ with the angular label are the model parameters, and the $tanh$ is the hyperbolic tangent function. , $\odot$ represents an elementwise multiplication operation. The input gate controls the intensity of the new input into the memory unit $c$, the forget gate controls the intensity of the memory unit to maintain the previous time value, and the output gate controls the intensity of the output memory unit. The three gates are calculated in a similar way, but with completely different parameters.They controll the memory unit $c$ in different ways, as shown in Figure 3:

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/06.understand_sentiment/image/lstm.png?raw=true" width = "65%" align="center"/><br />
Figure 3. LSTM for time $t$ [7]
</p>

LSTM enhances its ability to handle long-range dependencies by adding memory and control gates to RNN. A similar principle improvement is Gated Recurrent Unit (GRU)\[[8](#References)\], which is more concise in design. **These improvements are different, but their macro descriptions are the same as simple recurrent neural networks (as shown in Figure 2). That is, the hidden state changes according to the current input and the hidden state of the previous moment, and this process is continuous until the input is processed:**

$$ h_t=Recurrent(x_t,h_{t-1})$$

Among them, $Recurrent$ can represent a RNN, GRU or LSTM.



<a name="Stacked Bidirectional LSTM"></a>
### Stacked Bidirectional LSTM

For a normal directional RNN, $h_t$ contains the input information before the $t$ time, which is the above context information. Similarly, in order to get the following context information, we can use a RNN in the opposite direction (which will be processed in reverse order). Combined with the method of constructing deep-loop neural networks (deep neural networks often get more abstract and advanced feature representations), we can build a more powerful LSTM-based stack bidirectional recurrent neural network\[[9](#References )\] to model time series data.

As shown in Figure 4 (taking three layers as an example), the odd-numbered LSTM is forward and the even-numbered LSTM is inverted. The higher-level LSTM uses the lower LSTM and all previous layers of information as input. The maximum pooling of the highest-level LSTM sequence in the time dimension can be used to obtain a fixed-length vector representation of the text (this representation fully fuses the contextual information and deeply abstracts of the text), and finally we connect the text representation to the softmax to build the classification model.

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/06.understand_sentiment/image/stacked_lstm.jpg?raw=true" width=450><br/>
Figure 4. Stacked bidirectional LSTM for text categorization
</p>


## Dataset Introduction

We use the [IMDB sentiment analysis data set](http://ai.stanford.edu/%7Eamaas/data/sentiment/) as an example. The training and testing IMDB dataset contain 25,000 labeled movie reviews respectively. Among them, the score of the negative comment is less than or equal to 4, and the score of the positive comment is greater than or equal to 7, full score is 10.
```text
aclImdb
|- test
   |-- neg
   |-- pos
|- train
   |-- neg
   |-- pos
```
Paddle implements the automatic download and read the imdb dataset in `dataset/imdb.py`, and provides API for reading dictionary, training data, testing data, and so on.

## Model Configuration

In this example, we implement two text categorization algorithms based on the text convolutional neural network described in the [Recommender System](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system) section and [Stacked Bidirectional LSTM](#Stacked Bidirectional LSTM). We first import the packages we need to use and define global variables:

```python
from __future__ import print_function
import paddle
import paddle.fluid as fluid
import numpy as np
import sys
import math

CLASS_DIM = 2 #Number of categories for sentiment analysis
EMB_DIM = 128 #Dimensions of the word vector
HID_DIM = 512 #Dimensions of hide layer
STACKED_NUM = 3 #LSTM Layers of the bidirectional stack
BATCH_SIZE = 128 #batch size

```


### Text Convolutional Neural Network
We build the neural network `convolution_net`, the sample code is as follows.
Note that `fluid.nets.sequence_conv_pool` contains both convolution and pooling layers.

```python
#Textconvolution neural network
def convolution_net(data, input_dim, class_dim, emb_dim, hid_dim):
    emb = fluid.embedding(
        input=data, size=[input_dim, emb_dim], is_sparse=True)
    conv_3 = fluid.nets.sequence_conv_pool(
        input=emb,
        num_filters=hid_dim,
        filter_size=3,
        act="tanh",
        pool_type="sqrt")
    conv_4 = fluid.nets.sequence_conv_pool(
        input=emb,
        num_filters=hid_dim,
        filter_size=4,
        act="tanh",
        pool_type="sqrt")
    prediction = fluid.layers.fc(
        input=[conv_3, conv_4], size=class_dim, act="softmax")
    return prediction
```

The network input `input_dim` indicates the size of the dictionary, and `class_dim` indicates the number of categories. Here, we implement the convolution and pooling operations using the [`sequence_conv_pool`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/fluid/nets.py) API.

<a name="Stack value bidirectional LSTM"></a>

### Stacked bidirectional LSTM

The code of the stack bidirectional LSTM `stacked_lstm_net` is as follows:

```python
#Stack Bidirectional LSTM
def stacked_lstm_net(data, input_dim, class_dim, emb_dim, hid_dim, stacked_num):

    # Calculate word vectorvector
    emb = fluid.embedding(
        input=data, size=[input_dim, emb_dim], is_sparse=True)

    #First stack
    #Fully connected layer
    fc1 = fluid.layers.fc(input=emb, size=hid_dim)
    #lstm layer
    lstm1, cell1 = fluid.layers.dynamic_lstm(input=fc1, size=hid_dim)

    inputs = [fc1, lstm1]

    #All remaining stack structures
    for i in range(2, stacked_num + 1):
        fc = fluid.layers.fc(input=inputs, size=hid_dim)
        lstm, cell = fluid.layers.dynamic_lstm(
            input=fc, size=hid_dim, is_reverse=(i % 2) == 0)
        inputs = [fc, lstm]

    #pooling layer
    pc_last = fluid.layers.sequence_pool(input=inputs[0], pool_type='max')
    lstm_last = fluid.layers.sequence_pool(input=inputs[1], pool_type='max')

    #Fully connected layer, softmax prediction
    prediction = fluid.layers.fc(
        input=[fc_last, lstm_last], size=class_dim, act='softmax')
    return prediction
```
The above stacked bidirectional LSTM abstracts the advanced features and maps them to vectors of the same size as the number of classification. The 'softmax' activation function of the last fully connected layer is used to calculate the probability of a certain category.

Again, here we can call any network structure of `convolution_net` or `stacked_lstm_net` for training and learning. Let's take `convolution_net` as an example.

Next we define the prediction program (`inference_program`). We use `convolution_net` to predict the input of `fluid.layer.data`.

```python
def inference_program(word_dict):
    data = fluid.data(
        name="words", shape=[None], dtype="int64", lod_level=1)
    dict_dim = len(word_dict)
    net = convolution_net(data, dict_dim, CLASS_DIM, EMB_DIM, HID_DIM)
    # net = stacked_lstm_net(data, dict_dim, CLASS_DIM, EMB_DIM, HID_DIM, STACKED_NUM)
    return net
```

We define `training_program` here, which uses the result returned from `inference_program` to calculate the error. We also define the optimization function `optimizer_func`.

Because it is supervised learning, the training set tags are also defined in `fluid.layers.data`. During training, cross-entropy is used as a loss function in `fluid.layer.cross_entropy`.

During the testing, the classifier calculates the probability of each output. The first returned value is specified as cost.

```python
def train_program(prediction):
    label = fluid.data(name="label", shape=[None, 1], dtype="int64")
    cost = fluid.layers.cross_entropy(input=prediction, label=label)
    avg_cost = fluid.layers.mean(cost)
    accuracy = fluid.layers.accuracy(input=prediction, label=label)
    return [avg_cost, accuracy] #return average cost and accuracy acc

#Optimization function
def optimizer_func():
    return fluid.optimizer.Adagrad(learning_rate=0.002)
```

## Training Model

### Defining the training environment

Define whether your training is on the CPU or GPU:


```python
use_cuda = False #train on cpu
place = fluid.CUDAPlace(0) if use_cuda else fluid.CPUPlace()
```

### Defining the data creator

The next step is to define a data creator for training and testing. The creator reads in a data of size BATCH_SIZE. Paddle.dataset.imdb.word_dict will provide a size of BATCH_SIZE after each time shuffling, which is the cache size: buf_size.

Note: It may take a few minutes to read the IMDB data, please be patient.

```python
print("Loading IMDB word dict....")
word_dict = paddle.dataset.imdb.word_dict()

print ("Reading training data....")
train_reader = fluid.io.batch(
    fluid.io.shuffle(
        paddle.dataset.imdb.train(word_dict), buf_size=25000),
    batch_size=BATCH_SIZE)
print("Reading testing data....")
test_reader = fluid.io.batch(
    paddle.dataset.imdb.test(word_dict), batch_size=BATCH_SIZE)
```
Word_dict is a dictionary sequence, which is the correspondence between words and labels. You can see it specifically by running the next code:
```python
word_dict
```
Each line is a correspondence such as ('limited': 1726), which indicates that the label corresponding to the word limited is 1726.

### Construction Trainer
The trainer requires a training program and a training optimization function.

```python
exe = fluid.Executor(place)
prediction = inference_program(word_dict)
[avg_cost, accuracy] = train_program(prediction)#training program
sgd_optimizer = optimizer_func()# training optimization function
sgd_optimizer.minimize(avg_cost)
```

This function is used to calculate the result of the model on the test dataset.
```python
def train_test(program, reader):
    count = 0
    feed_var_list = [
        program.global_block().var(var_name) for var_name in feed_order
    ]
    feeder_test = fluid.DataFeeder(feed_list=feed_var_list, place=place)
    test_exe = fluid.Executor(place)
    accumulated = len([avg_cost, accuracy]) * [0]
    for test_data in reader():
        avg_cost_np = test_exe.run(
            program=program,
            feed=feeder_test.feed(test_data),
            fetch_list=[avg_cost, accuracy])
        accumulated = [
            x[0] + x[1][0] for x in zip(accumulated, avg_cost_np)
        ]
        count += 1
    return [x / count for x in accumulated]
```

### Providing data and building a main training loop

`feed_order` is used to define the mapping relationship between each generated data and `fluid.layers.data`. For example, the data in the first column generated by `imdb.train` corresponds to the `words` feature.

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Training process

We print the output of each step in the main loop of the training, and we can observe the training situation.

### Start training

Finally, we start the training main loop to start training. The training time is longer. If you want to get the result faster, you can shorten the training time by adjusting the loss value range or the number of training steps at the cost of reducing the accuracy.

```python
train_loop(fluid.default_main_program())
```

## Application Model

### Building a predictor

As the training process, we need to create a prediction process and use the trained models and parameters to make predictions. `params_dirname` is used to store the various parameters in the training process.

```python
place = fluid.CUDAPlace(0) if use_cuda else fluid.CPUPlace()
exe = fluid.Executor(place)
inference_scope = fluid.core.Scope()
```

### Generating test input data

In order to make predictions, we randomly select 3 comments. We correspond each word in the comment to the id in `word_dict`. If the word is not in the dictionary, set it to `unknown`.
Then we use `create_lod_tensor` to create the tensor of the detail level. For a detailed explanation of this function, please refer to [API documentation](http://paddlepaddle.org/documentation/docs/en/1.2/user_guides/howto/basic_concept/lod_tensor.html).

```python
reviews_str = [
    b'read the book forget the movie', b'this is a great movie', b'this is very bad'
]
reviews = [c.split() for c in reviews_str]

UNK = word_dict['<unk>']
lod = []
for c in reviews:
    lod.append([word_dict.get(words, UNK) for words in c])

base_shape = [[len(c) for c in lod]]
lod = np.array(sum(lod, []), dtype=np.int64)

tensor_words = fluid.create_lod_tensor(lod, base_shape, place)
```

## Applying models and making predictions

Now we can make positive or negative predictions for each comment.

```python
with fluid.scope_guard(inference_scope):

    [inferencer, feed_target_names,
     fetch_targets] = fluid.io.load_inference_model(params_dirname, exe)

    assert feed_target_names[0] == "words"
    results = exe.run(inferencer,
                      feed={feed_target_names[0]: tensor_words},
                      fetch_list=fetch_targets,
                      return_numpy=False)
    np_data = np.array(results[0])
    for i, r in enumerate(np_data):
        print("Predict probability of ", r[0], " to be positive and ", r[1],
              " to be negative for review \'", reviews_str[i], "\'")

```


## Conclusion

In this chapter, we take sentiment analysis as an example to introduce end-to-end short text classification using deep learning, and complete all relevant experiments using PaddlePaddle. At the same time, we briefly introduce two text processing models: convolutional neural networks and recurrent neural networks. In the following chapters, we will see the application of these two basic deep learning models on other tasks.

<a name="References"></a>

## References

1. Kim Y. [Convolutional neural networks for sentence classification](http://arxiv.org/pdf/1408.5882)[J]. arXiv preprint arXiv:1408.5882, 2014.
2. Kalchbrenner N, Grefenstette E, Blunsom P. [A convolutional neural network for modelling sentences](http://arxiv.org/pdf/1404.2188.pdf?utm_medium=App.net&utm_source=PourOver)[J]. arXiv preprint arXiv:1404.2188, 2014.
3. Yann N. Dauphin, et al. [Language Modeling with Gated Convolutional Networks](https://arxiv.org/pdf/1612.08083v1.pdf)[J] arXiv preprint arXiv:1612.08083, 2016.
4. Siegelmann H T, Sontag E D. [On the computational power of neural nets](http://research.cs.queensu.ca/home/akl/cisc879/papers/SELECTED_PAPERS_FROM_VARIOUS_SOURCES/05070215382317071.pdf)[C]//Proceedings of the fifth annual workshop on Computational learning theory. ACM, 1992: 440-449.
5. Hochreiter S, Schmidhuber J. [Long short-term memory](http://web.eecs.utk.edu/~itamar/courses/ECE-692/Bobby_paper1.pdf)[J]. Neural computation, 1997, 9(8): 1735-1780.
6. Bengio Y, Simard P, Frasconi P. [Learning long-term dependencies with gradient descent is difficult](http://www-dsi.ing.unifi.it/~paolo/ps/tnn-94-gradient.pdf)[J]. IEEE transactions on neural networks, 1994, 5(2): 157-166.
7. Graves A. [Generating sequences with recurrent neural networks](http://arxiv.org/pdf/1308.0850)[J]. arXiv preprint arXiv:1308.0850, 2013.
8. Cho K, Van Merriënboer B, Gulcehre C, et al. [Learning phrase representations using RNN encoder-decoder for statistical machine translation](http://arxiv.org/pdf/1406.1078)[J]. arXiv preprint arXiv:1406.1078, 2014.
9. Zhou J, Xu W. [End-to-end learning of semantic role labeling using recurrent neural networks](http://www.aclweb.org/anthology/P/P15/P15-1109.pdf)[C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics. 2015.

<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://paddlepaddleimage.cdn.bcebos.com/bookimage/camo.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> is contributed by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

---

### Doc Fluid User Guides Rec Case Recommender System README (`doc/fluid/user_guides/rec_case/recommender_system/README.md`)

# Recommender System

The source code of this tutorial is in [book/recommender_system](https://github.com/PaddlePaddle/book/tree/develop/05.recommender_system). For new users, please refer to [Running This Book](https://github.com/PaddlePaddle/book/blob/develop/README.md#running-the-book) .

## Background Introduction

With the continuous development of network technology and the ever-expanding scale of e-commerce, the number and variety of goods grow rapidly and users need to spend a lot of time to find the goods they want to buy. This is information overload. In order to solve this problem, recommendation system came into being.

The recommendation system is a subset of the Information Filtering System, which can be used in a range of areas such as movies, music, e-commerce, and Feed stream recommendations. The recommendation system discovers the user's personalized needs and interests by analyzing and mining user behaviors, and recommends information or products that may be of interest to the user. Unlike search engines, recommendation system do not require users to accurately describe their needs, but model their historical behavior to proactively provide information that meets user interests and needs.

The GroupLens system \[[1](#references)\] introduced by the University of Minnesota in 1994 is generally considered to be a relatively independent research direction for the recommendation system. The system first proposed the idea of completing recommendation task based on collaborative filtering. After that, the collaborative filtering recommendation based on the model led the development of recommendation system for more than ten years.

The traditional personalized recommendation system methods mainly include:

- Collaborative Filtering Recommendation: This method is one of the most widely used technologies which requires the collection and analysis of users' historical behaviors, activities and preferences. It can usually be divided into two sub-categories: User-Based Recommendation \[[1](#references)\] and Item-Based Recommendation \[[2](#references)\]. A key advantage of this method is that it does not rely on the machine to analyze the content characteristics of the item, so it does not need to understand the item itself to accurately recommend complex items such as movies. However, the disadvantage is that there is a cold start problem for new users without any behavior. At the same time, there is also a sparsity problem caused by insufficient interaction data between users and commodities. It is worth mentioning that social network \[[3](#references)\] or geographic location and other context information can be integrated into collaborative filtering.
- Content-Based Filtering Recommendation \[[4](#references)\] : This method uses the content description of the product to abstract meaningful features by calculating the similarity between the user's interest and the product description to make recommendations to users. The advantage is that it is simple and straightforward. It does not need to evaluate products based on the comments of users. Instead, it compares the product similarity by product attributes to recommend similar products to the users of interest. The disadvantage is that there is also a cold start problem for new users without any behavior.
- Hybrid Recommendation \[[5](#references)\]: Use different inputs and techniques to jointly recommend items to complement each single recommendation technique.

In recent years, deep learning has achieved great success in many fields. Both academia and industry are trying to apply deep learning to the field of recommendation systems. Deep learning has excellent ability to automatically extract features, can learn multi-level abstract feature representations, and learn heterogeneous or cross-domain content information, which can deal with the cold start problem \[[6](#references)\] of recommendation system to some extent. This tutorial focuses on the deep learning model of recommendation system and how to implement the model with PaddlePaddle.

## Result Demo

We use a dataset containing user information, movie information, and movie ratings as a recommendation system. When we train the model, we only need to input the corresponding user ID and movie ID, we can get a matching score (range [0, 5], the higher the score is regarded as the greater interest), and then according to the recommendation of all movies sort the scores and recommend them to movies that may be of interest to the user.

```
Input movie_id: 1962
Input user_id: 1
Prediction Score is 4.25
```

## Model Overview

In this chapter, we first introduce YouTube's video personalization recommendation system \[[7](#references)\], and then introduce the fusion recommendation model we implemented.

### YouTube's Deep Neural Network Personalized Recommendation System

YouTube is the world's largest video uploading, sharing and discovery site, and the YouTube Personalized Recommendation System recommends personalized content from a growing library to more than 1 billion users. The entire system consists of two neural networks: a candidate generation network and a ranking network. The candidate generation network generates hundreds of candidates from a million-level video library, and the ranking network sorts the candidates and outputs the highest ranked tens of results. The system structure is shown in Figure 1:

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/YouTube_Overview.png?raw=true" width="70%" ><br/>
Figure 1. YouTube personalized recommendation system structure
</p>

#### Candidate Generation Network

The candidate generation network models the recommendation problem as a multi-class classification problem with a large number of categories. For a Youtube user, using its watching history (video ID), search tokens, demographic information (such as geographic location, user login device), binary features (such as gender, whether to log in), and continuous features (such as user age), etc., multi-classify all videos in the video library to obtain the classification result of each category (ie, the recommendation probability of each video), eventually outputting hundreds of videos with high probability.

First, the historical information such as watching history and search token records are mapped to vectors and averaged to obtain a fixed length representation. At the same time, demographic characteristics are input to optimize the recommendation effect of new users, and the binary features and continuous features are normalized to the range [0, 1]. Next, put all the feature representations into a vector and input them to the non-linear multilayer perceptron (MLP, see [Identification Figures](https://github.com/PaddlePaddle/book/blob/develop/02.recognize_digits/README.md) tutorial). Finally, during training, the output of the MLP is classified by softmax. When predicting, the similarity of the user's comprehensive features (MLP output) to all videos' features is calculated, and the highest score of $k$ is obtained as the result of the candidate generation network. Figure 2 shows the candidate generation network structure.

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/Deep_candidate_generation_model_architecture.png?raw=true" width="70%" ><br/>
Figure 2. Candidate generation network structure
</p>

For a user $U$, the formula for predicting whether the video $\omega$ that the user wants to watch at the moment is video $i$ is:

$$P(\omega=i|u)=\frac{e^{v_{i}u}}{\sum_{j \in V}e^{v_{j}u}}$$

Where $u$ is the feature representation of the user $U$, $V$ is the video library collection, and $v_i$ is the feature representation of the $i$ video in the video library. $u$ and $v_i$ are vectors of equal length, and the dot product can be implemented by a fully connected layer.

Considering that the number of categories in the softmax classification is very large, in order to ensure a certain computational efficiency: 1) in the training phase, use negative sample category sampling to reduce the number of actually calculated categories to thousands; 2) in the recommendation (prediction) phase, ignore the normalized calculation of softmax (does not affect the result), and simplifies the category scoring problem into the nearest neighbor search problem in the dot product space, then takes the nearest $k$ video of $u$ as a candidate for generation.

#### Ranking Network
The structure of the ranking network is similar to the candidate generation network, but its goal is to perform finer ranking of the candidates. Similar to the feature extraction method in traditional advertisement ranking, a large number of related features (such as video ID, last watching time, etc.) for video sorting are also constructed here. These features are treated similarly to the candidate generation network, except that at the top of the ranking network is a weighted logistic regression that scores all candidate videos and sorts them from high to low. Then, return to the user.

### Fusion recommendation model
This section uses Convolutional Neural Networks to learn the representation of movie titles. The convolutional neural network for text and the fusion recommendation model are introduced in turn.

#### Convolutional Neural Network (CNN) for text

Convolutional neural networks are often used to deal with data of a grid-like topology. For example, an image can be viewed as a pixel of a two-dimensional grid, and a natural language can be viewed as a one-dimensional sequence of words. Convolutional neural networks can extract a variety of local features and combine them to obtain more advanced feature representations. Experiments show that convolutional neural networks can efficiently model image and text problems.

The convolutional neural network is mainly composed of convolution and pooling operations, and its application and combination methods are flexible and varied. In this section we will explain the network as shown in Figure 3:

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/text_cnn.png?raw=true" width = "80%" align="center"/><br />
Figure 3. Convolutional neural network text classification model
</p>

Suppose the length of the sentence to be processed is $n$, where the word vector of the $i$ word is $x_i\in\mathbb{R}^k$, and $k$ is the dimension size.

First, splicing the word vector: splicing each $h$ word to form a word window of size $h$, denoted as $x_{i:i+h-1}$, which represents the word sequence splicing of $x_{i}, x_{i+1}, \ldots, x_{i+h-1}$, where $i$ represents the position of the first word in the word window throughout the sentence, ranging from $1$ to $n-h+1$, $x_{i:i+h-1}\in\mathbb{R}^{hk}$.

Second, perform a convolution operation: apply the convolution kernel $w\in\mathbb{R}^{hk}$ to the window $x_{i:i+h-1}$ containing $h$ words. , get the feature $c_i=f(w\cdot x_{i:i+h-1}+b)$, where $b\in\mathbb{R}$ is the bias and $f$ is the non Linear activation function, such as $sigmoid$. Apply the convolution kernel to all word windows ${x_{1:h}, x_{2:h+1},\ldots,x_{n-h+1:n}}$ in the sentence, producing a feature map:

$$c=[c_1,c_2,\ldots,c_{n-h+1}], c \in \mathbb{R}^{n-h+1}$$

Next, using the max pooling over time for feature maps to obtain the feature $\hat c$, of the whole sentence corresponding to this convolution kernel, which is the maximum value of all elements in the feature map:

$$\hat c=max(c)$$

#### Fusion recommendation model overview

In the film personalized recommendation system that incorporates the recommendation model:

1. First, take user features and movie features as input to the neural network, where:

   - The user features incorporate four attribute information: user ID, gender, occupation, and age.

   - The movie feature incorporate three attribute information: movie ID, movie type ID, and movie name.

2. For the user feature, map the user ID to a vector representation with a dimension size of 256, enter the fully connected layer, and do similar processing for the other three attributes. Then the feature representations of the four attributes are fully connected and added separately.

3. For movie features, the movie ID is processed in a manner similar to the user ID. The movie type ID is directly input into the fully connected layer in the form of a vector, and the movie name is represented by a fixed-length vector using a text convolutional neural network. The feature representations of the three attributes are then fully connected and added separately.

4. After obtaining the vector representation of the user and the movie, calculate the cosine similarity of them as the score of the personalized recommendation system. Finally, the square of the difference between the similarity score and the user's true score is used as the loss function of the regression model.

<p align="center">
<img src="https://github.com/PaddlePaddle/book/blob/develop/05.recommender_system/image/rec_regression_network.png?raw=true" width="90%" ><br/>
Figure 4. Fusion recommendation model
</p>

## Data Preparation

### Data Introduction and Download

We take [MovieLens Million Dataset (ml-1m)](http://files.grouplens.org/datasets/movielens/ml-1m.zip) as an example. The ml-1m dataset contains 1,000,000 reviews of 4,000 movies by 6,000 users (scores ranging from 1 to 5, all integer), collected by the GroupLens Research lab.

Paddle provides modules for automatically loading data in the API. The data module is `paddle.dataset.movielens`


```python
from __future__ import print_function
import paddle
movie_info = paddle.dataset.movielens.movie_info()
print(list(movie_info.values())[0])
```


```python
# Run this block to show dataset's documentation
# help(paddle.dataset.movielens)
```

The original data includes feature data of the movie, user's feature data, and the user's rating of the movie.

For example, one of the movie features is:


```python
movie_info = paddle.dataset.movielens.movie_info()
print(list(movie_info.values())[0])
```

    <MovieInfo id(1), title(Toy Story ), categories(['Animation', "Children's", 'Comedy'])>


This means that the movie id is 1, and the title is 《Toy Story》, which is divided into three categories. These three categories are animation, children, and comedy.


```python
user_info = paddle.dataset.movielens.user_info()
print(list(user_info.values())[0])
```

    <UserInfo id(1), gender(F), age(1), job(10)>


This means that the user ID is 1, female, and younger than 18 years old. The occupation ID is 10.


Among them, the age uses the following distribution

*  1:  "Under 18"
* 18:  "18-24"
* 25:  "25-34"
* 35:  "35-44"
* 45:  "45-49"
* 50:  "50-55"
* 56:  "56+"

The occupation is selected from the following options:

*  0:  "other" or not specified
*  1:  "academic/educator"
*  2:  "artist"
*  3:  "clerical/admin"
*  4:  "college/grad student"
*  5:  "customer service"
*  6:  "doctor/health care"
*  7:  "executive/managerial"
*  8:  "farmer"
*  9:  "homemaker"
* 10:  "K-12 student"
* 11:  "lawyer"
* 12:  "programmer"
* 13:  "retired"
* 14:  "sales/marketing"
* 15:  "scientist"
* 16:  "self-employed"
* 17:  "technician/engineer"
* 18:  "tradesman/craftsman"
* 19:  "unemployed"
* 20:  "writer"

For each training or test data, it is <user features> + <movie feature> + rating.

For example, we get the first training data:


```python
train_set_creator = paddle.dataset.movielens.train()
train_sample = next(train_set_creator())
uid = train_sample[0]
mov_id = train_sample[len(user_info[uid].value())]
print("User %s rates Movie %s with Score %s"%(user_info[uid], movie_info[mov_id], train_sample[-1]))
```

    User <UserInfo id(1), gender(F), age(1), job(10)> rates Movie <MovieInfo id(1193), title(One Flew Over the Cuckoo's Nest ), categories(['Drama'])> with Score [5.0]


That is, the user 1 evaluates the movie 1193 as 5 points.

## Configuration Instruction

Below we begin to configure the model based on the form of the input data. First import the required library functions and define global variables.

- IS_SPARSE: whether to use sparse update in embedding
- PASS_NUM: number of epoch


```python
import math
import sys
import numpy as np
import paddle
import paddle.fluid as fluid
import paddle.fluid.layers as layers
import paddle.fluid.nets as nets

IS_SPARSE = True
BATCH_SIZE = 256
PASS_NUM = 20
```

Then define the model configuration for our user feature synthesis model

```
/* Detailed source-code truncated for AI context efficiency. */
```

As shown in the code above, for each user, we enter a 4-dimensional feature. This includes user_id, gender_id, age_id, job_id. These dimensional features are simple integer values. In order to facilitate the subsequent neural network processing of these features, we use the language model in NLP to transform these discrete integer values ​​into embedding. And form them into usr_emb, usr_gender_emb, usr_age_emb, usr_job_emb, respectively.

Then, we enter all the user features into a fully connected layer(fc). Combine all features into one 200-dimension feature.

Furthermore, we make a similar transformation for each movie feature, the network configuration is:


```
/* Detailed source-code truncated for AI context efficiency. */
```


The title of a movie is a sequence of integers, and the integer represents the subscript of the word in the index sequence. This sequence is sent to the `sequence_conv_pool` layer, which uses convolution and pooling on the time dimension. Because of this, the output will be fixed length, although the length of the input sequence will vary.

Finally, we define an `inference_program` to calculate the similarity between user features and movie features using cosine similarity.

```python
def inference_program():
    """the combined network"""

    usr_combined_features = get_usr_combined_features()
    mov_combined_features = get_mov_combined_features()

    inference = layers.cos_sim(X=usr_combined_features, Y=mov_combined_features)
    scale_infer = layers.scale(x=inference, scale=5.0)

    return scale_infer
```

Furthermore, we define a `train_program` to use the result computed by `inference_program`, and calculate the error with the help of the tag data. We also define an `optimizer_func` to define the optimizer.

```python
def train_program():
    """define the cost function"""

    scale_infer = inference_program()

    label = fluid.data(name='score', shape=[None, 1], dtype='float32')
    square_cost = layers.square_error_cost(input=scale_infer, label=label)
    avg_cost = layers.mean(square_cost)

    return [avg_cost, scale_infer]


def optimizer_func():
    return fluid.optimizer.SGD(learning_rate=0.2)
```


## Training Model

### Defining the training environment
Define your training environment and specify whether the training takes place on CPU or GPU.

```python
use_cuda = False
place = fluid.CUDAPlace(0) if use_cuda else fluid.CPUPlace()
```

### Defining the data provider
The next step is to define a data provider for training and testing. The provider reads in a data of size `BATCH_SIZE`. `paddle.dataset.movielens.train` will provide a data of size `BATCH_SIZE` after each scribbling, and the size of the out-of-order is the cache size `buf_size`.

```python
train_reader = fluid.io.batch(
    fluid.io.shuffle(
        paddle.dataset.movielens.train(), buf_size=8192),
    batch_size=BATCH_SIZE)

test_reader = fluid.io.batch(
    paddle.dataset.movielens.test(), batch_size=BATCH_SIZE)
```

### Constructing a training process (trainer)
We have constructed a training process here, including training optimization functions.

### Provide data

`feed_order` is used to define the mapping between each generated data and `paddle.layer.data`. For example, the data in the first column generated by `movielens.train` corresponds to the feature `user_id`.

```python
feed_order = [
    'user_id', 'gender_id', 'age_id', 'job_id', 'movie_id', 'category_id',
    'movie_title', 'score'
]
```

### Building training programs and testing programs
The training program and the test program are separately constructed, and the training optimizer is imported.

```python
main_program = fluid.default_main_program()
star_program = fluid.default_startup_program()
[avg_cost, scale_infer] = train_program()

test_program = main_program.clone(for_test=True)
sgd_optimizer = optimizer_func()
sgd_optimizer.minimize(avg_cost)
exe = fluid.Executor(place)

def train_test(program, reader):
    count = 0
    feed_var_list = [
        program.global_block().var(var_name) for var_name in feed_order
    ]
    feeder_test = fluid.DataFeeder(
    feed_list=feed_var_list, place=place)
    test_exe = fluid.Executor(place)
    accumulated = 0
    for test_data in reader():
        avg_cost_np = test_exe.run(program=program,
                                               feed=feeder_test.feed(test_data),
                                               fetch_list=[avg_cost])
        accumulated += avg_cost_np[0]
        count += 1
    return accumulated / count
```

### Build a training main loop and start training
We perform the training cycle according to the training cycle number (`PASS_NUM`) defined above and some other parameters, and perform a test every time. When the test result is good enough, we exit the training and save the trained parameters.

```
/* Detailed source-code truncated for AI context efficiency. */
```
Start training
```python
train_loop()
```

## Model Application

### Generate test data
Use the API of create_lod_tensor(data, lod, place) to generate the tensor of the detail level. `data` is a sequence, and each element is a sequence of index numbers. `lod` is the detail level's information, corresponding to `data`. For example, data = [[10, 2, 3], [2, 3]] means that it contains two sequences of lengths 3 and 2. Correspondingly lod = [[3, 2]], which indicates that it contains a layer of detail information, meaning that `data` has two sequences, lengths of 3 and 2.

In this prediction example, we try to predict the score given by user with ID1 for the movie 'Hunchback of Notre Dame'.

```python
infer_movie_id = 783
infer_movie_name = paddle.dataset.movielens.movie_info()[infer_movie_id].title
user_id = np.array([1]).astype("int64").reshape(-1)
gender_id = np.array([1]).astype("int64").reshape(-1)
age_id = np.array([0]).astype("int64").reshape(-1)
job_id = np.array([10]).astype("int64").reshape(-1)
movie_id = np.array([783]).astype("int64").reshape(-1) # Hunchback of Notre Dame
category_id = fluid.create_lod_tensor(np.array([10, 8, 9], dtype='int64'), [[3]], place) # Animation, Children's, Musical
movie_title = fluid.create_lod_tensor(np.array([1069, 4140, 2923, 710, 988], dtype='int64'), [[5]],
                                      place) # 'hunchback','of','notre','dame','the'
```

### Building the prediction process and testing
Similar to the training process, we need to build a prediction process, where `params_dirname` is the address used to store the various parameters in the training process.

```python
place = fluid.CUDAPlace(0) if use_cuda else fluid.CPUPlace()
exe = fluid.Executor(place)

inference_scope = fluid.core.Scope()
```

### Testing
Now we can make predictions. The `feed_order` we provide should be consistent with the training process.


```python
with fluid.scope_guard(inference_scope):
    [inferencer, feed_target_names,
    fetch_targets] = fluid.io.load_inference_model(params_dirname, exe)

    results = exe.run(inferencer,
                          feed={
                               'user_id': user_id,
                              'gender_id': gender_id,
                              'age_id': age_id,
                              'job_id': job_id,
                              'movie_id': movie_id,
                              'category_id': category_id,
                              'movie_title': movie_title
                          },
                          fetch_list=fetch_targets,
                          return_numpy=False)
    predict_rating = np.array(results[0])
    print("Predict Rating of user id 1 on movie \"" + infer_movie_name +
              "\" is " + str(predict_rating[0][0]))
    print("Actual Rating of user id 1 on movie \"" + infer_movie_name +
              "\" is 4.")
```

## Summary

This chapter introduced the traditional personalized recommendation system method and YouTube's deep neural network personalized recommendation system. It further took movie recommendation as an example, and used PaddlePaddle to train a personalized recommendation neural network model. The personalized recommendation system covers almost all aspects of e-commerce systems, social networks, advertising recommendations, search engines, etc. Deep learning technologies have played an important role in image processing, natural language processing, etc., and will also prevail in personalized recommendation systems.

<a name="references"></a>
## References

1. P. Resnick, N. Iacovou, etc. “[GroupLens: An Open Architecture for Collaborative Filtering of Netnews](http://ccs.mit.edu/papers/CCSWP165.html)”, Proceedings of ACM Conference on Computer Supported Cooperative Work, CSCW 1994. pp.175-186.
2. Sarwar, Badrul, et al. "[Item-based collaborative filtering recommendation algorithms.](http://files.grouplens.org/papers/www10_sarwar.pdf)" *Proceedings of the 10th international conference on World Wide Web*. ACM, 2001.
3. Kautz, Henry, Bart Selman, and Mehul Shah. "[Referral Web: combining social networks and collaborative filtering.](http://www.cs.cornell.edu/selman/papers/pdf/97.cacm.refweb.pdf)" Communications of the ACM 40.3 (1997): 63-65. APA
4. [Peter Brusilovsky](https://en.wikipedia.org/wiki/Peter_Brusilovsky) (2007). *The Adaptive Web*. p. 325.
5. Robin Burke , [Hybrid Web recommendation systems](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.435.7538&rep=rep1&type=pdf), pp. 377-408, The Adaptive Web, Peter Brusilovsky, Alfred Kobsa, Wolfgang Nejdl (Ed.), Lecture Notes in Computer Science, Springer-Verlag, Berlin, Germany, Lecture Notes in Computer Science, Vol. 4321, May 2007, 978-3-540-72078-2.
6. Yuan, Jianbo, et al. ["Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach."](https://arxiv.org/pdf/1611.05480v1.pdf) *arXiv preprint arXiv:1611.05480* (2016).
7. Covington P, Adams J, Sargin E. [Deep neural networks for youtube recommendations](https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/45530.pdf)[C]//Proceedings of the 10th ACM Conference on recommendation systems. ACM, 2016: 191-198.


<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://paddlepaddleimage.cdn.bcebos.com/bookimage/camo.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> is contributed by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

---

### Doc Fluid User Guides Simple Case Word2vec README (`doc/fluid/user_guides/simple_case/word2vec/README.md`)

# Word Vector

The source code of this tutorial is in [book/word2vec](https://github.com/PaddlePaddle/book/tree/develop/04.word2vec). For new users, please refer to [Running This Book](https://github.com/PaddlePaddle/book/blob/develop/README.md#running-the-book) .

## Background

In this chapter, we'll introduce the vector representation of words, also known as word embedding. Word vector is a common operation in natural language processing. It is a common technology underlying Internet services such as search engines, advertising systems, and recommendation systems.

In these Internet services, we often compare the correlation between two words or two paragraphs of text. In order to make such comparisons, we often have to express words in a way that is suitable for computer processing. The most natural way is probably the vector space model.In this way, each word is represented as a one-hot vector whose length is the dictionary size, and each dimension corresponds to each word in a dictionary, except that the value in the corresponding dimension of the word is 1, other elements are 0.

The One-hot vector is natural but has limitation. For example, in the internet advertising system, if the query entered by the user is "Mother's Day", the keyword of an advertisement is "Carnation". Although according to common sense, we know that there is a connection between these two words - Mother's Day should usually give the mother a bunch of carnations; but the distance between the two words corresponds to the one-hot vectors, whether it is Euclidean distance or cosine similarity, the two words are considered to be irrelevant due to their vector orthogonality. The root cause of this conclusion contradicting us is that the amount of information in each word itself is too small. Therefore, just giving two words is not enough for us to accurately determine whether they are relevant. To accurately calculate correlations, we need more information—knowledge from a large amount of data through machine learning methods.

In the field of machine learning, all kinds of "knowledge" are represented by various models, and the word embedding model is one of them. A one-hot vector can be mapped to a lower-dimensional embedding vector by the word embedding model, such as $embedding (Mother's day) = [0.3, 4.2, -1.5, ...], embedding (carnation) = [0.2, 5.6, -2.3, ...]$. In this representation of the embedding vector to which it is mapped, it is desirable that the word vectors corresponding to the similar words on the two semantics (or usages) are "more like", such that the cosine similarity of the corresponding word vectors of "Mother's Day" and "Carnation" is no longer zero.

The word embedding model can be a probability model, a co-occurrence matrix model, or a neural network model. Before implementing neural networks to calculate the embedding vector, the traditional method is to count the co-occurrence matrix $X$ of a word. $X$ is a matrix of $|V| \times |V|$ size, $X_{ij}$ means that in all corpora, The number of words appearing simultaneously with the i-th word and the j-th word in the vocabulary $V$(vocabulary), $|V|$ is the size of the vocabulary. Do matrix decomposition for $X$ (such as singular value decomposition, Singular Value Decomposition \[[5](#references)\]), and the result $U$ is treated as the embedding vector for all words:

$$X = USV^T$$

But such traditional method has many problems:

1) Since many words do not appear, the matrix is extremely sparse, so additional processing of the word frequency is needed to achieve a good matrix decomposition effect;

2) The matrix is very large and the dimensions are too high (usually up to $10^6 \times 10^6$);

3) You need to manually remove the stop words (such as although, a, ...), otherwise these frequently occurring words will also affect the effect of matrix decomposition.

The neural-network-based model does not need to calculate and store a large table that is statistically generated on the whole corpus, but obtains the word vector by learning the semantic information, so the problem above can be well solved. In this chapter, we will show the details of training word vectors based on neural networks and how to train a word embedding model with PaddlePaddle.


##  Result Demo

In this chapter, after the embedding vector is trained, we can use the data visualization algorithm t-SNE\[[4](#references)\] to draw the projection of the word features in two dimensions (as shown below). As can be seen from the figure, semantically related words (such as a, the, these; big, huge) are very close in projection, and semantic unrelated words (such as say, business; decision, japan) are far away from the projection.

<p align="center">
    <img src = "https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/2d_similarity.png?raw=true" width=400><br/>
    Figure 1. Two-dimensional projection of a word vector
</p>

On the other hand, we know that the cosine of two vectors is in the interval of $[-1,1]$: two identical vector cosines are 1, and the cosine value between two mutually perpendicular vectors is 0, The vector cosine of the opposite direction is -1, which the correlation is proportional to the magnitude of the cosine. So we can also calculate the cosine similarity of two word vectors:

```

please input two words: big huge
Similarity: 0.899180685161

please input two words: from company
Similarity: -0.0997506977351

```

The results above can be obtained by running `calculate_dis.py`, loading the words in the dictionary and the corresponding training feature results. We will describe the usage for details in [model application](#model application).


## Overview of Models

Here we introduce three models of training word vectors: N-gram model, CBOW model and Skip-gram model. Their central idea is to get the probability of a word appearing through the context. For the N-gram model, we will first introduce the concept of the language model. In the section [training model](#training model), we'll tutor you to implement it with PaddlePaddle. The latter two models are the most famous neuron word vector models in recent years, developed by Tomas Mikolov in Google \[[3](#references)\], although they are very simple, but the training effect is very good.

### Language Model

Before introducing the word embedding model, let us introduce a concept: the language model.
The language model is intended to model the joint probability function $P(w_1, ..., w_T)$ of a sentence, where $w_i$ represents the ith word in the sentence. The goal of the language model isn that the model gives a high probability to meaningful sentences and a small probability to meaningless sentences.Such models can be applied to many fields, such as machine translation, speech recognition, information retrieval, part-of-speech tagging, handwriting recognition, etc., All of which hope to obtain the probability of a continuous sequence. Take information retrieval as an example, when you search for "how long is a football bame" (bame is a medical term), the search engine will prompt you if you want to search for "how long is a football game", because the probability of calculating "how long is a football bame" is very low, and the word is similar to bame, which may cause errors, the game will maximize the probability of generating the sentence.

For the target probability of the language model $P(w_1, ..., w_T)$, if it is assumed that each word in the text is independent, the joint probability of the whole sentence can be expressed as the product of the conditional probabilities of all the words. which is:

$$P(w_1, ..., w_T) = \prod_{t=1}^TP(w_t)$$

However, we know that the probability of each word in the statement is closely related to the word in front of it, so in fact, the language model is usually represented by conditional probability:

$$P(w_1, ..., w_T) = \prod_{t=1}^TP(w_t | w_1, ... , w_{t-1})$$



### N-gram neural model

In computational linguistics, n-gram is an important text representation method that represents a continuous n items in a text. Each item can be a letter, word or syllable based on the specific application scenario. The n-gram model is also an important method in the statistical language model. When n-gram is used to train the language model, the nth word is generally predicted by the content of the n-1 words of each n-gram.

Scientists such as Yoshua Bengio introduced how to learn a word vector model of a neural network representation in the famous paper Neural Probabilistic Language Models \[[1](#references)\ in 2003. The Neural Network Language Model (NNLM) in this paper connects the linear model and a nonlinear hidden layer. It learns the language model and the word vector simultaneously, that is, by learning a large number of corpora to obtain the vector expression of the words, and the probability of the entire sentence is obtained by using these vectors. Since all words are represented by a low-dimensional vector, learning the language model in this way can overcome the curse of dimensionality.
Note: Because the "Neural Network Language Model" is more general, we do not use the real name of NNLM here, considering its specific practice, this model here is called N-gram neural model.

We have already mentioned above using the conditional probability language model, that is, the probability of the $t$ word in a sentence is related to the first $t-1$ words of the sentence. The farther the word actually has the smaller effect on the word, then if you consider an n-gram, each word is only affected by the preceding `n-1` words, then:

$$P(w_1, ..., w_T) = \prod_{t=n}^TP(w_t|w_{t-1}, w_{t-2}, ..., w_{t-n+1 })$$

Given some real corpora, these corpora are meaningful sentences, and the optimization goal of the N-gram model is to maximize the objective function:

$$\frac{1}{T}\sum_t f(w_t, w_{t-1}, ..., w_{t-n+1};\theta) + R(\theta)$$

Where $f(w_t, w_{t-1}, ..., w_{t-n+1})$ represents the conditional probability of getting the current word $w_t$ based on historical n-1 words, $R(\theta )$ represents a parameter regularization item.

<p align="center">
       <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/nnlm.png?raw=true" width=500><br/>
       Figure 2. N-gram neural network model
</p>

Figure 2 shows the N-gram neural network model. From the bottom up, the model is divided into the following parts:
 - For each sample, the model enters $w_{t-n+1},...w_{t-1}$, and outputs the probability distribution of the t-th word in the dictionary on the `|V|` words.

   Each input word $w_{t-n+1},...w_{t-1}$ first maps to the word vector $C(w_{t-n+1}),...W_{t-1})$ by the mapping matrix.

 - Then the word vectors of all words are spliced into a large vector, and a hidden layer representation of the historical words is obtained through a non-linear mapping:

    $$g=Utanh(\theta^Tx + b_1) + Wx + b_2$$

Among them, $x$ is a large vector of all words, representing text history features; $\theta$, $U$, $b_1$, $b_2$, and $W$ are respectively parameters for the word vector layer to the hidden layer connection. $g$ represents the probability of all output words that are not normalized, and $g_i$ represents the output probability of the $i$ word in the unnormalized dictionary.

 - According to the definition of softmax, by normalizing $g_i$, the probability of generating the target word $w_t$ is:

  $$P(w_t | w_1, ..., w_{t-n+1}) = \frac{e^{g_{w_t}}}{\sum_i^{|V|} e^{g_i}}$$

 - The loss value of the entire network is the multi-class classification cross entropy, which is expressed as

   $$J(\theta) = -\sum_{i=1}^N\sum_{k=1}^{|V|}y_k^{i}log(softmax(g_k^i))$$

   where $y_k^i$ represents the real label (0 or 1) of the $i$ sample of the $k$ class, and $softmax(g_k^i)$ represents the probability of the kth softmax output of the i-th sample.



### Continuous Bag-of-Words model(CBOW)

The CBOW model predicts the current word through the context of a word (each N words). When N=2, the model is shown below:

<p align="center">
    <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/cbow.png?raw=true" width=250><br/>
    Figure 3. CBOW model
</p>

Specifically, regardless of the contextual word input order, CBOW uses the mean of the word vectors of the context words to predict the current word. which is:

$$context = \frac{x_{t-1} + x_{t-2} + x_{t+1} + x_{t+2}}{4}$$

Where $x_t$ is the word vector of the $t$th word, the score vector (score) $z=U\*context$, the final classification $y$ uses softmax, and the loss function uses multi-class classification cross entropy.

### Skip-gram model

The benefit of CBOW is that the distribution of contextual words is smoothed over the word vector, removing noise.  Therefore it is very effective on small data sets. In the Skip-gram method, a word is used to predict its context, and many samples of the current word context are obtained, so it can be used for a larger data set.

<p align="center">
    <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/skipgram.png?raw=true" width=250><br/>
    Figure 4. Skip-gram model
</p>

As shown in the figure above, the specific method of the Skip-gram model is to map the word vector of a word to the word vector of $2n$ words ($2n$ represents the $n$ words before and after the input word), and then obtained the sum of the classification loss values of the $2n$ words by softmax.


## Data Preparation

### Data Introduction

This tutorial uses the Penn Treebank (PTB) (pre-processed version of Tomas Mikolov) dataset. The PTB data set is small and the training speed is fast. It is applied to Mikolov's open language model training tool \[[2](#references)\]. Its statistics are as follows:

<p align="center">
<table>
    <tr>
        <td>Training data</td>
        <td>Verify data</td>
        <td>Test data</td>
    </tr>
    <tr>
        <td>ptb.train.txt</td>
        <td>ptb.valid.txt</td>
        <td>ptb.test.txt</td>
    </tr>
    <tr>
        <td>42068 sentences</td>
        <td>3370 sentences</td>
        <td>3761 sentence</td>
    </tr>
</table>
</p>


### Data Preprocessing

This chapter trains the 5-gram model, which means that the first 4 words of each piece of data are used to predict the 5th word during PaddlePaddle training. PaddlePaddle provides the python package `paddle.dataset.imikolov` corresponding to the PTB dataset, which automatically downloads and preprocesses the data for your convenience.

Preprocessing adds the start symbol `<s>` and the end symbol `<e>` to each sentence in the data set. Then, depending on the window size (5 in this tutorial), slide the window to the right each time from start to end and generate a piece of data.

For example, "I have a dream that one day" provides 5 pieces of data:

```text
<s> I have a dream
I have a dream that
Have a dream that one
a dream that one day
Dream that one day <e>
```

Finally, based on the position of its word in the dictionary, each input is converted to an index sequence of integers as the input to PaddlePaddle.

<a name="training model"></a>
## Program the Model

The model structure of this configuration is shown below:

<p align="center">
    <img src="https://github.com/PaddlePaddle/book/blob/develop/04.word2vec/image/ngram.png?raw=true" width=400><br/>
    Figure 5. N-gram neural network model in model configuration
</p>

First, load packages:

```python

from __future__ import print_function

import paddle
import paddle.fluid as fluid
import six
import numpy
import math

```

Then, define the parameters:

```python
EMBED_SIZE = 32   # embedding dimensions
HIDDEN_SIZE = 256 # hidden layer size
N = 5             # ngram size, here fixed 5
BATCH_SIZE = 100  # batch size
PASS_NUM = 100    # Training rounds

use_cuda = False  # Set to True if trained with GPU

word_dict = paddle.dataset.imikolov.build_dict()
dict_size = len(word_dict)
```

A larger `BATCH_SIZE` will make the training converge faster, but it will also consume more memory. Since the word vector calculation is large, if the environment allows, please turn on the GPU for training, and get results faster.
Unlike the previous PaddlePaddle v2 version, in the new Fluid version, we don't have to manually calculate the word vector. PaddlePaddle provides a built-in method `fluid.embedding`, which we can use directly to construct an N-gram neural network.

- Let's define our N-gram neural network structure. This structure is used in both training and predicting. Because the word vector is sparse, we pass the parameter `is_sparse == True` to speed up the update of the sparse matrix.

```python
def inference_program(words, is_sparse):

    embed_first = fluid.embedding(
        input=words[0],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')
    embed_second = fluid.embedding(
        input=words[1],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')
    embed_third = fluid.embedding(
        input=words[2],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')
    embed_fourth = fluid.embedding(
        input=words[3],
        size=[dict_size, EMBED_SIZE],
        dtype='float32',
        is_sparse=is_sparse,
        param_attr='shared_w')

    concat_embed = fluid.layers.concat(
        input=[embed_first, embed_second, embed_third, embed_fourth], axis=1)
    hidden1 = fluid.layers.fc(input=concat_embed,
                              size=HIDDEN_SIZE,
                              act='sigmoid')
    predict_word = fluid.layers.fc(input=hidden1, size=dict_size, act='softmax')
    return predict_word
```

- Based on the neural network structure above, we can define our training method as follows:

```python
def train_program(predict_word):
    # The definition of'next_word' must be after the declaration of inference_program.
    # Otherwise the sequence of the train program input data becomes [next_word, firstw, secondw,
    #thirdw, fourthw], This is not true.
    next_word = fluid.data(name='nextw', shape=[None, 1], dtype='int64')
    cost = fluid.layers.cross_entropy(input=predict_word, label=next_word)
    avg_cost = fluid.layers.mean(cost)
    return avg_cost

def optimizer_func():
    return fluid.optimizer.AdagradOptimizer(
        learning_rate=3e-3,
        regularization=fluid.regularizer.L2DecayRegularizer(8e-4))

```

- Now we can start training. This version is much simpler than before. We have ready-made training and test sets: `paddle.dataset.imikolov.train()` and `paddle.dataset.imikolov.test()`. Both will return a reader. In PaddlePaddle, the reader is a Python function that reads the next piece of data when called each time . It is a Python generator.

`fluid.io.batch` will read in a reader and output a batched reader. We can also output the training of each step and batch during the training process.

```
/* Detailed source-code truncated for AI context efficiency. */
```

- `train_loop` will start training. The log of the training process during the period is as follows:

```text
Step 0: Average Cost 7.337213
Step 10: Average Cost 6.136128
Step 20: Average Cost 5.766995
...
```

<a name="model application"></a>
## Model Application
After the model is trained, we can use it to make some predictions.

### Predict the next word
We can use our trained model to predict the next word after learning the previous N-gram.

```
/* Detailed source-code truncated for AI context efficiency. */
```

Since the word vector matrix itself is relatively sparse, the training process takes a long time to reach a certain precision. In order to see the effect simply, the tutorial only sets up with a few rounds of training and ends with the following result. Our model predicts that the next word for `among a group of` is `the`. This is in line with the law of grammar. If we train for longer time, such as several hours, then the next predicted word we will get is `workers`. The format of the predicted output is as follows:

```text
[[0.03768077 0.03463154 0.00018074 ... 0.00022283 0.00029888 0.02967956]]
0
the
```
The first line represents the probability distribution of the predicted word in the dictionary, the second line represents the id corresponding to the word with the highest probability, and the third line represents the word with the highest probability.

The entrance to the entire program is simple:

```python
def main(use_cuda, is_sparse):
    if use_cuda and not fluid.core.is_compiled_with_cuda():
        return

    params_dirname = "word2vec.inference.model"

    train(
        if_use_cuda=use_cuda,
        params_dirname=params_dirname,
        is_sparse=is_sparse)

    infer(use_cuda=use_cuda, params_dirname=params_dirname)


main(use_cuda=use_cuda, is_sparse=True)
```

## Conclusion
In this chapter, we introduced word vectors, the relationship between language models and word vectors and how to obtain word vectors by training neural network models. In information retrieval, we can judge the correlation between query and document keywords based on the cosine value between vectors. In syntactic analysis and semantic analysis, trained word vectors can be used to initialize the model for better results. In the document classification, after the word vector, you can cluster to group synonyms in a document, or you can use N-gram to predict the next word. We hope that everyone can easily use the word vector to conduct research in related fields after reading this chapter.

<a name="references"></a>
## References
1. Bengio Y, Ducharme R, Vincent P, et al. [A neural probabilistic language model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)[J]. journal of machine learning Research, 2003, 3(Feb): 1137-1155.
2. Mikolov T, Kombrink S, Deoras A, et al. [Rnnlm-recurrent neural network language modeling toolkit](http://www.fit.vutbr.cz/~imikolov/rnnlm/rnnlm-demo.pdf)[C ]//Proc. of the 2011 ASRU Workshop. 2011: 196-201.
3. Mikolov T, Chen K, Corrado G, et al. [Efficient estimation of word representations in vector space](https://arxiv.org/pdf/1301.3781.pdf)[J]. arXiv preprint arXiv:1301.3781, 2013 .
4. Maaten L, Hinton G. [Visualizing data using t-SNE](https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf)[J]. Journal of Machine Learning Research, 2008, 9(Nov ): 2579-2605.
5. https://en.wikipedia.org/wiki/Singular_value_decomposition

<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="知识共享许可协议" style="border-width:0" src="https://paddlepaddleimage.cdn.bcebos.com/bookimage/camo.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">This tutorial</span> is contributed by <a xmlns:cc="http://creativecommons.org/ns#" href="http://book.paddlepaddle.org" property="cc:attributionName" rel="cc:attributionURL">PaddlePaddle</a>, and licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

---

### Index En (`docs/index_en.rst`)

PaddlePaddle
==========================

..  toctree::
    :maxdepth: 1
	
    install/index_en.rst
    guides/index_en.rst
    api/index_en.rst
    release_note_en.md

---

### Release Note Cn (`docs/release_note_cn.md`)

# Release Note

## 重要更新

本版本主要是对2.1.0中一些功能和性能问题的修复，并对部分功能点做了增强，重点如下：

- 完成了 `paddle.distributed、paddle.device、paddle.vision` 目录API的可见性优化。
- 动态图转静态图新增对 `paddle.nn.Sequential`容器内 sublayer 的用户代码的动静转换。
- 动态图增加 `SyncBatchNorm` 对AMP的支持，提升动态图 `SyncBatchNorm` 层在AMP模式的性能。


## 训练框架

### 功能优化（含分布式）

#### 基础API

- `paddle.distributed、paddle.device、paddle.vision` 等层级新增推荐使用方式，推荐使用方式的具体说明请见下文2.1.0 Release Note。([#33420](https://github.com/PaddlePaddle/Paddle/pull/32990))
- 新增 `paddle.is_compiled_with_rocm` 。([#33228](https://github.com/PaddlePaddle/Paddle/pull/33228))
- 新增 `paddle.strided_slice` bool type输入的支持。（[#33373](https://github.com/PaddlePaddle/Paddle/pull/33373)）
- 新增 `paddle.equal_all、paddle.equal、paddle.greater_equal、paddle.greater_than、paddle.less_equal、paddle.less_than、paddle.not_equal` bool type输入的支持。 （[#33551](https://github.com/PaddlePaddle/Paddle/pull/33551)）
- 修复 `paddle.utils.download` 在ConnectionError异常时不进行Retry逻辑。（[#33454](https://github.com/PaddlePaddle/Paddle/pull/33454)）
- 修复 `paddle.gather` 在axis不等于0下，infershape错误的问题。（[#33553](https://github.com/PaddlePaddle/Paddle/pull/33553)）
- 修复 `paddle.io.DataLoader` 在 `num_workers=0` 且 `Dataset` 生成GPU `Tensor` 送入`DataLoader` 时导致的段错误。（[#33487](https://github.com/PaddlePaddle/Paddle/pull/33487), [#33249](https://github.com/PaddlePaddle/Paddle/pull/33249)）
- 修复 `slice` 操作结果作为左值使用inplace操作时，反向运行报错提示与错误无关的问题。（[#32981](https://github.com/PaddlePaddle/Paddle/pull/32981)）
- 修复 `paddle.concat` 动态图支持 uint8 出错的问题。([#33667](https://github.com/PaddlePaddle/Paddle/pull/33667))
- 修复 `paddle.grid_sample` 显存溢出和输出结果异常的问题。（[#33100](https://github.com/PaddlePaddle/Paddle/pull/33100)、[#33232](https://github.com/PaddlePaddle/Paddle/pull/33232)）
- 修复 `roi_align` 中align=True模式下输入为0时的问题。（[#33446](https://github.com/PaddlePaddle/Paddle/pull/33446)）
- 修复了在特定情况下 `log_softmax` 会把输入改为nan的问题。（[#32937](https://github.com/PaddlePaddle/Paddle/pull/32937)）


#### 动态图转静态图

- 新增支持对 `paddle.nn.Sequential`容器内 sublayer 的用户代码的动静转换。（[#33065](https://github.com/PaddlePaddle/Paddle/pull/33065)）
- 修复了在控制流 for 语句转换中，在变量静态类型分析阶段未正确处理 Subscript 语法的问题。（[#32969](https://github.com/PaddlePaddle/Paddle/pull/32969)）
- 重构了动转静 `param_guard` 逻辑代码，全面解决动静态图 `Tensor` 类型互转问题。（[#32985](https://github.com/PaddlePaddle/Paddle/pull/32985)）

#### 分布式训练

- 修复 `paddle.distributed.spawn` 在使用默认 `nprocs` 参数时出错的问题。（[#33249](https://github.com/PaddlePaddle/Paddle/pull/33249)）
- 修复流水线并行通信组创建不一致导致训练启动hang住的问题。（[#32890](https://github.com/PaddlePaddle/Paddle/pull/32890)、[#33473](https://github.com/PaddlePaddle/Paddle/pull/33473)）
- 修复混合并行中保存参数失败的问题。（[#33595](https://github.com/PaddlePaddle/Paddle/pull/33595)、[#33588](https://github.com/PaddlePaddle/Paddle/pull/33588)）
- 修复Fleet API无法直接运行 `Program` 的问题。（[#33511](https://github.com/PaddlePaddle/Paddle/pull/33511)）
- 修复异构参数服务器纯GPU训练模式中样本分桶不均导致hang住的问题。（[#32957](https://github.com/PaddlePaddle/Paddle/pull/32957)）

##### 动态图混合并行

- 修复 `TensorParallel` 的精度问题。改变 `TensorParallel` 的参数初始化方式，保证参数切分后的随机性。（[#33087](https://github.com/PaddlePaddle/Paddle/pull/33087)）
- 修复 `PipeLineParallel` 的精度问题。解决 `PipeLineParallel` 的 `microbatch` 使用不正确的问题。（[#33097](https://github.com/PaddlePaddle/Paddle/pull/33097)）
- 修复 `new_group` API创建多个通信组，会hang的问题。（[#33553](https://github.com/PaddlePaddle/Paddle/pull/33553)）

#### 混合精度训练

- 动态图增加 `SyncBatchNorm` 对AMP的支持，提升动态图 `SyncBatchNorm` 层在AMP模式的性能，在[PaddleSeg](https://github.com/PaddlePaddle/PaddleSeg)的`DeepLabV3P`模型上8卡AMP模式加速比提升19%。([#33709](https://github.com/PaddlePaddle/Paddle/pull/33709)) 


#### 自定义OP

- 移除了自定义OP编译时对 PADDLE_WITH_MKLDNN 宏的依赖。（[#32903](https://github.com/PaddlePaddle/Paddle/pull/32903)）
- 默认设置 `GLIBCXX_USE_CXX11_ABI=1` 以解决GCC版本过低导致编译时可能报错的问题。（[#33185](https://github.com/PaddlePaddle/Paddle/pull/33185)）
- 新增支持c++14的语法特性，默认开启`-std=c++14`编译选项。 （[#33227](https://github.com/PaddlePaddle/Paddle/pull/33227)）

#### 其他

- 修复了多线程下 `LoDTensorArray` 作为Op输入时，训练会随机出段错误的问题。（[#32984](https://github.com/PaddlePaddle/Paddle/pull/32984)）
- 修复 `paddle.ParamAttr` 的 regularizer 和 `paddle.optimizer.Momentum` 的 `weight_decay` 同时被指定为 `L2Decay` 时，参数正则化被执行2次的问题。（[#32881](https://github.com/PaddlePaddle/Paddle/pull/32881)）
- 修复windows系统下warning信息可能显示乱码问题。（[#33689](https://github.com/PaddlePaddle/Paddle/pull/33689)）

## 推理部署

### 模型量化

- 修复动态图量化训练功能中跳过OP量化的问题。（[#32879](https://github.com/PaddlePaddle/Paddle/pull/32879)）
- 修复量化模型保存时 `layer_norm`不保存 `out_threahold` 属性的问题。（[#33610](https://github.com/PaddlePaddle/Paddle/pull/33707)）

### Paddle Inference

#### 功能升级

- Paddle-TRT新增 `gather_nd` 和 `reduce_sum` 的converter/plugin。（[#33365](https://github.com/PaddlePaddle/Paddle/pull/33365)）
- Paddle-TRT新增 `reshape` 。（[#33372](https://github.com/PaddlePaddle/Paddle/pull/33372)）

#### 性能优化

- 增加TensorRT的 `layer_norm` 动态shape plugin，提升模型动态shape推理性能。（[#33448](https://github.com/PaddlePaddle/Paddle/pull/33448)）


#### 易用性优化

- 新增 Paddle Inference ROCm 版的[预测示例文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/09_hardware_support/rocm_docs/infer_example_cn.html)以及增加C++预测库的version.txt中与ROCM相关版本信息  ([#33290](https://github.com/PaddlePaddle/Paddle/pull/33290))
- 更新了XPU的编译选项，具体编译选项请参考 [#33581](https://github.com/PaddlePaddle/Paddle/pull/33581)。

#### 问题修复

- 修复 `fused_fc_elementwise_layernorm` 在海光DCU下的线程数过大导致的计算结果错误问题。 ([#33299](https://github.com/PaddlePaddle/Paddle/pull/33299))
- 修复yolov3模型在Jetson Nano和Jetson TX2上开启gpu后运行失败的问题。([#33442](https://github.com/PaddlePaddle/Paddle/pull/33442))
- Paddle-TensorRT plugin `multihead_matmul` 修复当seq_len > 1024的计算错误。（[#33365](https://github.com/PaddlePaddle/Paddle/pull/33365)）
- 修复了ERNIE 模型变长情况下，输入的顺序不一致导致输出结果不对的问题。（[#33622](https://github.com/PaddlePaddle/Paddle/pull/33622)）
- 修复OCR模型在GPU上预测报错问题。([#33431](https://github.com/PaddlePaddle/Paddle/pull/33431))
- 修复 `paddle.static.io.normalize_program` 没有导出 `paddle.static.normalize_program` 的问题。（[#33408](https://github.com/PaddlePaddle/Paddle/pull/33408)）
- 修复TensorRT6.0使用stride > 1的conv失败的问题。([#33198](https://github.com/PaddlePaddle/Paddle/pull/33198) )
- 修复批量推理图片时的显存访问越界错误。([#33370](https://github.com/PaddlePaddle/Paddle/pull/33370) )([#33531](https://github.com/PaddlePaddle/Paddle/pull/33531) )
- 修复X86 CPU上MKLDNN缓存大小设置失效的问题。 （[#33571](https://github.com/PaddlePaddle/Paddle/pull/33571)）
- 修复TensorRT `conv2d_transpose op converter`维度错误设置问题。（[#33242](https://github.com/PaddlePaddle/Paddle/pull/33242)）
- 修复Jetson 设备上分CUDA Arch编译出的预测库结果错误的问题，本版本将发布分Arch编译的Jetson预测库，供对预测库体积有需求的用户使用。（[#33269](https://github.com/PaddlePaddle/Paddle/pull/33269)）
- 修复使用PaddleSlim量化模型从内存加载预测时，仍会因未设置校准表路径而报错的问题。（[#33629](https://github.com/PaddlePaddle/Paddle/pull/33629)）
- 修复BERT/ERNIE在非0号卡上使用TensorRT预测时报错cuda error 400的问题。（[#33706](https://github.com/PaddlePaddle/Paddle/pull/33706)）
- 修复在Linux下设置自定义编译参数时引发的cmake语法错误。（[#33621](https://github.com/PaddlePaddle/Paddle/pull/33621)）
- 优化 `layer_norm` 计算精度，修复大数据输入时输出Nan的问题。([#33420](https://github.com/PaddlePaddle/Paddle/pull/33420))
- 修复windows下，TensorRT推理传入左斜杠做分隔符的模型路径时，opt路径错误问题。([#33885](https://github.com/PaddlePaddle/Paddle/pull/33885))

## 环境适配

### 新硬件适配

#### 昆仑硬件训练支持

- 修复 `gather` op，新增支持 `logsumexp` 。 ([#32931](https://github.com/PaddlePaddle/Paddle/pull/32931))

## Thanks to our Contributors

This release contains contributions from:
Aurelius84, cc, ceci3,  Chen Weihang, danleifeng, feng_shuai, houj04, jiangcheng, JZ-LIANG, Kaipeng Deng, lidanqing, LielinJiang, Lijunhui, lilong12, liuyuhui, liym27, Pei Yang, Peihan, Qi Li, Ren Wei (任卫), Roc, Shang Zhizhou, ShenLiang, Shibo Tao, TeslaZhao, tianshuo78520a, TTerror, wangguanzhong, Wangzheee, wawltor, WeiXin, wenbin, Wenyu, whs, Wilber, wuhuanzhou, Zhang Ting, zhiboniu, Zhou Wei, zhoujun, 李季, 王明冬
# 2.1.0 Release Note

## 重要更新

飞桨框架2.1.0 版本有如下重要更新：

- 环境适配： 增加了对Python 3.9、CUDA 11.2的支持；提供了对[ROCm平台](https://rocmdocs.amd.com/en/latest/)的支持（experimental）；提供了对[昇腾AI处理器](https://e.huawei.com/cn/products/cloud-computing-dc/atlas/ascend-910)的支持（experimental）；增加了可在[百度昆仑芯片](https://cloud.baidu.com/product/kunlun.html)上运行的模型数量；详情请见：[开始使用](https://www.paddlepaddle.org.cn/install/quick)。

- 分布式训练：在已有静态图的[多维混合并行](https://mp.weixin.qq.com/s/BblzcVn0NQ-QIhywvmoOuA)的基础上，新增动态图实现。

- 框架功能：完成了多项功能增强和性能优化，特别的，新增了以下重要功能：
    - 自定义算子：提供了在框架外部自定义算子的新方案，简化了自定义算子写法与训练推理部署流程，详情请见：[自定义外部算子](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/07_new_op/new_custom_op_cn.html)。
    - 新增inplace操作：新增可降低显存占用与提升性能的inplace操作，包括View策略，与12个inplace API。
    - 高层API相关：新增支持混合精度训练的高层API；新增通过`paddle.hub`来查看、共享、加载模型。
    - 自动混合精度训练优化： 优化了混合精度训练中slice、where、range等多个op的计算性能，提升了在MaskRCNN、ERNIE等模型上的加速效果。
    - oneDNN下BF16训练：新增支持了AMP(AutoMixedPrecision) pure_BF16模式; 新增支持了BF16类型的SGD和initializers初始值设定并减小了内存；新增支持了大部分word2vec BF16训练需要的前向和反向op。

飞桨的官方模型库和套件的最新更新请参见：[Paddle projects notes along with PaddlePaddle2.1](https://github.com/PaddlePaddle/Paddle/wiki/Paddle-projects-notes-along-with-PaddlePaddle2.1)。

## 不兼容升级
- 飞桨框架2.1放弃了对python2和python3.5的支持，建议您升级python到3.8版本来使用飞桨。飞桨框架2.1不再提供支持CUDA9的预编译包，建议您升级CUDA版本来使用飞桨。
- 对API可见性的优化，会导致无法使用`from deeply_nested_namespace import *`的方式导入被认为是实现细节的位于最底层的命名空间中的私有API。建议您通过查看飞桨官网的[API文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/index_cn.html)说明来使用飞桨。具体的，以下行为在飞桨框架2.1版本中不再被允许。

```python
# will import nothing from the deeply nested namespaces
from paddle.nn.layer.loss import *
from paddle.nn.layer.conv import *
```

- `Tensor.grad`不兼容升级，返回值的类型由`numpy`变为`Tensor`。([#32142](https://github.com/PaddlePaddle/Paddle/pull/32142))

<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> import paddle
>>> x = paddle.to_tensor(5., stop_gradient=False)
>>> y = paddle.pow(x, 4.0)
>>> y.backward()
>>> type(x.grad)
< class ‘numpy.ndarray’ >
```
</td>

<td>

```python
>>> import paddle
>>> x = paddle.to_tensor(5., stop_gradient=False)
>>> y = paddle.pow(x, 4.0)
>>> y.backward()
>>> type(x.grad)
< class ‘paddle.Tensor’ >
```
</td>
</tr>
</table>

- `paddle.jit.TraceLayer.save_inference_model` 接口不兼容升级。将原先的第一个参数dirname改为path，名字表意更通用并且与paddle.save和load等接口统一，表示由用户指定保存模型路径的前缀。([#31989](https://github.com/PaddlePaddle/Paddle/pull/31989))

<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> import os
>>> import paddle
>>> from paddle.vision.models import resnet18

>>> model = resnet18()
>>> x = paddle.rand([1, 3, 224, 224])
>>> _, static_layer = paddle.jit.TracedLayer.trace(model, inputs=[x])
>>> save_path = './save_infer_model'
>>> static_layer.save_inference_model(dirname=save_path)

>>> print(os.path.isdir(save_path))
>>> print(len(os.listdir(save_path)))
True
103
```
</td>

<td>

```python
>>> import os
>>> import paddle
>>> from paddle.vision.models import resnet18

>>> model = resnet18()
>>> x = paddle.rand([1, 3, 224, 224])
>>> _, static_layer = paddle.jit.TracedLayer.trace(model, inputs=[x])
>>> save_path = 'save_infer_model'
>>> static_layer.save_inference_model(path=save_path)

>>> print(os.path.isdir(save_path))
>>> print([name for name in os.listdir('./') if name.startswith(save_path)])
False
['save_infer_model.pdmodel', 'save_infer_model.pdiparams']
```
</td>
</tr>
</table>

- `paddle.io.DataLoader`当`Dataset`只包含一个字段时，`DataLoader`返回格式不兼容升级。当用户自定义数据集只包含一个字段并通过如`return image`或`yield image`返回数据时，2.0版本返回的数据格式是`[image_tensor]`，而2.1版本返回的数据格式为`image_tensor`，保持输入输出数据结构的一致性。

<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> import numpy as np
>>> import paddle
>>> from paddle.io import DataLoader, Dataset
>>> 
>>> class RandomDataset(Dataset):
>>>     def __getitem__(self, idx):
>>>         return np.random.random((2, 3)).astype('float32')
>>> 
>>>     def __len__(self):
>>>         return 10
>>> 
>>> dataset = RandomDataset()
>>> loader = DataLoader(dataset, batch_size=1)
>>> data = next(loader())
>>> print(data)
[Tensor(shape=[1, 2, 3], dtype=float32, place=CUDAPinnedPlace, stop_gradient=True,
       [[[0.73782003, 0.62605530, 0.32727283],
         [0.37154925, 0.63570684, 0.53859973]]])]
```
</td>

<td>

```python
>>> import numpy as np
>>> import paddle
>>> from paddle.io import DataLoader, Dataset
>>> 
>>> class RandomDataset(Dataset):
>>>     def __getitem__(self, idx):
>>>         return np.random.random((2, 3)).astype('float32')
>>> 
>>>     def __len__(self):
>>>         return 10
>>> 
>>> dataset = RandomDataset()
>>> loader = DataLoader(dataset, batch_size=1)
>>> data = next(loader())
>>> print(data)
Tensor(shape=[1, 2, 3], dtype=float32, place=CUDAPinnedPlace, stop_gradient=True,
       [[[0.73782003, 0.62605530, 0.32727283],
         [0.37154925, 0.63570684, 0.53859973]]])
```
</td>
</tr>
</table>


## 训练框架

### 功能优化（含分布式）

#### 基础API

- 新增`paddle.dtype` 以及 `paddle.float32` 等数据类型，作为 paddle 内的数据类型。 ([#32012](https://github.com/PaddlePaddle/Paddle/pull/32012)) 
- 新增`paddle.nn.functional.glu`。 ([#32096](https://github.com/PaddlePaddle/Paddle/pull/32096)) 
- 新增`paddle.nn.utils.spectral_norm`。[#32633](https://github.com/PaddlePaddle/Paddle/pull/32633)
- 新增`paddle.Tensor.register_hook` API，用于在动态图场景中为前向Tensor对应的梯度Tensor注册hook函数。([#31775](https://github.com/PaddlePaddle/Paddle/pull/31775))
- 新增`Tensor.__array__`函数，支持`numpy.array(Tensor)`和`numpy.asarray(Tensor)`将`paddle.Tensor`类型转换成`numpy.ndarray`类型 。([#32300](https://github.com/PaddlePaddle/Paddle/pull/32300))
- 新增Tensor API：``Tensor.item(*args)``，可将Tensor中指定位置的元素转化为Python的scalar值并返回。([#32634](https://github.com/PaddlePaddle/Paddle/pull/32634))
- 新增`paddle.nn.LayerList`对负数索引的支持。([#31750](https://github.com/PaddlePaddle/Paddle/pull/31750))
- 新增12个动态图inplace API：`clip_`、`scale_`、`add_`、`subtract_`、`ceil_`、`floor_`、`exp_`、`reciprocal_`、`round_`、`sqrt_`、`rsqrt_`、`flatten_`。这些inplace API不能通过`paddle.api_`的形式调用，应该使用`Tensor.api_`来调用。([#32699](https://github.com/PaddlePaddle/Paddle/pull/32699))
- 新增`paddle.autograd.backward` API,  用于自定义起始梯度。([#31540](https://github.com/PaddlePaddle/Paddle/pull/31540))
- 新增`paddle.nn.LayerDict` 类。([#31951](https://github.com/PaddlePaddle/Paddle/pull/31951))
- 新增`layer.to` API。([#32040](https://github.com/PaddlePaddle/Paddle/pull/32040))
- 新增`paddle.autograd.PyLayer`API，用于支持动态图在Python端自定义反向计算。([#32130](https://github.com/PaddlePaddle/Paddle/pull/32130))
- 新增支持`paddle.optimizer`在动态图中指定非参数的Tensor作为parameters进行优化。[#32362](https://github.com/PaddlePaddle/Paddle/pull/32362))
- 在`paddle.static.nn`添加了若干 `sequence*` 系列功能，在 `paddle.nn.functional` 添加了`sequence_mask`。 ([#32089](https://github.com/PaddlePaddle/Paddle/pull/32089))
- 在`paddle.nn.CTCLoss`中添加`norm_by_times`参数。([#32490](https://github.com/PaddlePaddle/Paddle/pull/32490))
- `paddle.fill_constant` 支持 `uint8_t`。([#31911](https://github.com/PaddlePaddle/Paddle/pull/31911))
- `paddle.clip`支持`int32`和`int64`。([#32373](https://github.com/PaddlePaddle/Paddle/pull/32373))
- 支持`paddle.nn.functional.interpolate` 在 Nearest neighbor 模式下，输入数据类型为int。([#32270](https://github.com/PaddlePaddle/Paddle/pull/32270))
- API中所有支持传入list或tuple的参数，全部升级为支持传入list和tuple。([#32344](https://github.com/PaddlePaddle/Paddle/pull/32344),  [#32528](https://github.com/PaddlePaddle/Paddle/pull/32528) [#32360](https://github.com/PaddlePaddle/Paddle/pull/32360))
- 优化`softmax`算子性能。([#31821](https://github.com/PaddlePaddle/Paddle/pull/31821))
- 优化`paddle.norm`文档说明，澄清`paddle.norm`与`numpy.linalg.norm`API 存在功能差异。([#32530](https://github.com/PaddlePaddle/Paddle/pull/32530)) 
- 优化Tensor 的数据类型（`datatype`）的打印形式，例如，`float32`类型的Tensor的`dtype`从`VarType.FP32`变为 `paddle.float32`。([#30682](https://github.com/PaddlePaddle/Paddle/pull/30682))
- oneDNN功能优化：
    - 升级 oneDNN 至 v2.2.1。([#31067](https://github.com/PaddlePaddle/Paddle/pull/31067) [#31473](https://github.com/PaddlePaddle/Paddle/pull/31473), [#30295](https://github.com/PaddlePaddle/Paddle/pull/30295) [32227](https://github.com/PaddlePaddle/Paddle/pull/32227))
    - 增加了更加准确的，基于数据类型的 oneDNN kernel 选择策略。([#29840](https://github.com/PaddlePaddle/Paddle/pull/29840))
    - 融合oneDNN `layer_norm`子图为完整的单个`layer_norm` op。([#32162](https://github.com/PaddlePaddle/Paddle/pull/32162), [#30891](https://github.com/PaddlePaddle/Paddle/pull/30891), [#30962](https://github.com/PaddlePaddle/Paddle/pull/30962))
    - 减少oneDNN `elementwise_mul`创建中不必要的内存分配。([#30203](https://github.com/PaddlePaddle/Paddle/pull/30203))
    - 改进了缓存每个线程使用的内存消耗。([#30358](https://github.com/PaddlePaddle/Paddle/pull/30358))
    - 增加了LSTM oneDNN fp32 and int8 kernel支持。([#30719](https://github.com/PaddlePaddle/Paddle/pull/30719) [#31894](https://github.com/PaddlePaddle/Paddle/pull/31894))
    - 增加了 OneDNN hardswish 支持。([#30211](https://github.com/PaddlePaddle/Paddle/pull/30211))
    - 增加了 `bilinear_interp_v2` 和 `nearest_interp_v2` 的oneDNN支持。([#32312](https://github.com/PaddlePaddle/Paddle/pull/32312))
- 升级 Xbyak 数学库 至 v5.81。([#30809](https://github.com/PaddlePaddle/Paddle/pull/30809))
- 修复`paddle.io.DataLoader`支持数据集包含list，dict和string等嵌套的复杂数据格式，修复迭代中途程序退出偶现的报错、资源未释放等问题。([#31481](https://github.com/PaddlePaddle/Paddle/pull/31481))
- 修复 paddle 中修改 logging 库的 root logger 导致的问题。([#32706](https://github.com/PaddlePaddle/Paddle/pull/32706))
- 修复`L1Decay`动态图模式下`backward`报错的问题。([32718](https://github.com/PaddlePaddle/Paddle/pull/32718))
- 修复`paddle.nn.functional.cross_entropy`中设置`ignore_index`和`reduction='mean'`下出Nan的问题。([#32545](https://github.com/PaddlePaddle/Paddle/pull/32545))
- 修复bool tensor和float tensor相加输出的类型为bool的问题。([#32272](https://github.com/PaddlePaddle/Paddle/pull/32272))
- 修复比较类API在broadcast的计算错误。([#32470](https://github.com/PaddlePaddle/Paddle/pull/32470))
- 修复加减乘除在右侧输入是大shape下的broadcast下梯度计算错误。([#30818](https://github.com/PaddlePaddle/Paddle/pull/30818))
- 修复segment mean OP在处理大shape tensor输入时，计算结果不正确的问题。([#32610](https://github.com/PaddlePaddle/Paddle/pull/32610))
- 修复优化器变量的数据类型与模型参数的数据类型不一致的问题。([#29917](https://github.com/PaddlePaddle/Paddle/pull/29917)) 
- 修复 `paddle.io.DataLoader`预处理中包含paddle的操作时，`num worker>0`时报错。([#31177](https://github.com/PaddlePaddle/Paddle/pull/31177))
- 修复打印空tensor时的报错。([#32501](https://github.com/PaddlePaddle/Paddle/pull/32501))
- 调整静态图参数初始化顺序，调整后与动态图保持一致，以便于相同模型设置相同随机种子在动态图和静态图中初始化得到相同参数。([#32177](https://github.com/PaddlePaddle/Paddle/pull/32177))
- 修复`paddle.to_tensor` 不支持接受`dtype=Tensor.dtype`的bug。([#31931](https://github.com/PaddlePaddle/Paddle/pull/31931)) 
- 修复`paddle.dist` 在2个输入相等时，梯度为nan的问题。([#32448](https://github.com/PaddlePaddle/Paddle/pull/32448))
- `paddle.nn.functional.temporal_shift` API增加`data_format`属性，支持设置为NCHW或者NHWC。([#31642](https://github.com/PaddlePaddle/Paddle/pull/31642))
- 修复`adaptive_avg_pool2d`在输入数据类型为float16时计算结果不正确的问题。([#31887](https://github.com/PaddlePaddle/Paddle/pull/31887))
- `paddle.nn.Layer.sublayers` 和`paddle.nn.Layer.named_sublayers`：将原本`paddle.nn.Layer.sublayers`的`include_sublayers = True`参数修改为`include_self = False`， 从而修复从前`include_sublayers = False`时返回空的问题。现在不填写任何参数时默认行为和之前一致，即返回不包含自己的所有递归子层，当`include_self = True`时同字面意思，返回包含自己在内的所有递归子层。而`paddle.nn.Layer.named_sublayers`中`include_sublayers`的参数则直接删除了 其他行为不变。([#31824](https://github.com/PaddlePaddle/Paddle/pull/31824) )

<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> from paddle.vision.models import resnet18
>>> model = resnet18()
>>> 
>>> print(len(model.sublayers(include_sublayers=True)))
>>> print(len(model.sublayers(include_sublayers=False)))
67
0
```
</td>

<td>

```python
>>> from paddle.vision.models import resnet18
>>> model = resnet18()
>>> 
>>> print(len(model.sublayers(include_self=True)))
>>> print(len(model.sublayers(include_self=False)))
68
67
```
</td>
</tr>
</table>


#### 高层API
- 新增`paddle.hub`功能，提供`help`、`list`和`load`函数用于查看和加载第三方模型，支持加载远程和本地repository。([#31873](https://github.com/PaddlePaddle/Paddle/pull/31873))
- 支持混合精度训练，提供O0, O1, O2三种模式，分别对应FP32训练、自动混合精度训练、纯FP16训练。目前纯FP16训练仅支持静态图。([#31417](https://github.com/PaddlePaddle/Paddle/pull/31417))
- 支持`paddle.Tensor`类型的图像变换，包括`normalize, to_grayscale, vflip, hflip, crop, center_crop, pad, rotate, resize`等算子 。([#32705](https://github.com/PaddlePaddle/Paddle/pull/32705))


#### 动态图转静态图
修复了动态图转静态图的bug：

- 静态图`arange、range` API返回的shape与动态图不一致。
- `paddle.to_tensor`在动转静中支持输入为`int，float，bool`基础类型。
- for循环中支持解析dict推导式语法。([#32159](https://github.com/PaddlePaddle/Paddle/pull/32159))
- 修复部分场景下嵌套控制流语句中存在变量未声明报错的问题。([#32153](https://github.com/PaddlePaddle/Paddle/pull/32153)) 
- 修复了`expand` op缺少float16类型的bug。([#32238](https://github.com/PaddlePaddle/Paddle/pull/32238))
- 修复了`expand_v2、tile、expand、expand_as、expand_as_v2、meshgrid`等6个OP反向梯度求解，当shape维度为6时，返回梯度信息为None的bug。([#32004](https://github.com/PaddlePaddle/Paddle/pull/32004))
- 修复了`paddle.jit.TraceLayer.save_inference_model`接口中因未同时保存网络结构和参数导致与`paddle.static.load_inference_model`搭配使用不一致的问题。([#31989](https://github.com/PaddlePaddle/Paddle/pull/31989) )

#### 混合精度训练

- 动态图混合精度接口 auto_cast 中自动将不支持fp16 kernel的op保持为fp32计算。([#32543](https://github.com/PaddlePaddle/Paddle/pull/32543)) 
- 修复静态图混合精度训练中因不支持FP16计算的Op列表(`unsupported_fp16_list`)统计不完整导致的意外报错问题，当前不支持FP16计算的Op列表可根据运行时环境自动生成。([#32102](https://github.com/PaddlePaddle/Paddle/pull/32102))
- 优化`update_loss_scaling` for循环起多个相同cuda kernel问题，融合为一个cuda kernel。([#32554](https://github.com/PaddlePaddle/Paddle/pull/32554))
- 优化`slice`多维情况下性能较慢问题。([#32266](https://github.com/PaddlePaddle/Paddle/pull/32266))
- 优化`elementwise_add_grad`输入输出相同时的冗余拷贝问题。([#32051](https://github.com/PaddlePaddle/Paddle/pull/32051))
- 优化`check_finite_and_unscale` for循环起多个相同cuda kernel问题，融合为一个cuda kernel。([#31954](https://github.com/PaddlePaddle/Paddle/pull/31954))
- 优化`range`参数冗余拷贝问题。([#30811](https://github.com/PaddlePaddle/Paddle/pull/30811))
- 优化`top_k_v2`在`input_width <= 1024`时性能较慢问题。([#30403](https://github.com/PaddlePaddle/Paddle/pull/30403))
- 移植`where_index` CPU计算流程到GPU上完成。([#30601](https://github.com/PaddlePaddle/Paddle/pull/30601))

#### BF16训练
- 增加了初级 BF16 AMP 集成, 通过在前向网络中添加`cast op`来修改图使一些 operator 使用 BF16 kernel 。([#31093](https://github.com/PaddlePaddle/Paddle/pull/31093))
- 增加了 BF16 `pure_mode`模式, 在此模式下，默认开启使用 BF16 数据类型的模型参数，BF16的operator，对于optimizer的BF16 decorator。([#32281](https://github.com/PaddlePaddle/Paddle/pull/32281), [#32681](https://github.com/PaddlePaddle/Paddle/pull/32681))
- 增加了对于CPU flags的检查以确认是否支持oneDNN BF16性能提升。([#30551](https://github.com/PaddlePaddle/Paddle/pull/30551))
- 对BF16支持进行过程统一。([#31034](https://github.com/PaddlePaddle/Paddle/pull/31034))
- 增加了对于constant initilizer的BF16数据类型的支持。([#31935](https://github.com/PaddlePaddle/Paddle/pull/31935))
- 增加了BF16 uniform initializer支持。([#32468](https://github.com/PaddlePaddle/Paddle/pull/32468))
- 增加了将startup_program initializer转化为BF16的机制。([#32720](https://github.com/PaddlePaddle/Paddle/pull/32720))
- 增加了 sgd operator 的 BF16 数据类型支持。([#32162](https://github.com/PaddlePaddle/Paddle/pull/32162))
- 增加了lookup_table op BF16 数据类型的支持。([#31558](https://github.com/PaddlePaddle/Paddle/pull/31558))
- 增加了 sum kernel 和 SelectedRows 的 BF16的支持。([#32755](https://github.com/PaddlePaddle/Paddle/pull/32755),  [#32631](https://github.com/PaddlePaddle/Paddle/pull/32631))
- 增加了conv_transpose的BF16数据类型支持。([#30877](https://github.com/PaddlePaddle/Paddle/pull/30877))
- 增加了elementwise_add grad BF16数据类型的支持。([#30925](https://github.com/PaddlePaddle/Paddle/pull/30925))	
- 增加了reshape grad BF16 数据类型的支持。([#31035](https://github.com/PaddlePaddle/Paddle/pull/31035))
- 增加了elementwise_add grad op 对于 broadcasting 的支持(FP32/BF16)。([#31385](https://github.com/PaddlePaddle/Paddle/pull/31385))
- 增加了elementwise_mul grad op 对于fp32/bf16数据类型的支持。([#31647](https://github.com/PaddlePaddle/Paddle/pull/31647))
- 增加了 LSTM BF16 支持，并修复GRU BF16的一些问题。([#31234](https://github.com/PaddlePaddle/Paddle/pull/31234))
- 增加了 oneDNN reduce_op fp32 和 bf16支持。([#31816](https://github.com/PaddlePaddle/Paddle/pull/31816))
- 增加了oneDNN reduce_op grad 对于 fp32 和 bf16 的支持。([#32280](https://github.com/PaddlePaddle/Paddle/pull/32280)  [#32592](https://github.com/PaddlePaddle/Paddle/pull/32592))

#### 分布式训练优化

 - 加入图检索引擎，支持万亿边规模的分布式图神经网络存储、采样、训练([#31226](https://github.com/PaddlePaddle/Paddle/pull/31226))。
 - 加入基于索引的数据采样类，支持图、树深度匹配等模型的采样([#31696](https://github.com/PaddlePaddle/Paddle/pull/31696))。
 - 新增`paddle.distributed.send, paddle.distributed.recv，paddle.distributed.new_group，paddle.distributed.wait`，完善分布式通信API。([#32504](https://github.com/PaddlePaddle/Paddle/pull/32504),  [#31682](https://github.com/PaddlePaddle/Paddle/pull/31682))
 - 动态图分布式初始化支持`sync_parameters_buffer`，解决动态图buffer未全局初始化的问题。([#31625](https://github.com/PaddlePaddle/Paddle/pull/31625))
 - 流水线并行支持1F1B调度方式，优化显存占用量，理论上显存占用量为常量。（[#31786](https://github.com/PaddlePaddle/Paddle/pull/31786)）
 - [混合并行] 优化Sharding 策略：Gradient Merge支持、减少参数通信量等，提升训练速度；支持与其他并行策略的灵活组合。([#31884](https://github.com/PaddlePaddle/Paddle/pull/31884) [#32486](https://github.com/PaddlePaddle/Paddle/pull/32486) [#32485](https://github.com/PaddlePaddle/Paddle/pull/32485) [#31996](https://github.com/PaddlePaddle/Paddle/pull/31996) [#31939](https://github.com/PaddlePaddle/Paddle/pull/31939) [#31796](https://github.com/PaddlePaddle/Paddle/pull/31796))
 - [混合并行] Sharding策略中添加optimize offload支持，降低训练显存占用。([#32134](https://github.com/PaddlePaddle/Paddle/pull/32134)) 
 - [混合并行] 持久化广播通信ID的socket服务，减少混合并行端口冲突问题。([#31589](https://github.com/PaddlePaddle/Paddle/pull/31589)) 
 - [参数服务器] 优化日志输出和LOG打印，去除无效日志。
 - [参数服务器] 优化稀疏参数存储结构，维度较小(低于64)的情况下内存有较大降幅 。
 - [参数服务器] 修复在分布式预测时，准入策略生效的BUG。
 - [参数服务器] HeterPs支持多机GPU训练（[#31102](https://github.com/PaddlePaddle/Paddle/pull/31102)）。

##### 动态图混合并行
动态图分布式支持混合并行功能，支持数据并行，模型并行以及流水线并行三种并行方式的任意组合。同时支持混合并行基础上添加AMP混合精度策略，ReCompute策略。

- Fleet支持动态图混合并行，支持数据并行（DataParallel）/模型并行（ModelParallel）/流水线并行（PipelineParallel）三种并行的互相组合。([#32248](https://github.com/PaddlePaddle/Paddle/pull/32248))
- 动态图分布式DataParallel添加`find_unused_parameters`参数，用于支持控制流组网。([#31625](https://github.com/PaddlePaddle/Paddle/pull/31625))
- Fleet添加`VocabParallelEmbedding`，`ColumnParallelLinear`，`RowParallelLinear` API用于模型并行组网。添加`model_parallel_random_seed` / `get_rng_state_tracker`用于ModelParallel的随机性控制。([#32248](https://github.com/PaddlePaddle/Paddle/pull/32248))
- Fleet添加`distributed_scaler` 接口，用于混合并行AMP策略下的loss scaler。([#32354](https://github.com/PaddlePaddle/Paddle/pull/32354))
- Fleet添加`PipelineLyaer`用于流水线并行组网切图，添加`LayerDesc`用于动态图Layer描述以减少显存初始化。([#32449](https://github.com/PaddlePaddle/Paddle/pull/32449))
- 动态图新增 Recompute 策略。([#32516](https://github.com/PaddlePaddle/Paddle/pull/32516))


#### 自定义OP

- 新增支持Mac平台上使用自定义OP功能。([#31976](https://github.com/PaddlePaddle/Paddle/pull/31976))。
- Mac平台下支持C++/v11头文件目录的自动搜索功能，兼容本地可能存在多版本clang的情况。
- 新增支持Op前反向函数Attribute参数以及inferShape, InferDtype函数输入参数使用const &类型。([#31588](https://github.com/PaddlePaddle/Paddle/pull/31588))
- 新增支持在自定义Op实现时使用三种框架内部数据类型`paddle::complex64, paddle::complex128, paddle::float16`。([#31602](https://github.com/PaddlePaddle/Paddle/pull/31602), [#31657](https://github.com/PaddlePaddle/Paddle/pull/31657), [#31669](https://github.com/PaddlePaddle/Paddle/pull/31669), [#31725](https://github.com/PaddlePaddle/Paddle/pull/31725))
- 新增支持在自定义Op中使用`std::vector<paddle::Tensor>`类型参数作为前反向函数的输入。([#31535](https://github.com/PaddlePaddle/Paddle/pull/31535))
- 新增支持InferShape函数使用Attribute参数作为输入。([#31713](https://github.com/PaddlePaddle/Paddle/pull/31713))
- 优化自动生成的Python API在动态图下的调用栈，提升执行效率。([#32209](https://github.com/PaddlePaddle/Paddle/pull/32209))
- 降低Windows上检查编译器cl.exe时的报错条件，增强Windows环境自检的鲁棒性。([#32769](https://github.com/PaddlePaddle/Paddle/pull/32769))
- 修复Windows上安装多个CUDA环境时编译器选择时的bug。([#31694](https://github.com/PaddlePaddle/Paddle/pull/31694))
- 修复Windows安装中文版本VS时出现的Python编码问题的bug。([#31493](https://github.com/PaddlePaddle/Paddle/pull/31493)) 
- 移除对单独动态库文件的依赖，仅链接框架核心动态库文件。([#32404](https://github.com/PaddlePaddle/Paddle/pull/32404)、[#32769](https://github.com/PaddlePaddle/Paddle/pull/32769))
- 移除之前的旧自定义OP方案，并对whl包中多余的库文件与头文件进行了清理，降低了whl包大小约11M。([#31813](https://github.com/PaddlePaddle/Paddle/pull/31813)), ([#32463](https://github.com/PaddlePaddle/Paddle/pull/32463))


#### 模型保存与载入
- `paddle.save, paddle.load`支持Tensor的保存加载。([#31756](https://github.com/PaddlePaddle/Paddle/pull/31756))
- `paddle.save, paddle.load`支持`list[Tensor]、dict[Tensor]、tuple[Tensor]`以及`list、tuple、dict`嵌套的包含Tensor的结构的保存加载。([#32446](https://github.com/PaddlePaddle/Paddle/pull/32446))
- `paddle.save, paddle.load`支持Layer的保存加载。([#32446](https://github.com/PaddlePaddle/Paddle/pull/32446))
- `paddle.save, paddle.load`支持Program的保存加载。([#32336](https://github.com/PaddlePaddle/Paddle/pull/32336))
- `paddle.save, paddle.load`支持C++二进制格式单个Tensor的保存加载。([#32211](https://github.com/PaddlePaddle/Paddle/pull/32211))
- `paddle.jit.save, paddle.jit.load`支持无参数的Fucntion的保存加载。([#32430](https://github.com/PaddlePaddle/Paddle/pull/32430))

### 性能优化（含分布式）
- 优化重点算子，提升多个模型单GPU训练性能，Deeplabv3+单卡FP32和AMP性能分别提升11%、72%，TSM单卡AMP性能提升44.5%，HRNet单卡FP32、AMP分别提升46%、51%。
- 增加 `index_sample` CUDA实现。([#30380](https://github.com/PaddlePaddle/Paddle/pull/30380))
- 实现`relu, leaky_relu`算子的CUDA Kernel，代替原Eigen实现，正反向共提升5% ～ 20%。([#31869](https://github.com/PaddlePaddle/Paddle/pull/31869), [#31841](https://github.com/PaddlePaddle/Paddle/pull/31841))
- `temporal_shift` 性能提升20%～40%。([#31642](https://github.com/PaddlePaddle/Paddle/pull/31642))
- 优化`depthwise_conv2d`，NHWC format下性能提升30%～50%。([#31667](https://github.com/PaddlePaddle/Paddle/pull/31677))
- 优化`interp_bilinear_grad`算子NCHW性能，提升19%~303%。([#30950](https://github.com/PaddlePaddle/Paddle/pull/30950))
- 优化`adaptive_avg_pool2d`算子NCHW、output_size = 1情况下的性能，提升80%~90% 。([#31197](https://github.com/PaddlePaddle/Paddle/pull/31197)) 
- conv op当dtype为float16时，forward和backward支持开启`exhaustive_search`。([#30959](https://github.com/PaddlePaddle/Paddle/pull/30959))
- `momentum`的`weight_decay`参数设置为float类型时，实现`momentum`和`L2Decay`的融合。([#30881](https://github.com/PaddlePaddle/Paddle/pull/30881))
- 实现`log_softmax`算子`axis`为最后一维、维度<=1024时的CUDA Kernel，相比原Eigen实现，正反向算子性能提升4.55x ~ 26.45x。([#31630](https://github.com/PaddlePaddle/Paddle/pull/31630), [#32180](https://github.com/PaddlePaddle/Paddle/pull/32180))

## 推理部署

### 模型量化

- 新增支持将FP32模型保存为FP16模型。([#32112](https://github.com/PaddlePaddle/Paddle/pull/32112))
- 重构动态图量化训练中统计输出量化信息模块，支持多Block和多分支的模型，增强通用性。([#31680](https://github.com/PaddlePaddle/Paddle/pull/31680) [#31710](https://github.com/PaddlePaddle/Paddle/pull/31710) [#31784](https://github.com/PaddlePaddle/Paddle/pull/31784) [#31861](https://github.com/PaddlePaddle/Paddle/pull/31861))
- 动态图量化训练功能支持跳过量化OP，并且和预测端形成打通。([#31704](https://github.com/PaddlePaddle/Paddle/pull/31704))


### Paddle Inference

#### 功能升级

 - 发布C API (experimental)， 功能与C++ API基本对齐。([#32225](https://github.com/PaddlePaddle/Paddle/pull/32225))
 -  重构Tensor 底层代码，与旧有 ZeroCopyTensor 数据结构解耦。此升级不涉及用户 API 改动，对用户透明。([#31402](https://github.com/PaddlePaddle/Paddle/pull/31402))
 - 预测框架python接口接入训练自定义算子。用户在训练过程中加载自定义算子后，即可像框架原生算子那样，通过 PaddlePredictor 直接执行包含此自定义算子的预测模型部署。([#32533](https://github.com/PaddlePaddle/Paddle/pull/32533)) 
 - 支持从内存加载模型时TensorRT序列化和反序列化功能。([#31342](https://github.com/PaddlePaddle/Paddle/pull/31342))

#### 性能优化
- 支持ERNIE量化模型在NV GPU上混合精度推理，其中MatMul以Int8精度计算，其他部分以FP16精度计算。相比纯FP16推理，在T4上batch size=40时，标准ERNIE模型在XNLI数据集上推理性能由1898 seq/s提升至2310 seq/s，提升17.8%。([#32232](https://github.com/PaddlePaddle/Paddle/pull/32232))

#### 易用性优化
- 用户开启TensorRT变长输入，输入shape超出限定范围时增加报错信息。([#32155](https://github.com/PaddlePaddle/Paddle/pull/32155))
- 增加运行时TensorRT版本检查，若运行和编译时TensorRT大版本号不一致会以warning提示。([#32443](https://github.com/PaddlePaddle/Paddle/pull/32443))
- 增加TensorRT VERBOSE级别log开关，用户可通过`export GLOG_v=3`开启TensorRT VERBOSE日志，打印更多调试信息。([#32459](https://github.com/PaddlePaddle/Paddle/pull/32459))


#### BugFix
- 修复预测结束后可能出现非指定使用显卡显存不足的错误。([#32655](https://github.com/PaddlePaddle/Paddle/pull/32655))
- 修复动态图下原生推理非正规值引起的CPU推理性能问题。([#32350](https://github.com/PaddlePaddle/Paddle/pull/32350)) 
- 修复在使用PaddleSlim量化模型开启TensorRT推理时，若从内存读入模型，仍会要求设置校准表路径的问题。([#32676](https://github.com/PaddlePaddle/Paddle/pull/32676))
- 升级TensorRT量化校准表接口，修复在DLA上不支持TensorRT离线量化的问题。([#31060](https://github.com/PaddlePaddle/Paddle/pull/31060))
- 修复当使用变长方式进行ERNIE/BERT模型推理时（EnableTensorRtOSS），不支持裁剪Attention的header数量的问题。([#31497](https://github.com/PaddlePaddle/Paddle/pull/31497))
- 修复2.0之后训练的BERT模型QK输入顺序不稳定带来的结果偶现diff问题。([#32659](https://github.com/PaddlePaddle/Paddle/pull/32659))
- 修复ERNIE模型开启TensorRT varlen加速时因输入变量名顺序错误导致报错或结果错误问题。([#32482](https://github.com/PaddlePaddle/Paddle/pull/32482))
- 修复TensorRT的plugin ElementwisePluginDynamic序列化失败的问题。([#31587](https://github.com/PaddlePaddle/Paddle/pull/31587))
- 修复TensorRT动态shape下FC layer维度补1带来的后续OP维度报错的问题。([#32458](https://github.com/PaddlePaddle/Paddle/pull/32458), [#31803](https://github.com/PaddlePaddle/Paddle/pull/31803))
- 修复FC使用Padding时`repeated_fc_relu_fuse_pass.cc`错误的问题。([#32648](https://github.com/PaddlePaddle/Paddle/pull/32648/files))
- 修复conv2d_transpose op使用TensorRT推理时结果错误的问题。([#32593](https://github.com/PaddlePaddle/Paddle/pull/32593))
- 修复NAN的错误比较导致的 OCR INT8 模型 oneDNN 预测报错的问题。([#32227](https://github.com/PaddlePaddle/Paddle/pull/32227))
- 修复部署多个模型在多executor上多线程进行oneDNN 预测时出现数据争用的问题。([#32499](https://github.com/PaddlePaddle/Paddle/pull/32499),  [#32136](https://github.com/PaddlePaddle/Paddle/pull/32136) [#32664](https://github.com/PaddlePaddle/Paddle/pull/32664))


## 环境适配

### 编译安装

- 新增支持CUDA11.2编译，支持3070/3080/3090显卡架构的编译。([#31529](https://github.com/PaddlePaddle/Paddle/pull/31529))
- 新增支持Windows Visual Studio 2017编译，并将发版、CI/CE、编译文档等各项配套设施，由VS2015全面升级至VS2017。([#311652](https://github.com/PaddlePaddle/Paddle/pull/31652))
- 新增对cuda11.2镜像的支持。([#32531](https://github.com/PaddlePaddle/Paddle/pull/32531))
- cuda10.1镜像支持gcc 5.4。([#32531](https://github.com/PaddlePaddle/Paddle/pull/32531))
- 镜像中新增对python 3.9的支持。([#32385](https://github.com/PaddlePaddle/Paddle/pull/32385))
- 修复`run_check`接口的bug，并在`run_check`接口里新增了对动态图的检查：现在`run_check`检测paddle安装的逻辑里，首先检测用户机器上是否有GPU，没有则报warning，未考虑安装cpu包的用户。([#32428](https://github.com/PaddlePaddle/Paddle/pull/32428))
- 修复Windows系统上缺乏 symlink 方法的问题。([#31006](https://github.com/PaddlePaddle/Paddle/pull/31006))

### 新硬件训练支持

- 新增支持海光芯片：飞桨基于 ROCM 4.0.1 版本可以在海光CPU与DCU上进行模型训练与推理。已经验证支持图像分类、目标检测、图像分割、自然语言处理、推荐系统、视频分类与语音合成共计7个分类的36个模型。([#29342](https://github.com/PaddlePaddle/Paddle/pull/29342), [#30758](https://github.com/PaddlePaddle/Paddle/pull/30758), [#30639](https://github.com/PaddlePaddle/Paddle/pull/30639), [#31009](https://github.com/PaddlePaddle/Paddle/pull/31009), [#31077](https://github.com/PaddlePaddle/Paddle/pull/31077))
- 新增支持昇腾芯片：支持在昇腾NPU上进行单机多卡训练。([#31957](https://github.com/PaddlePaddle/Paddle/pull/31957), [#32381](https://github.com/PaddlePaddle/Paddle/pull/32381),  [#32197](https://github.com/PaddlePaddle/Paddle/pull/32197), ...)
- 昆仑硬件训练支持
	- 昆仑XPU支持动态图分布式训练。([#30455](https://github.com/PaddlePaddle/Paddle/pull/30455),  [#30671](https://github.com/PaddlePaddle/Paddle/pull/30671))
	- 昆仑XPU支持fleet分布式训练。([#30858](https://github.com/PaddlePaddle/Paddle/pull/30858))
	- 昆仑XPU支持spawn启动多卡训练，优化XPU动态图多卡性能。([#31130](https://github.com/PaddlePaddle/Paddle/pull/31130))
	- 昆仑XPU静态图多卡支持fuse allreduce及gradient merge优化。([#31104](https://github.com/PaddlePaddle/Paddle/pull/31104)) 
	- 支持昆仑XPU暴露all_reduce/reduce集合通信API。([#32303](https://github.com/PaddlePaddle/Paddle/pull/32302))
	- 修复昆仑XPU动态图多卡随机hang住的bug。([#32662](https://github.com/PaddlePaddle/Paddle/pull/32662))


## Thanks to our Contributors

This release contains contributions from:

123malin, Adam Osewski, alncat,  arlesniak, AshburnLee, Aurelius84, Bai Yifan, Baibaifan, Bin Lu, cc, ceci3, chajchaj, chalsliu, channings, Chen Long, Chen Weihang, chen zhiyu, Chengmo, chentianyu03, cnn, CtfGo, cucuzg, danleifeng, denglin-github, Double\_V, fangshuixun007, Feiyu Chan, fluffyrita, FlyingQianMM, FNRE, furnace, GaoWei8, GeminiCarrie, gongweibao, Gradie, GT-Zhang, Guanghua Yu, Guo Sheng, guofei, hong, houj04, huangjun12, huangxu96, Huihuang Zheng, hutuxian, iducn, Jacek Czaja, Jack Zhou, jakpiase, JamesLim, Jiabin Yang, jiangcheng, Jiaqi Liu, Jiawei Wang, joanna.wozna.intel, joejiong, JZ-LIANG, Kaipeng Deng, Kqnonrime, kuizhiqing, Lei.C, Leo Chen, lidanqing, LielinJiang, lijianshe02, lilong12, limingshu, littletomatodonkey, liu zhengxi, LiuChiachi, liuyuhui, liym27, LoveAn, LutaoChu, minghaoBD, mls1999725, niuliling123, Ouyang Chao, pangyoki, parap1uie-s, Pei Yang, procr, Qi Li, qingqing01, QingshuChen, Ren Wei (任卫), ronnywang, ruri, seemingwang, Shang Zhizhou, shanliang1992, ShenLiang, Shibo Tao, Steffy-zxf, syyxsxx, taixiurong, tangwei12, Tao Luo, Thomas Young, Thunderbrook, tianshuo78520a, TTerror, wangchaochaohu, wangguanzhong, wanghuancoder, wangna11BD, WangXi, wangxinxin08, wawltor, Wei Shengyu, weihaoji, WeiXin, wenbin, Wenyu, whs, Wilber, winter-wang, Wojciech Uss, wuhuanzhou, wuyefeilin, XGZhang, XiangGao, XiaoguangHu, xiaoting, xiegegege, xiemoyuan, xingfeng01, Yang Zhang, yaoxuefeng, yiak, yingshengBD, yinhaofeng, Yiqun Liu, ykkk2333, yongqiangma, Yuang Liu, yukavio, YUNSHEN XIE, Y_Xuan, Zhang Jun, Zhang Ting, zhang wenhui, Zhang Zheng, zhangchunle, Zhen Wang, zhiboniu, Zhong Hui, Zhou Wei, zhulei, zhupengyang, zlsh80826, 卖鱼的哲学, 石晓伟

---

### Release Note En (`docs/release_note_en.md`)

# 2.1.1 Release Note

## Important Updates

This version fixed some function and performance issues of PaddlePaddle 2.1.0, and optimized some function. The important updates are as following:

- Optimize the API visibility of `paddle.distributed、paddle.device、paddle.vision` .
- Add support for dynamic conversion of user code for sublayer in the `paddle.nn.Sequential`.
- Add `SyncBatchNorm` support for AMP in dynamic graph, to improve the performance of dynamic graph `SyncBatchNorm` layer in AMP mode,

## Training Framework

### Functional optimization (including distributed)

#### Basic API

- Optimize the API visibility of `paddle.distributed、paddle.device、paddle.vision` , for more information, please see 2.1.0 Release Note. ([#33420](https://github.com/PaddlePaddle/Paddle/pull/32990))
- Add `paddle.is_compiled_with_rocm`. ([#33228](https://github.com/PaddlePaddle/Paddle/pull/33228))
- Add the `paddle.strided_slice` to  support bool type.（[#33373](https://github.com/PaddlePaddle/Paddle/pull/33373)）
- Add `paddle.equal_all、paddle.equal、paddle.greater_equal、paddle.greater_than、paddle.less_equal、paddle.less_than、paddle.not_equal` to support bool type. （[#33551](https://github.com/PaddlePaddle/Paddle/pull/33551)）
- Fix `paddle.utils.download` does not perform Retry when ConnectionError is abnormal.（[#33454](https://github.com/PaddlePaddle/Paddle/pull/33454)）
- Fix the issue of infershape error when `paddle.gather` axis is not equal to 0.（[#33553](https://github.com/PaddlePaddle/Paddle/pull/33553)）
- Fix segment fault caused by `paddle.io.DataLoader` when `num_workers=0` and `Dataset` returns GPU `Tensor` and sends it to `DataLoader` .（[#33487](https://github.com/PaddlePaddle/Paddle/pull/33487), [#33249](https://github.com/PaddlePaddle/Paddle/pull/33249)）
- Fix the issue that when use `slice` result as an lvalue of inplace operation, the error message of backward is not related to the error. （[#32981](https://github.com/PaddlePaddle/Paddle/pull/32981)）
- Fix the issue of `paddle.concat` support uint8 in dynamic graph.([#33667](https://github.com/PaddlePaddle/Paddle/pull/33667))
- Fix the issue of `paddle.grid_sample` GPU memory overflow and abnormal output. （[#33100](https://github.com/PaddlePaddle/Paddle/pull/33100)、[#33232](https://github.com/PaddlePaddle/Paddle/pull/33232)）
- Fix bug of roi_align, when the input width or height of rois is 0, the output feature should be 0 .（[#33446](https://github.com/PaddlePaddle/Paddle/pull/33446)）
- Fixed in some corner cases, input was modified to 'nan' bug of log_softmax op. （[#32937](https://github.com/PaddlePaddle/Paddle/pull/32937)）

#### Dynamic Graphs to Static Graphs

- Add support for dynamic conversion of user code for sublayer in the `paddle.nn.Sequential` .（[#33065](https://github.com/PaddlePaddle/Paddle/pull/33065)）
- Fix the issue of subscript syntax errors in the phase of static type analysis of variables in control flow for statement conversions. （[#32969](https://github.com/PaddlePaddle/Paddle/pull/32969)）
- Refactor the dynamic to static `param_guard` logic code to comprehensively solve the dynamic to static graph `Tensor` type conversion problem.（[#32985](https://github.com/PaddlePaddle/Paddle/pull/32985)）

#### Distributed Training

- Fix the error in `paddle.distributed.spawn` when using the default `nprocs` argument.（[#33249](https://github.com/PaddlePaddle/Paddle/pull/33249)）
- Fix the hang issue of training start caused by the inconsistent creation of pipeline parallel communication group.（[#32890](https://github.com/PaddlePaddle/Paddle/pull/32890)、[#33473](https://github.com/PaddlePaddle/Paddle/pull/33473)）
- Fix the issue of failed to save parameters in mixed parallelism.（[#33595](https://github.com/PaddlePaddle/Paddle/pull/33595)、[#33588](https://github.com/PaddlePaddle/Paddle/pull/33588)）
- Fix the issue that Fleet API cannot run `Program` directly.（[#33511](https://github.com/PaddlePaddle/Paddle/pull/33511)）
- Fix the hang issue caused by the uneven sample bucketing in the pure GPU training mode of heterogeneous parameter server.（[#32957](https://github.com/PaddlePaddle/Paddle/pull/32957)）

##### Hybrid Parallelism with Dynamic Graph

- Fix the the accuracy error of`TensorParallel`. Change the parameter initialization method of `TensorParallel` to ensure the randomness of the parameter after slicing.（[#33087](https://github.com/PaddlePaddle/Paddle/pull/33087)）
- Fix an accuracy error of `PipeLineParallel`. Fix the incorrect use of `microbatch` for `PipeLineParallel`.（[#33097](https://github.com/PaddlePaddle/Paddle/pull/33097)）
- Fix the issue that `new_group` API will hang when creating multiple communication groups.（[#33553](https://github.com/PaddlePaddle/Paddle/pull/33553)）

#### Mixed Precision Training

- Add `SyncBatchNorm` support for AMP in Dynamic graph, to improve the performance of dynamic graph `SyncBatchNorm` layer in AMP mode, and improve the 8-card AMP mode speedup ratio by 19% on `DeepLabV3P` model of [PaddleSeg].([#33709](https://github.com/PaddlePaddle/Paddle/pull/33709))

#### Custom OP

- Remove the dependency on `PADDLE_WITH_MKLDNN` macro for custom OP compilation.（[#32903](https://github.com/PaddlePaddle/Paddle/pull/32903)）
- Default setting `GLIBCXX_USE_CXX11_ABI=1` to resolve the issue of low GCC version that may cause compile-time errors.（[#33185](https://github.com/PaddlePaddle/Paddle/pull/33185)）
- Add support for c++14 syntax feature, and enable `-std=c++14` compile option by default. （[#33227](https://github.com/PaddlePaddle/Paddle/pull/33227)）

#### Others

- Fix the random segment error of training when `LoDTensorArray` is input of Op under multi-threading.（[#32984](https://github.com/PaddlePaddle/Paddle/pull/32984)）
- Fix an issue where parameter regularization is executed twice when both the regularizer of `paddle.ParamAttr` and the `weight_decay` of `paddle.optimize` are specified as `L2Decay`.（[#32881](https://github.com/PaddlePaddle/Paddle/pull/32881)）
- Fix the issue of corrupted characters of warning information in windows system.（[#33689](https://github.com/PaddlePaddle/Paddle/pull/33689)）

## Inference Deployment

### Model Quantification

- Fix the issue of skipping OP quantization in dynamic graph quantization training function.（[#32879](https://github.com/PaddlePaddle/Paddle/pull/32879)）
- Fix the issue that `layer_norm` does not save `out_threahold` attribute when quantized model is saved.（[#33610](https://github.com/PaddlePaddle/Paddle/pull/33707)）

### Paddle Inference

#### Function Upgrades

- Add converter/plugin of `gather_nd` 和 `reduce_sum` in Paddle-TRT.（[#33365](https://github.com/PaddlePaddle/Paddle/pull/33365)）
- Add `reshape` in Paddle-TRT.（[#33372](https://github.com/PaddlePaddle/Paddle/pull/33372)）

#### Performance Optimization

- Add the dynamic shape plugin of TensorRT` layer_norm` to improve model dynamic shape inference performance.（[#33448](https://github.com/PaddlePaddle/Paddle/pull/33448)）


#### 易用性优化

- Add Paddle Inference ROCm version of [Prediction Example Document](https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/09_hardware_support/rocm_docs/infer_example_cn.html),  so as to add C++ prediction library version.txt with ROCm related version information.   ([#33290](https://github.com/PaddlePaddle/Paddle/pull/33290))
- Update XPU compilation options. Please refer to [#33581](https://github.com/PaddlePaddle/Paddle/pull/33581) for specific compilation options.

#### Bug Fixes

-  Fix the calculation error of `fused_fc_elementwise_layernorm` caused by too large number of threads under DCU. ([#33299](https://github.com/PaddlePaddle/Paddle/pull/33299))
-  Fix the issue that yolov3 model fails to run after gpu is turned on on nano and TX2.([#33442](https://github.com/PaddlePaddle/Paddle/pull/33442))
-  Fix the computation error when seq_len > 1024 in Paddle-TRT `multihead_matmul plugin` .（[#33365](https://github.com/PaddlePaddle/Paddle/pull/33365)）
-  Fix the incorrect output error caused by inconsistent order of input when ERNIE model becomes longer.（[#33622](https://github.com/PaddlePaddle/Paddle/pull/33622)）
-  Fix the reports error of OCR model in prediction on GPU.([#33431](https://github.com/PaddlePaddle/Paddle/pull/33431))
-  Fix the issue that `paddle.static.io.normalize_program` failed to export `paddle.static.normalize_program`.（[#33408](https://github.com/PaddlePaddle/Paddle/pull/33408)）
-  Fix the issue that conv with stride > 1 fails in TRT6.0 and below.([#33198](https://github.com/PaddlePaddle/Paddle/pull/33198) )
-  Fix the out-of-bounds error of GPU memory access when batch predicting images. ([#33370](https://github.com/PaddlePaddle/Paddle/pull/33370) )([#33531](https://github.com/PaddlePaddle/Paddle/pull/33531) )
-  Fix the issue of cache size setting failure on X86 CPU.  （[#33571](https://github.com/PaddlePaddle/Paddle/pull/33571)）
-  Fix TRT `conv2d_transpose op converter` dimension error setting. Now the model of `conv2d_transpose` op can work normally on TRT.（[#33242](https://github.com/PaddlePaddle/Paddle/pull/33242)）
-  Fix the error of prediction library compiled by sub-CUDA Arch on Jetson devices. This version will release the Jetson prediction library compiled by sub-Arch for users who have demand for shrinked prediction library binary size.（[#33269](https://github.com/PaddlePaddle/Paddle/pull/33269)）
-  Fix the issue that when using PaddleSlim quantitative model to load prediction from memory,  it still reports an error because the calibration table path is not set.（[#33629](https://github.com/PaddlePaddle/Paddle/pull/33629)）
-  Fix the issue that BERT/ERNIE gives wrong cuda error 400 when using TRT prediction on non-0 card.（[#33706](https://github.com/PaddlePaddle/Paddle/pull/33706)）
-  Fix a cmake syntax error caused by setting custom compilation parameters under Linux.（[#33621](https://github.com/PaddlePaddle/Paddle/pull/33621)）
-  Optimize the calculation accuracy of `layer_norm` and fix the problem of outputting Nan when input is large data. ([#33420](https://github.com/PaddlePaddle/Paddle/pull/33420))

## Environment Adaptation

### Compile and install

### Support of new hardware training

#### support of Kunlun chips

- Fix the `gather` op, add support of logsumexp op. ([#32931](https://github.com/PaddlePaddle/Paddle/pull/32931))

## Thanks to our Contributors

This release contains contributions from:
Aurelius84, cc, ceci3,  Chen Weihang, danleifeng, feng_shuai, houj04, jiangcheng, JZ-LIANG, Kaipeng Deng, lidanqing, LielinJiang, Lijunhui, lilong12, liuyuhui, liym27, Pei Yang, Peihan, Qi Li, Ren Wei (任卫), Roc, Shang Zhizhou, ShenLiang, Shibo Tao, TeslaZhao, tianshuo78520a, TTerror, wangguanzhong, Wangzheee, wawltor, WeiXin, wenbin, Wenyu, whs, Wilber, wuhuanzhou, Zhang Ting, zhiboniu, Zhou Wei, zhoujun, 李季, 王明冬


# 2.1.0 Release Note

## Highlights

The PaddlePaddle Framework V2.1.0 has the following important updates:

- Environment Adaptation: Add the support for Python 3.9, CUDA 11.2; Provide the support for [ROCm platform](https://rocmdocs.amd.com/en/latest/) (experimental); Provide the support for [Ascend AI processor](https://e.huawei.com/cn/products/cloud-computing-dc/atlas/ascend-910) (experimental); Add the number of models that can run on [Baidu Kunlun chip](https://cloud.baidu.com/product/kunlun.html) . For details, please see: [Getting Started](https://www.paddlepaddle.org.cn/install/quick).

- Distributed training: besides [multidimensional hybrid parallelism](https://mp.weixin.qq.com/s/BblzcVn0NQ-QIhywvmoOuA) in static graph mode,  implementation in dynamic graph  is added.

- Framework function: Complete a number of enhancements and performance optimizations, in particular, including the following important new functions:
  
  - Customized OP: Provide a new solution for customizing operators outside the framework, simplifying the process of writing custom operators and deploying training inference. For details see: [Customizing External Operators](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/07_new_op/new_custom_op_cn.html).
  - Inplace Operation: Add the inplace operation to reduce the memory consumption and improve performance, including View strategy, and 12 inplace APIs.
  - High-level API related: Add the high-level APIs to support mixed precision training; add `paddle.hub` to view, share, and load models.
  - Automatic mixed precision training optimization: Optimized the computational performance of multiple OPs in mixed precision training such as slice, where, range, etc., and improved the acceleration effect on MaskRCNN, ERNIE and other models.
  - oneDNN BF16 training: Enabled AMP (AutoMixedPrecision) pure_BF16 mode. Enabled BF16 SGD and initializers for less memory consumption. Enabled most of FWD & BWD BF16 ops for BF16 word2vec training.

For the latest updates to the official model libraries and suites of PaddlePaddle, please see: [Paddle projects notes along with PaddlePaddle2.1](https://github.com/PaddlePaddle/Paddle/wiki/Paddle-projects-notes-along-with-PaddlePaddle2.1).

## Backwards Incompatible Changes

- The PaddlePaddle Framework 2.1 drops the support for python2 and python3.5. It is recommended that you upgrade your python to version 3.8 before using the PaddlePaddle. PaddlePaddle Framework 2.1 no longer provides the support for CUDA9 pre-built package. It is recommended that you upgrade the CUDA version before using the PaddlePaddle.
- The optimization of API visibility makes it impossible to import private APIs located in the deeply nested namespaces that are considered as implementation details by using `from deeply_nested_namespace import *`. It is recommended that you use the PaddlePaddle by following the instructions in the [API Documentation](https://www.paddlepaddle.org.cn/documentation/docs/en/api/index_en.html) on the PaddlePaddle website. Specifically, the following actions are no longer allowed in the PaddlePaddle Framework 2.1.

```python
# will import nothing from the deeply nested namespaces
from paddle.nn.layer.loss import *
from paddle.nn.layer.conv import *
```

- `Tensor.grad` Incompatible upgrade. The type of return value is changed from `numpy` to `Tensor`. ([#32142](https://github.com/PaddlePaddle/Paddle/pull/32142)) 


<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> import paddle
>>> x = paddle.to_tensor(5., stop_gradient=False)
>>> y = paddle.pow(x, 4.0)
>>> y.backward()
>>> type(x.grad)
< class ‘numpy.ndarray’ >
```
</td>

<td>

```python
>>> import paddle
>>> x = paddle.to_tensor(5., stop_gradient=False)
>>> y = paddle.pow(x, 4.0)
>>> y.backward()
>>> type(x.grad)
< class ‘paddle.Tensor’ >
```
</td>
</tr>
</table>


- `paddle.jit.TraceLayer.save_inference_model` Interface incompatibility upgrade. Changed the original first parameter dirname to path, the name symbol is more generic and unified with interfaces such as paddle.save and load, indicating that the user specifies the prefix for saving the model path. ([#31989](https://github.com/PaddlePaddle/Paddle/pull/31989)) 

<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> import os
>>> import paddle
>>> from paddle.vision.models import resnet18

>>> model = resnet18()
>>> x = paddle.rand([1, 3, 224, 224])
>>> _, static_layer = paddle.jit.TracedLayer.trace(model, inputs=[x])
>>> save_path = './save_infer_model'
>>> static_layer.save_inference_model(dirname=save_path)

>>> print(os.path.isdir(save_path))
>>> print(len(os.listdir(save_path)))
True
103
```
</td>

<td>

```python
>>> import os
>>> import paddle
>>> from paddle.vision.models import resnet18

>>> model = resnet18()
>>> x = paddle.rand([1, 3, 224, 224])
>>> _, static_layer = paddle.jit.TracedLayer.trace(model, inputs=[x])
>>> save_path = 'save_infer_model'
>>> static_layer.save_inference_model(path=save_path)

>>> print(os.path.isdir(save_path))
>>> print([name for name in os.listdir('./') if name.startswith(save_path)])
False
['save_infer_model.pdmodel', 'save_infer_model.pdiparams']
```
</td>
</tr>
</table>


- `paddle.io.DataLoader`return format incompatibility upgrade when user-define dataset only contains single field。If user-define dataset only contains single field and output data with code like `return image` or `yield image`，output data format in Paddle 2.0 is `[image_tensor]`，and output data format in Paddle 2.1 is `image_tensor` to keep data structure same with input.

<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> import numpy as np
>>> import paddle
>>> from paddle.io import DataLoader, Dataset
>>> 
>>> class RandomDataset(Dataset):
>>>     def __getitem__(self, idx):
>>>         return np.random.random((2, 3)).astype('float32')
>>> 
>>>     def __len__(self):
>>>         return 10
>>> 
>>> dataset = RandomDataset()
>>> loader = DataLoader(dataset, batch_size=1)
>>> data = next(loader())
>>> print(data)
[Tensor(shape=[1, 2, 3], dtype=float32, place=CUDAPinnedPlace, stop_gradient=True,
       [[[0.73782003, 0.62605530, 0.32727283],
         [0.37154925, 0.63570684, 0.53859973]]])]
```
</td>

<td>

```python
>>> import numpy as np
>>> import paddle
>>> from paddle.io import DataLoader, Dataset
>>> 
>>> class RandomDataset(Dataset):
>>>     def __getitem__(self, idx):
>>>         return np.random.random((2, 3)).astype('float32')
>>> 
>>>     def __len__(self):
>>>         return 10
>>> 
>>> dataset = RandomDataset()
>>> loader = DataLoader(dataset, batch_size=1)
>>> data = next(loader())
>>> print(data)
Tensor(shape=[1, 2, 3], dtype=float32, place=CUDAPinnedPlace, stop_gradient=True,
       [[[0.73782003, 0.62605530, 0.32727283],
         [0.37154925, 0.63570684, 0.53859973]]])
```
</td>
</tr>
</table>


## Training Framework

### Functional optimization (including distributed)

#### Basic API

- Add data types such as `paddle.dtype` and `paddle.float32` as data types within the Paddle. ([#32012](https://github.com/PaddlePaddle/Paddle/pull/32012))
- Add `paddle.nn.functional.glu`. ([#32096](https://github.com/PaddlePaddle/Paddle/pull/32096))
- Add `paddle.nn.utils.spectral_norm`. ([#32633](https://github.com/PaddlePaddle/Paddle/pull/32633))
- Add `paddle.Tensor.register_hook` API for registering the hook function for the gradient Tensor corresponding to the forward Tensor in dynamic graph scenes. ([#31775](https://github.com/PaddlePaddle/Paddle/pull/31775))
- Add the `Tensor.__array__` function to support `numpy.array(Tensor)` and `numpy.asarray(Tensor)` to convert `paddle.Tensor` type to `numpy.ndarray` type . ([#32300](https://github.com/PaddlePaddle/Paddle/pull/32300))
- Add the Tensor API: `Tensor.item(*args)`. It can convert the element at the specified position in Tensor to Python scalar value and return it. ([#32634](https://github.com/PaddlePaddle/Paddle/pull/32634))
- Add the `paddle.nn.LayerList` support for negative indexing. ([#31750](https://github.com/PaddlePaddle/Paddle/pull/31750))
- Add 12 dynamic graph inplace APIs: `clip_`, `scale_`, `add_`, `subtract_`, `ceil_`, `floor_`, `exp_`, `reciprocal_`, `round_`, `sqrt_`, `rsqrt_`, and `flatten_`. These inplace APIs cannot be called by using `paddle.api_` and should be called by using `Tensor.api_`. ([#32699](https://github.com/PaddlePaddle/Paddle/pull/32699))
- Add `paddle.autograd.backward` API for customizing the starting gradient. ([#31540](https://github.com/PaddlePaddle/Paddle/pull/31540))
- Add `paddle.nn.LayerDict` class. ([#31951](https://github.com/PaddlePaddle/Paddle/pull/31951))
- Add `layer.to` API. ([#32040](https://github.com/PaddlePaddle/Paddle/pull/32040))
- Add `paddle.autograd.PyLayer` API for supporting custom backward calculation of dynamic graphs on Python side. ([#32130](https://github.com/PaddlePaddle/Paddle/pull/32130))
- Add the support for `paddle.optimizer` to specify non-parametric Tensor as parameters for optimization in dynamic graphs. ([#32362](https://github.com/PaddlePaddle/Paddle/pull/32362))
- Add several `sequence*` functions in `paddle.static.nn`. Add `paddle.nn.functional` in `sequence_mask`. ([#32089](https://github.com/PaddlePaddle/Paddle/pull/32089))
- Add `paddle.nn.CTCLoss` parameters in `norm_by_times`. ([#32490](https://github.com/PaddlePaddle/Paddle/pull/32490))
- `paddle.fill_constant` supports `uint8_t`. ([#31911](https://github.com/PaddlePaddle/Paddle/pull/31911))
- `paddle.clip` supports `int32` and `int64`. ([#32373](https://github.com/PaddlePaddle/Paddle/pull/32373))
- Support the input data type to be int in Nearest neighbor mode in `paddle.nn.functional.interpolate`. ([#32270](https://github.com/PaddlePaddle/Paddle/pull/32270))
- All parameters in API that support passing in list or tuple are upgraded to support passing in list and tuple. ([#32344](https://github.com/PaddlePaddle/Paddle/pull/32344),  [#32528](https://github.com/PaddlePaddle/Paddle/pull/32528) [#32360](https://github.com/PaddlePaddle/Paddle/pull/32360))
- Optimize `softmax` operator performance. ([#31821](https://github.com/PaddlePaddle/Paddle/pull/31821))
- Optimize `paddle.norm` documentation description to clarify the functional differences between `paddle.norm` and `numpy.linalg.norm` API. ([#32530](https://github.com/PaddlePaddle/Paddle/pull/32530))
- Optimize the printing form of data type (`datatype`) of Tensor, for example, the `dtype` of Tensor of `float32` type is changed from `VarType.FP32` to `paddle.float32`. ([#30682](https://github.com/PaddlePaddle/Paddle/pull/30682))
- OneDNN Functional optimization
    - Upgraded oneDNN to 2.2.1 ([#31067](https://github.com/PaddlePaddle/Paddle/pull/31067) [#31473](https://github.com/PaddlePaddle/Paddle/pull/31473) [#30295](https://github.com/PaddlePaddle/Paddle/pull/30295) [32227](https://github.com/PaddlePaddle/Paddle/pull/32227))
    - Added more precise mkldnn kernel rules in GetExpectedKernelType based on kernel's data type. ([#29840](https://github.com/PaddlePaddle/Paddle/pull/29840))
    - Fused `layer_norm` subgraphs to single `layer_norm` op. ([#32162](https://github.com/PaddlePaddle/Paddle/pull/32162), [#30891](https://github.com/PaddlePaddle/Paddle/pull/30891), [#30962](https://github.com/PaddlePaddle/Paddle/pull/30962))
    - Reduced unnecessary memory allocation during creation of `elementwise_mul` operator ([#30203](https://github.com/PaddlePaddle/Paddle/pull/30203))
    - Improved memory consumption used in cache per thread ([#30358](https://github.com/PaddlePaddle/Paddle/pull/30358))
    - Added oneDNN FP32 and INT8 support for vanilla LSTM ([#30719](https://github.com/PaddlePaddle/Paddle/pull/30719) [#31894](https://github.com/PaddlePaddle/Paddle/pull/31894))
    - Added OneDNN `hardswish` support ([#30211](https://github.com/PaddlePaddle/Paddle/pull/30211))
    - Added `bilinear_interp_v2` and `nearest_interp_v2` oneDNN FP32 kernels ([#32312](https://github.com/PaddlePaddle/Paddle/pull/32312))
- Updated Xbyak to v5.81 ([#30809](https://github.com/PaddlePaddle/Paddle/pull/30809))
- Fix `paddle.io.DataLoader` to support data sets containing nested complex data formats such as list, dict and string, and fix the occasional error report and unreleased resources when the program exits during the iteration. ([#31481](https://github.com/PaddlePaddle/Paddle/pull/31481))
- Fix the problem caused by modifying the root logger of logging library in paddle. ([#32706](https://github.com/PaddlePaddle/Paddle/pull/32706))
- Fix the problem of `L1Decay` error report in `backward` dynamic graph mode. ([#32718](https://github.com/PaddlePaddle/Paddle/pull/32718))
- Fix the problem that nan comes out in setting `ignore_index` and `reduction='mean'` in `paddle.nn.functional.cross_entropy`. ([#32545](https://github.com/PaddlePaddle/Paddle/pull/32545))
- Fix the problem that the output type is bool during the summing of bool tensor and float tensor. ([#32272](https://github.com/PaddlePaddle/Paddle/pull/32272))
- Fix the calculation error of comparison class API in broadcast. ([#32470](https://github.com/PaddlePaddle/Paddle/pull/32470))
- Fix the gradient calculation error under broadcast where right input is large shape in addition, subtraction, multiplication and division. ([#30818](https://github.com/PaddlePaddle/Paddle/pull/30818))
- Fix the problem of the calculation result of segment mean OP being incorrect when processing the large shape tensor input. ([#32610](https://github.com/PaddlePaddle/Paddle/pull/32610))
- Fix the problem of the data type of optimizer variables not matching with the data type of model parameters. ([#29917](https://github.com/PaddlePaddle/Paddle/pull/29917))
- Fix the error report in `num worker>0` when the `paddle.io.DataLoader` pre-processing includes the paddle operation. ([#31177](https://github.com/PaddlePaddle/Paddle/pull/31177))
- Fix the error report when printing empty tensor. ([#32501](https://github.com/PaddlePaddle/Paddle/pull/32501))
- Adjust the initialization order of static graph parameters, and keep consistency with dynamic graphs after adjustment, so that the same model is set with the same random seed to get the same parameters initialized in dynamic graphs and static graphs. ([#32177](https://github.com/PaddlePaddle/Paddle/pull/32177))
- Fix the bug that `paddle.to_tensor` does not support accepting `dtype=Tensor.dtype`. ([#31931](https://github.com/PaddlePaddle/Paddle/pull/31931))
- Fix the bug that the gradient is nan when 2 inputs are equal in `paddle.dist`. ([#32448](https://github.com/PaddlePaddle/Paddle/pull/32448))
- `paddle.nn.functional.temporal_shift` added `data_format` property to support to set to NCHW or NHWC. ([#31642](https://github.com/PaddlePaddle/Paddle/pull/31642))
- Fix the problem of the calculation result being incorrect in `adaptive_avg_pool2d` when the input data type is float16. ([#31887](https://github.com/PaddlePaddle/Paddle/pull/31887))
- `paddle.nn.Layer.sublayers` and `paddle.nn.Layer.named_sublayers`: Modify the `include_sublayers = True` parameter of original `paddle.nn.Layer.sublayers` to `include_self = False`, thus fixing the problem of returning null of the former `include_sublayers = False`. Now the default behavior is the same as that when no parameter is filled in, that is, return all recursive sublevels that don't contain themselves. When `include_self = True` is the same as the literal meaning, return all recursive sublevels that contain themselves. The `include_sublayers` parameter in `paddle.nn.Layer.named_sublayers` is directly removed. Other behaviors remain unchanged. ([#31824](https://github.com/PaddlePaddle/Paddle/pull/31824) )

<table>
<tr>
<th>
2.0
</th>
<th>
2.1
</th>
</tr>

<tr>
<td>

```python
>>> from paddle.vision.models import resnet18
>>> model = resnet18()
>>> 
>>> print(len(model.sublayers(include_sublayers=True)))
>>> print(len(model.sublayers(include_sublayers=False)))
67
0
```
</td>

<td>

```python
>>> from paddle.vision.models import resnet18
>>> model = resnet18()
>>> 
>>> print(len(model.sublayers(include_self=True)))
>>> print(len(model.sublayers(include_self=False)))
68
67
```
</td>
</tr>
</table>


#### High-level API

- Add the `paddle.hub` function. Provide `help`, `list` and `load` functions for viewing and loading third-party models, and support the loading of remote and local repository. ([#31873](https://github.com/PaddlePaddle/Paddle/pull/31873))
- Support the mixed precision training. Provide O0, O1, O2 three modes, which correspond to FP32 training, automatic mixed precision training, pure FP16 training respectively. At present, pure FP16 training only supports static graphs. ([#31417](https://github.com/PaddlePaddle/Paddle/pull/31417))
- Support the image transformation of the `paddle.Tensor` type, including operators such as `normalize, to_grayscale, vflip, hflip, crop, center_crop, pad, rotate, resize`. ([#32705](https://github.com/PaddlePaddle/Paddle/pull/32705))

#### Dynamic Graphs to Static Graphs

Fix the bug of dynamic graphs converted to static graphs.

- The shape returned by the static graph `arange、range` API is not consistent with the dynamic graph.
- `paddle.to_tensor` supports the input as `int，float，bool` basic type in dynamic to static.
- Support the parsing of the dict derivative syntax in the for loop. ([#32159](https://github.com/PaddlePaddle/Paddle/pull/32159))
- Fix the problem of undeclared variables errors in the nested control flow statements in some scenarios. ([#32153](https://github.com/PaddlePaddle/Paddle/pull/32153))
- Fix the bug that the float16 type is missed in `expand` op. ([#32238](https://github.com/PaddlePaddle/Paddle/pull/32238))
- Fix the bug of returning the gradient information as None when the shape dimension is 6 in the `expand_v2、tile、expand、expand_as、expand_as_v2、meshgrid` 6 OP backward gradient solution. ([#32004](https://github.com/PaddlePaddle/Paddle/pull/32004))
- Fix the problem that the `paddle.jit.TraceLayer.save_inference_model` interface is inconsistent with `paddle.static.load_inference_model` because the network structure and parameters are not saved at the same time. ([#31989](https://github.com/PaddlePaddle/Paddle/pull/31989))

#### Mixed Precision Training

- The op that does not support fp16 kernel is automatically kept as fp32 calculation in the dynamic graph mixed precision interface auto\_cast. ([#32543](https://github.com/PaddlePaddle/Paddle/pull/32543))
- Fix the unexpected error in the static graph mixed precision training caused by the incomplete statistics of the Op list (`unsupported_fp16_list`) which does not support FP16 calculation. The list of Op that currently does not support FP16 calculation can be generated automatically according to the runtime environment. ([#32102](https://github.com/PaddlePaddle/Paddle/pull/32102))
- In the for loop in the `update_loss_scaling`, optimize the problem that multiple identical cuda kernel are fused into one cuda kernel. ([#32554](https://github.com/PaddlePaddle/Paddle/pull/32554))
- Optimize the slow performance in `slice` multi-dimensional cases. ([#32266](https://github.com/PaddlePaddle/Paddle/pull/32266))
- Optimize the redundant copy problem when `elementwise_add_grad` inputs and outputs are the same. ([#32051](https://github.com/PaddlePaddle/Paddle/pull/32051))
- In the for loop in the `check_finite_and_unscale`, optimize the problem that multiple identical cuda kernel are fused into one cuda kernel. ([#31954](https://github.com/PaddlePaddle/Paddle/pull/31954))
- Optimize the `range` parameter redundant copy problem. ([#30811](https://github.com/PaddlePaddle/Paddle/pull/30811))
- Optimize the slow performance problem of  `top_k_v2` in `input_width <= 1024`. ([#30403](https://github.com/PaddlePaddle/Paddle/pull/30403))
- Migrate `where_index` CPU calculation process to GPU for completion. ([#30601](https://github.com/PaddlePaddle/Paddle/pull/30601))

#### BF16 Training 

- Added initial bf16 amp integration that modify models by adding cast ops to BF16 enabled ops in the forward pass. [#31093](https://github.com/PaddlePaddle/Paddle/pull/31093) 
- Added BF16 pure_mode, which means adding support for BF16 training based on BF16-enabled ops list and enable BF16 parameters, BF16 operators, BF16 decorator for optimizer during training.  [#32281](https://github.com/PaddlePaddle/Paddle/pull/32281) [#32681](https://github.com/PaddlePaddle/Paddle/pull/32681)
- Added CPU core flags verification for BF16 fast performance support. [#30551](https://github.com/PaddlePaddle/Paddle/pull/30551)
- Unification of BF16 enablement process [#31034](https://github.com/PaddlePaddle/Paddle/pull/31034)
- Added BF16 Constant Initializer and for other initializers, add cast op to convert other initializer output to be BF16 datatype. [#31935](https://github.com/PaddlePaddle/Paddle/pull/31935) 
- Added BF16 uniform random initializer [#32468](https://github.com/PaddlePaddle/Paddle/pull/32468)
- Added mechanism that converts startup_program initializers to BF16 [#32720](https://github.com/PaddlePaddle/Paddle/pull/32720)
- Added BF16 support for sgd operator CPU kernel. [#32162](https://github.com/PaddlePaddle/Paddle/pull/32162) 
- Added BF16 support for lookup_table operator. [#31558](https://github.com/PaddlePaddle/Paddle/pull/31558)
- Added Sum kernel for CPU supporting BF16 and SelectedRows [#32755](https://github.com/PaddlePaddle/Paddle/pull/32755) [#32631](https://github.com/PaddlePaddle/Paddle/pull/32631) 
- Added Conv Transpose BF16 support [#30877](https://github.com/PaddlePaddle/Paddle/pull/30877) 
- Added elementwise_add bf16 grad [#30925](https://github.com/PaddlePaddle/Paddle/pull/30925)
- Added reshape op BWD grad bf16 [#31035](https://github.com/PaddlePaddle/Paddle/pull/31035)
- Added broadcasting support in elementwise_add grad bf16/fp32 [#31385](https://github.com/PaddlePaddle/Paddle/pull/31385) 
- Added Elementwise Mul grad fp32/bf16 [#31647](https://github.com/PaddlePaddle/Paddle/pull/31647)
- Added LSTM BF16 and fixed GRU BF16 [#31234](https://github.com/PaddlePaddle/Paddle/pull/31234) 
- Added oneDNN reduce_op fp32 and bf16 kernels [#31816](https://github.com/PaddlePaddle/Paddle/pull/31816) 
- Added oneDNN reduce_op GRAD fp32 and bf16 kernels [#32280](https://github.com/PaddlePaddle/Paddle/pull/32280) [#32592](https://github.com/PaddlePaddle/Paddle/pull/32592) 

#### Distributed Training Optimization

- New graph-based retrieval engine for training distributed graph neural network over trillion edges([#31226](https://github.com/PaddlePaddle/Paddle/pull/31226)).
- Added index-based data sampling class to support sampling from graph and TDM/OTM tree([#31696](https://github.com/PaddlePaddle/Paddle/pull/31696)).
- Added `paddle.distributed.send, paddle.distributed.recv, paddle.distributed.new_group, paddle.distributed.wait` to improve the distributed communication API. ([#32504](https://github.com/PaddlePaddle/Paddle/pull/32504),  [#31682](https://github.com/PaddlePaddle/Paddle/pull/31682))
- Support to initialize the  `sync_parameters_buffer`in the distributed dynamic graph, which solved the issue that the buffer of the dynamic graph is not globally initialized. ([#31625](https://github.com/PaddlePaddle/Paddle/pull/31625))
- Pipeline Parallelism supports 1F1B scheduling method to optimize the memory usage of GPU. Theoretically, it is constant([#31786](https://github.com/PaddlePaddle/Paddle/pull/31786)).
- \[Hybrid Parallel] Sharding strategy optimization: support Gradients aggregation, reducing the amount of parameter communication, and improving the speed of training; Could be used flexibly with other parallelism strategies. ([#31884](https://github.com/PaddlePaddle/Paddle/pull/31884) [#32486](https://github.com/PaddlePaddle/Paddle/pull/32486) [#32485](https://github.com/PaddlePaddle/Paddle/pull/32485) [#31996](https://github.com/PaddlePaddle/Paddle/pull/31996) [#31939](https://github.com/PaddlePaddle/Paddle/pull/31939) [#31796](https://github.com/PaddlePaddle/Paddle/pull/31796))
- \[Hybrid Parallel] Added optimizer state offload in the Sharding strategy, to reduce the memory usage of GPU. ([#32134](https://github.com/PaddlePaddle/Paddle/pull/32134))
- \[Hybrid Parallel] Support the persistence of the broadcast ID’s socket service, reduced the conflicts of ports in the hybrid parallelism. ([#31589](https://github.com/PaddlePaddle/Paddle/pull/31589))
- \[Parameter Server] Optimize the output and printing of LOG, and remove invalid logs.
- \[Parameter Server] Optimize the sparse parameter storage structure, with large memory reduction for small dimensions (below 64).
- \[Parameter Server] Fix the bug of access policy taking effect in the distributed prediction.
- HeterPs supports multiple machines. ([#31102](https://github.com/PaddlePaddle/Paddle/pull/31102))

##### Hybrid Parallelism with dynamic Graph

Support hybrid parallelism in the distributed dynamic graph mode, powered by data parallelism, model parallelism and pipeline parallelism, in addition, they can combine with AMP and the new ReCompute strategy to achieve better efficiency.

- Support hybrid parallelism with the Fleet dynamic graph API, and any arbitrary combination of data/model/pipeline parallelism. ([#32248](https://github.com/PaddlePaddle/Paddle/pull/32248))
-  Added parameter `find_unused_parameters` n the data parallelism of distributed dynamic graph to support grouping control flow in the network. ([#31625](https://github.com/PaddlePaddle/Paddle/pull/31625))
- Added `VocabParallelEmbedding`, `ColumnParallelLinear`, `RowParallelLinear` Fleet API for model parallelism. Added `model_parallel_random_seed`/`get_rng_state_tracker` for the random control used in model parallelism. ([#32248](https://github.com/PaddlePaddle/Paddle/pull/32248))
- Added `distributed_scaler` interface for loss scaler of AMP combined with the hybrid parallelism strategy. ([#32354](https://github.com/PaddlePaddle/Paddle/pull/32354))
- Added `PipelineLyaer` to partition graph in the pipeline parallelism, added `LayerDesc` or description of dynamic graph Layer to reduce memory initialization. ([#32449](https://github.com/PaddlePaddle/Paddle/pull/32449))
- Add Recompute strategy for dynamic graphs. ([#32516](https://github.com/PaddlePaddle/Paddle/pull/32516))

#### Custom OP

- Add support for using custom OP function on Mac platform. ([#31976](https://github.com/PaddlePaddle/Paddle/pull/31976))
- Support automatic search function of C++/v11 header file directory on Mac platform, compatible with the situation that multiple versions of clang may exist locally.
- Add support for Op forward/backward function Attribute parameter, inferShape, and InferDtype function input parameter using the `const &` type. ([#31588](https://github.com/PaddlePaddle/Paddle/pull/31588))
- Add support for using three framework internal data types `paddle::complex64, paddle::complex128, paddle::float16`  in the implementation of custom Op. ([#31602](https://github.com/PaddlePaddle/Paddle/pull/31602), [#31657](https://github.com/PaddlePaddle/Paddle/pull/31657), [#31669](https://github.com/PaddlePaddle/Paddle/pull/31669), [#31725](https://github.com/PaddlePaddle/Paddle/pull/31725))
- Add support for using `std::vector<paddle::Tensor>` type parameters as input of forward/backward functions in custom Op. ([#31535](https://github.com/PaddlePaddle/Paddle/pull/31535))
- Add support for the InferShape function using Attribute parameter as input. ([#31713](https://github.com/PaddlePaddle/Paddle/pull/31713))
- Optimize the call stack of auto-generated Python API under dynamic graph to improve the execution efficiency. ([#32209](https://github.com/PaddlePaddle/Paddle/pull/32209))
- Reduce the error reporting condition when checking the compiler cl.exe on Windows, and enhance the self-test robustness in Windows environment. ([#32769](https://github.com/PaddlePaddle/Paddle/pull/32769))
- Fix a bug in compiler selection when installing multiple CUDA environments on Windows. ([#31694](https://github.com/PaddlePaddle/Paddle/pull/31694))
- Fix a bug in Python encoding issue when installing Chinese version of VS on Windows. ([#31493](https://github.com/PaddlePaddle/Paddle/pull/31493))
- Remove the dependency on separate dynamic library files and link only the framework core dynamic library files. ([#32404](https://github.com/PaddlePaddle/Paddle/pull/32404)、[#32769](https://github.com/PaddlePaddle/Paddle/pull/32769))
- Remove the previous old custom OP scheme and clean up the redundant library files and header files in the whl package, reducing the whl package size by about 11M. ([#31813](https://github.com/PaddlePaddle/Paddle/pull/31813)), ([#32463](https://github.com/PaddlePaddle/Paddle/pull/32463))

#### Model saving and loading

- `paddle.save, paddle.load` supports saving and loading of Tensor. ([#31756](https://github.com/PaddlePaddle/Paddle/pull/31756))
- `paddle.save, paddle.load` supports saving and loading of `list[Tensor]、dict[Tensor]、tuple[Tensor]` and `list、tuple、dict` nested structures containing Tensor. ([#32446](https://github.com/PaddlePaddle/Paddle/pull/32446))
- `paddle.save, paddle.load` supports saving and loading of Layer. ([#32446](https://github.com/PaddlePaddle/Paddle/pull/32446))
- `paddle.save, paddle.load` supports saving and loading of Program. ([#32336](https://github.com/PaddlePaddle/Paddle/pull/32336))
- `paddle.save, paddle.load` supports saving and loading of single Tensor in C++ binary format. ([#32211](https://github.com/PaddlePaddle/Paddle/pull/32211))
- `paddle.jit.save, paddle.jit.load` supports saving and loading of Fucntion without parameters. ([#32430](https://github.com/PaddlePaddle/Paddle/pull/32430))

### Performance optimization (including distributed)

- Optimize key operators to improve single GPU training performance of multiple models. Deeplabv3+ single card FP32 and AMP performance are improved by 11% and 72% respectively. TSM single card AMP performance is improved by 44.5%. HRNet single card FP32 and AMP are improved by 46% and 51% respectively.
- Add `index_sample` CUDA implementation. ([#30380](https://github.com/PaddlePaddle/Paddle/pull/30380))
- Implement the CUDA Kernel of `relu, leaky_relu` operator, replacing the original Eigen implementation, with a total improvement of 5% - 20% in forward and backward directions. ([#31869](https://github.com/PaddlePaddle/Paddle/pull/31869), [#31841](https://github.com/PaddlePaddle/Paddle/pull/31841))
- `temporal_shift` Performance improvement by 20% to 40%. ([#31642](https://github.com/PaddlePaddle/Paddle/pull/31642))
- Optimize `depthwise_conv2d`. Performance is improved by 30% to 50% under the NHWC format. ([#31667](https://github.com/PaddlePaddle/Paddle/pull/31677))
- Optimize `interp_bilinear_grad` operator NCHW performance with improvement by 19% - 303%. ([#30950](https://github.com/PaddlePaddle/Paddle/pull/30950))
- Optimize the performance of `adaptive_avg_pool2d` operator NCHW. In case of output\_size = 1 case, improve by 80%~90%. ([#31197](https://github.com/PaddlePaddle/Paddle/pull/31197))
- In conv op, when dtype is float16, forward and backward support the enabling of `exhaustive_search`. ([#30959](https://github.com/PaddlePaddle/Paddle/pull/30959))
- When `weight_decay` parameter of `momentum` is set to float type, the fusion of  `momentum` and `L2Decay` is achieved ([#30881](https://github.com/PaddlePaddle/Paddle/pull/30881))
- Implement CUDA Kernel when `log_softmax` operator `axis` is the last dimension and dimension is equal to or smaller than 1024. Compared to the original Eigen, the forward and backward operator performance is improved by 4.55x ~ 26.45x. ([#31630](https://github.com/PaddlePaddle/Paddle/pull/31630), [#32180](https://github.com/PaddlePaddle/Paddle/pull/32180))

## Inference Deployment

### Model Quantization

- Add the support for saving FP32 model as FP16 model. ([#32112](https://github.com/PaddlePaddle/Paddle/pull/32112))
- Refactor the module of statistical output quantization information in dynamic graph quantization training to support multi-Block and multi-branch models to enhance generality. ([#31680](https://github.com/PaddlePaddle/Paddle/pull/31680) [#31710](https://github.com/PaddlePaddle/Paddle/pull/31710) [#31784](https://github.com/PaddlePaddle/Paddle/pull/31784) [#31861](https://github.com/PaddlePaddle/Paddle/pull/31861))
- Dynamic graph quantization training function supports the skipping of the quantization OP and forms the successful connection at the prediction side. ([#31704](https://github.com/PaddlePaddle/Paddle/pull/31704))

### Paddle Inference

#### Function Upgrade

- Release C API (experimental).  The function of  new C API is basically equal to that of C + +. ([#32225](https://github.com/PaddlePaddle/Paddle/pull/32225))
- The prediction framework python interface access to the train custom operators. After loading a custom operator during training, users can execute the deployment of prediction models containing this custom operator directly through PaddlePredictor, just like the framework's native operator. ([#32533](https://github.com/PaddlePaddle/Paddle/pull/32533))
- The underlying implementation of Tensor has been refactored internally to decouple from the old ZeroCopyTensor data structure. This upgrade does not involve user API changes and is transparent to users. ([#31402](https://github.com/PaddlePaddle/Paddle/pull/31402))
- Support TensorRT serialization and deserialization when loading models from memory. ([#31342](https://github.com/PaddlePaddle/Paddle/pull/31342))

#### Performance Optimization

- Support quantilized  ERNIE models to be inferred with  the mixed precision using TensorRT, where Matmul is computed with Int8 precision and other parts are computed with FP16 precision. Compared with the pure FP16 inference, the inference performance of the standard ERNIE model on XNLI dataset is improved from 1898 seq/s to 2310 seq/s at batch size=40 on T4, improving by 17.8%. ([#32232](https://github.com/PaddlePaddle/Paddle/pull/32232))

#### Ease-of-use optimization

- Add  error messages when the user enables the TensorRT variable-length input settings, and the wrong input shape  is provided. ([#32155](https://github.com/PaddlePaddle/Paddle/pull/32155))
- Add runtime TensorRT version check. If the major version number of TensorRT at runtime and compile time differs, the warning is generated. ([#32443](https://github.com/PaddlePaddle/Paddle/pull/32443))
- Add the TensorRT VERBOSE level log switch. Users can enable the TensorRT VERBOSE log by `export GLOG_v=3` to print more debugging information. ([#32459](https://github.com/PaddlePaddle/Paddle/pull/32459))

#### BugFix
- Fix the error of insufficient graphics card or video memory of unspecified usage at the end of prediction. ([#32655](https://github.com/PaddlePaddle/Paddle/pull/32655))
- Fix the CPU performance issue caused by informal values of native inference in dynamic graphs. ([#32350](https://github.com/PaddlePaddle/Paddle/pull/32350))
- Fix the problem of requiring the setting of the calibration table path in the data read from memory when TensorRT inference is enabled by using the PaddleSlim quantization model. ([#32676](https://github.com/PaddlePaddle/Paddle/pull/32676))
- Upgrade the TensorRT quantization calibration table interface, fix the problem that TensorRT offline quantization is not supported on DLA. ([#31060](https://github.com/PaddlePaddle/Paddle/pull/31060))
-  Fix the problem of the number of header of crop Attention not being supported when using variable length method for ERNIE/BERT model inference (EnableTensorRtOSS). ([#31497](https://github.com/PaddlePaddle/Paddle/pull/31497))
- Fix the occasional diff problem caused by the instable QK input sequence of the BERT model trained after version 2.0 ([#32659](https://github.com/PaddlePaddle/Paddle/pull/32659))
- Fix the problem that ERNIE model reports an error or incorrect result due to the wrong order of input variable names when TensorRT varlen acceleration is enabled. ([#32482](https://github.com/PaddlePaddle/Paddle/pull/32482))
- Fix the bug that plugin ElementwisePluginDynamic serialization of TensorRT fails. ([#31587](https://github.com/PaddlePaddle/Paddle/pull/31587))
- Fix the problem of subsequent OP dimension error caused by FC layer dimension complement 1 under TensorRT dynamic shape. ([#32458](https://github.com/PaddlePaddle/Paddle/pull/32458), [#31803](https://github.com/PaddlePaddle/Paddle/pull/31803))
- Fix the problem of `repeated_fc_relu_fuse_pass.cc` error when FC uses Padding. ([#32648](https://github.com/PaddlePaddle/Paddle/pull/32648/files))
- Fix the problem of the result of conv2d\_transpose op being wrong when using TensorRT inference. ([#32593](https://github.com/PaddlePaddle/Paddle/pull/32593))
- Fix the problem with OCR INT8 model oneDNN prediction errors caused by incorrect comparison of NAN. ([#32227](https://github.com/PaddlePaddle/Paddle/pull/32227))
- Fix the problem of data contention when deploying multiple models for oneDNN prediction on multiple executors with multiple threads. ([#32499](https://github.com/PaddlePaddle/Paddle/pull/32499),  [#32136](https://github.com/PaddlePaddle/Paddle/pull/32136) [#32664](https://github.com/PaddlePaddle/Paddle/pull/32664))


## Environment Adaptation

### Compile and install

- Add support for CUDA11.2 compilation. Support the compilation based on the 3070/3080/3090 graphics card architecture. ([#31529](https://github.com/PaddlePaddle/Paddle/pull/31529))
- Add the support for compilation of Windows Visual Studio 2017. Upgrade all supporting facilities such as release, CI/CE, compilation documentation, etc. from VS2015 to VS2017 comprehensively. ([#311652](https://github.com/PaddlePaddle/Paddle/pull/31652))
- Add support for cuda11.2 image. ([#32531](https://github.com/PaddlePaddle/Paddle/pull/32531))
- cuda10.1 image support for gcc 5.4. ([#32531](https://github.com/PaddlePaddle/Paddle/pull/32531))
- Add support for python 3.9 in mirrors. ([#32385](https://github.com/PaddlePaddle/Paddle/pull/32385))
- Fix the bug of `run_check` interface, and add the check of dynamic graph in  `run_check` interface: Now the logic of `run_check` detecting paddle installation first detects whether there is a GPU on the user's machine. If not, report warning, without considering the users who install the cpu package ([#32428](https://github.com/PaddlePaddle/Paddle/pull/32428))
- Fix the problem of lack of symlink method on Windows system. ([#31006](https://github.com/PaddlePaddle/Paddle/pull/31006))

### New hardware training support

- Add the support for Hygon chips:  PaddlePaddle, based on ROCM version 4.0.1, can train and infer models on Hygon CPU and DCU. A total of 36 models of 7 categories of image classification, target detection, image segmentation, natural language processing, recommendation systems, video classification and speech synthesis have been validated. ([#29342](https://github.com/PaddlePaddle/Paddle/pull/29342), [#30758](https://github.com/PaddlePaddle/Paddle/pull/30758), [#30639](https://github.com/PaddlePaddle/Paddle/pull/30639), [#31009](https://github.com/PaddlePaddle/Paddle/pull/31009), [#31077](https://github.com/PaddlePaddle/Paddle/pull/31077), and more)
- Add the support of Ascend chips: support for single hosting, multiple accelerators training on Ascend NPUs.  ([#31957](https://github.com/PaddlePaddle/Paddle/pull/31957), [#32381](https://github.com/PaddlePaddle/Paddle/pull/32381),  [#32197](https://github.com/PaddlePaddle/Paddle/pull/32197),  and more)
- Kunlun hardware training support
  - Kunlun XPU supports dynamic graph distributed training. ([#30455](https://github.com/PaddlePaddle/Paddle/pull/30455),  [#30671](https://github.com/PaddlePaddle/Paddle/pull/30671))
  - Kunlun XPU supports fleet distributed training. ([#30858](https://github.com/PaddlePaddle/Paddle/pull/30858))
  - Kunlun XPU supports spawn to start multi-card training and optimize XPU dynamic graph multi-card performance. ([#31130](https://github.com/PaddlePaddle/Paddle/pull/31130))
  - Kunlun XPU static graph multi-card supports the optimization of fuse allreduce and gradient merge. ([#31104](https://github.com/PaddlePaddle/Paddle/pull/31104))
  - Support Kunlun XPU in the exposure of all\_reduce/reduce collection communication API. ([#32303](https://github.com/PaddlePaddle/Paddle/pull/32302))
  - Fix the bug of the random hang of Kunlun XPU dynamic graph multi-card. ([#32662](https://github.com/PaddlePaddle/Paddle/pull/32662))

## Thanks to our Contributors

This release contains contributions from:

123malin, Adam Osewski, alncat,  arlesniak, AshburnLee, Aurelius84, Bai Yifan, Baibaifan, Bin Lu, cc, ceci3, chajchaj, chalsliu, channings, Chen Long, Chen Weihang, chen zhiyu, Chengmo, chentianyu03, cnn, CtfGo, cucuzg, danleifeng, denglin-github, Double\_V, fangshuixun007, Feiyu Chan, fluffyrita, FlyingQianMM, FNRE, furnace, GaoWei8, GeminiCarrie, gongweibao, Gradie, GT-Zhang, Guanghua Yu, Guo Sheng, guofei, hong, houj04, huangjun12, huangxu96, Huihuang Zheng, hutuxian, iducn, Jacek Czaja, Jack Zhou, jakpiase, JamesLim, Jiabin Yang, jiangcheng, Jiaqi Liu, Jiawei Wang, joanna.wozna.intel, joejiong, JZ-LIANG, Kaipeng Deng, Kqnonrime, kuizhiqing, Lei.C, Leo Chen, lidanqing, LielinJiang, lijianshe02, lilong12, limingshu, littletomatodonkey, liu zhengxi, LiuChiachi, liuyuhui, liym27, LoveAn, LutaoChu, minghaoBD, mls1999725, niuliling123, Ouyang Chao, pangyoki, parap1uie-s, Pei Yang, procr, Qi Li, qingqing01, QingshuChen, Ren Wei (任卫), ronnywang, ruri, seemingwang, Shang Zhizhou, shanliang1992, ShenLiang, Shibo Tao, Steffy-zxf, syyxsxx, taixiurong, tangwei12, Tao Luo, Thomas Young, Thunderbrook, tianshuo78520a, TTerror, wangchaochaohu, wangguanzhong, wanghuancoder, wangna11BD, WangXi, wangxinxin08, wawltor, Wei Shengyu, weihaoji, WeiXin, wenbin, Wenyu, whs, Wilber, winter-wang, Wojciech Uss, wuhuanzhou, wuyefeilin, XGZhang, XiangGao, XiaoguangHu, xiaoting, xiegegege, xiemoyuan, xingfeng01, Yang Zhang, yaoxuefeng, yiak, yingshengBD, yinhaofeng, Yiqun Liu, ykkk2333, yongqiangma, Yuang Liu, yukavio, YUNSHEN XIE, Y_Xuan, Zhang Jun, Zhang Ting, zhang wenhui, Zhang Zheng, zhangchunle, Zhen Wang, zhiboniu, Zhong Hui, Zhou Wei, zhulei, zhupengyang, zlsh80826, 卖鱼的哲学, 石晓伟

---

### Doc About About Us (`doc/about/about_us.rst`)

=========
关于我们
=========

什么是PaddlePaddle
--------------------

- PaddlePaddle是百度自主研发并开源的深度学习框架，它能够让开发者和企业安全、快速地实现自己的AI想法

- 项目团队汇聚了全球优秀的深度学习科学家，致力于为开发者和企业提供流畅的深度学习研发体验

- 框架具有易学、易用、安全、高效四大特性，是非常适合中国开发者和企业的深度学习工具

PaddlePaddle的技术特色
-------------------------

- 新一代深度学习框架： PaddlePaddle是基于“深度学习编程语言”的新一代深度学习框架，在保证性能的同时，极大的提升了框架对模型的表达能力，能够描述任意潜在可能出现的模型

- 对大规模计算更加友好：经过百度内多种大规模计算业务的打磨，PaddlePaddle在分布式计算上表现优异，基于EDL技术能够节约大量计算资源，同时也能支持大规模稀疏模型的训练

- 提供可视化的深度学习：通过Visual DL可以帮助开发者方便的观测训练整体趋势、数据样本质量和中间结果、参数分布和变化趋势、以及模型的结构，帮助开发者更便捷的完成编程过程

提供基于PaddlePaddle的教育体系
--------------------------------

- 深度学习课程：百度与中国市场优秀的教育、培训机构共同开发了深度学习精品课程以及学习教材，帮助开发者从零掌握深度学习

- 深度学习实训：对于目的是科研和学习的用户，PaddlePaddle提供了无需安装、线上运行的开发环境，并提供算法、算力、数据支持

- 线下培训：提供丰富、高质量的线下教育活动，如青年教师培训、线下实战营、沙龙等多种形式的培训和交流


提供基于PaddlePaddle的AI服务
------------------------------

- EadyDL：可以帮助零算法基础的企业快速完成一个深度学习任务，只需少量的数据即可得到优质的模型

- AI市场：提供标准化的AI 能力、产品的交易机制，帮助企业快速找到所需，有效开展AI业务

- 深度学习竞赛： PaddlePaddle汇聚优秀深度学习开发者，企业可以发布自己的商业问题，通过竞赛方式快速找到最优的解决方案

你对PaddlePaddle有任何的问题都可以通过以下方式联系到我们
-----------------------------------------------------------

- 学习/使用问题：可以在 `PaddlePaddle开源社区 <https://github.com/PaddlePaddle/Paddle/issues>`_，以及 `PaddlePaddle中文社区 <http://ai.baidu.com/forum/topic/list/168>`_ 向我们反馈

- 对PaddlePaddle框架发展的建议：可发送邮件至Paddle-better@baidu.com

我们期待与你一起打造世界级的深度学习框架，共同推动AI技术的进步



PaddlePaddle团队

---

### Doc Fluid Index En (`doc/fluid/index_en.rst`)

PaddlePaddle Fluid
==========================

..  toctree::
    :maxdepth: 1
	
    install/index_en.rst
    beginners_guide/index_en.rst
    advanced_guide/index_en.rst
    api/index_en.rst
    release_note_en.md

---

### Doc Fluid Release Note Cn (`doc/fluid/release_note_cn.md`)

# Release Note

## 重要声明

 - 此版本为测试版，还在迭代中，目前还没有稳定，后续API会根据反馈有可能进行不兼容的升级。对于想要体验飞桨最新特性的开发者，欢迎试用此版本；对稳定性要求高的工业级应用场景推荐使用Paddle 1.8稳定版本。

 - 此版本主推命令式编程模式（动态图）的开发方式，并提供了高层API的封装。命令式编程模式具有很好的灵活性，高层API可以大幅减少重复代码。对于初学者或基础的任务场景，推荐使用高层API的开发方式，简单易用；对于资深开发者想要实现复杂的功能，推荐使用命令式编程模式的API，灵活高效。

 - 此版本同时对飞桨的API目录体系做了优化，原目录下API会建立alias仍然可用，但建议新的程序使用新目录结构。

## 基础框架

### 基础API

- 组网类API实现动静统一，支持在命令式编程模式和声明式编程模式（静态图）两种模式下运行
- API目录结构调整，Paddle 1.x 版本的API主要位于paddle.fluid目录，本版本对API目录结构进行调整，使得分类更为合理，具体调整规则如下：
  - 原fluid.layers下跟tensor操作相关的API移动到paddle.tensor目录
  - 原fluid.layers下跟组网相关的操作移动到paddle.nn目录，带有参数的类型放到paddle.nn.layers目录，函数式的API放到paddle.nn.functional目录
  - 原fluid.dygraph下命令式编程模式专用API移动到paddle.imperative目录
  - 创建paddle.framework目录，用来存放跟框架相关的Program, Executor等API
  - 创建paddle.distributed目录，用来存放分布式相关的API
  - 创建paddle.optimizer目录，用来存放优化算法相关的API
  - 创建paddle.metric目录，用来创建评估指标计算相关的API
  - 创建paddle.incubate目录，用来存放孵化中的代码，其中的API有可能会发生调整，该目录存放了复数计算complex和高层API相关的代码
  - 所有在paddle.tensor和paddle.framework目录下的API，在paddle目录下创建别名，比如：paddle.tensor.creation.ones可以使用paddle.ones别名

- 新增API如下：
  - 在paddle.nn目录新增8个组网类的API: interpolate, LogSoftmax, ReLU, Sigmoid, loss.BCELoss, loss.L1Loss, loss.MSELoss, loss.NLLLoss
  - 在paddle.tensor目录新增59个Tensor相关API：add, addcmul, addmm, allclose, arange, argmax, atan, bmm, cholesky, clamp, cross, diag_embed, dist, div, dot, elementwise_equal, elementwise_sum, equal, eye, flip, full, full_like, gather, index_sample, index_select, linspace, log1p, logsumexp, matmul, max, meshgrid, min, mm, mul, nonzero, norm, ones, ones_like, pow, randint, randn, randperm, roll, sin, sort, split, sqrt, squeeze, stack, std, sum, t, tanh, tril, triu, unsqueeze, where, zeros, zeros_like
  - 新增device_guard用来指定设备，新增seed用来初始化随机数种子

- 部分原fluid目录下API，并没有迁移到paddle目录下
  - 原fluid.contrib目录下API，保留在原位置，未迁移：BasicGRUUnit, BasicLSTMUnit, BeamSearchDecoder, Compressor, HDFSClient, InitState, QuantizeTranspiler, StateCell, TrainingDecoder, basic_gru, basic_lstm, convert_dist_to_sparse_program, ctr_metric_bundle, extend_with_decoupled_weight_decay, fused_elemwise_activation, fused_embedding_seq_pool, load_persistables_for_increment, load_persistables_for_inference, match_matrix_tensor, memory_usage, mixed_precision.AutoMixedPrecisionLists, mixed_precision.decorate, multi_download, multi_upload, multiclass_nms2, op_freq_statistic, search_pyramid_hash, sequence_topk_avg_pooling, shuffle_batch, tree_conv, var_conv_2d
  - 原LodTensor相关的API，目前还在开发中，暂未迁移：LoDTensor, LoDTensorArray, create_lod_tensor, create_random_int_lodtensor, DynamicRNN, array_length, array_read, array_write, create_array, ctc_greedy_decoder, dynamic_gru, dynamic_lstm, dynamic_lstmp, im2sequence, linear_chain_crf, lod_append, lod_reset, sequence_concat, sequence_conv, sequence_enumerate, sequence_expand, sequence_expand_as, sequence_first_step, sequence_last_step, sequence_mask, sequence_pad, sequence_pool, sequence_reshape, sequence_reverse, sequence_scatter, sequence_slice, sequence_softmax, sequence_unpad, tensor_array_to_tensor
  - 原fluid下分布式相关API，目前还在开发中，暂未迁移
  - 原fluid目录以下API，将在高层API中重新实现，未迁移：nets.glu, nets.img_conv_group, nets.scaled_dot_product_attention, nets.sequence_conv_pool, nets.simple_img_conv_pool
  - 原fluid目录以下API，有待进一步完善，暂未迁移：dygraph.GRUUnit, layers.DecodeHelper, layers.GreedyEmbeddingHelper, layers.SampleEmbeddingHelper, layers.TrainingHelper, layers.autoincreased_step_counter, profiler.cuda_profiler, profiler.profiler, profiler.reset_profiler, profiler.start_profiler, profiler.stop_profiler
  - 原fluid目录以下API不再推荐使用，未迁移：DataFeedDesc, DataFeeder, clip.ErrorClipByValue, clip.set_gradient_clip, dygraph_grad_clip.GradClipByGlobalNorm, dygraph_grad_clip.GradClipByNorm, dygraph_grad_clip.GradClipByValue, initializer.force_init_on_cpu, initializer.init_on_cpu, io.ComposeNotAligned.with_traceback, io.PyReader, io.load_params, io.load_persistables, io.load_vars, io.map_readers, io.multiprocess_reader, io.save_params, io.save_persistables, io.save_vars, io.xmap_readers, layers.BasicDecoder, layers.BeamSearchDecoder, layers.Decoder, layers.GRUCell, layers.IfElse, layers.LSTMCell, layers.RNNCell, layers.StaticRNN, layers.Switch, layers.While, layers.create_py_reader_by_data, layers.crop, layers.data, layers.double_buffer, layers.embedding, layers.fill_constant_batch_size_like, layers.gaussian_random_batch_size_like, layers.get_tensor_from_selected_rows, layers.load, layers.merge_selected_rows, layers.one_hot, layers.py_reader, layers.read_file, layers.reorder_lod_tensor_by_rank, layers.rnn, layers.uniform_random_batch_size_like, memory_optimize, release_memory, transpiler.memory_optimize, transpiler.release_memory

### 高层API

- 新增paddle.incubate.hapi目录，对模型开发过程中常见的组网、训练、评估、预测、存取等操作进行封装，实现低代码开发，MNIST手写数字识别任务对比命令式编程模式实现方式，高层API可减少80%执行类代码。
- 新增Model类封装，继承Layer类，封装模型开发过程中常用的基础功能，包括：
  - 提供prepare接口，用于指定损失函数和优化算法
  - 提供fit接口，实现训练和评估，可通过callback方式实现训练过程中执行自定义功能，比如模型存储等
  - 提供evaluate接口，实现评估集上的预测和评估指标计算
  - 提供predict接口，实现特定的测试数据推理预测
  - 提供train_batch接口，实现单batch数据的训练
- 新增Dataset接口，对常用数据集进行封装，支持数据的随机访问
- 新增常见Loss和Metric类型的封装
- 新增CV领域Resize, Normalize等16种常见的数据处理接口
- 新增CV领域lenet, vgg, resnet, mobilenetv1, mobilenetv2图像分类骨干网络
- 新增NLP领域MultiHeadAttention, BeamSearchDecoder, TransformerEncoder, TransformerDecoder , DynamicDecode API接口
- 发布基于高层API实现的12个模型，Transformer，Seq2seq,  LAC，BMN, ResNet,  YOLOv3, , VGG, MobileNet, TSM, CycleGAN,  Bert, OCR

### 性能优化

- 新增`reshape+transpose+matmul` fuse，使得Ernie量化后 INT8 模型在原来基础上性能提升~4%（在6271机器上）。量化后INT8模型相比未经过DNNL优化（包括fuses等）和量化的FP32模型提速~6.58倍

### 调试分析

- 针对Program打印内容过于冗长，在调试中利用效率不高的问题，大幅简化Program、Block、Operator、Variable等对象的打印字符串，不损失有效信息的同时提升调试效率
- 针对第三方库接口`boost::get`不安全，运行中抛出异常难以调试的问题，增加`BOOST_GET`系列宏替换了Paddle中600余处存在风险的`boost::get`，丰富出现`boost::bad_get`异常时的报错信息，具体增加了C++报错信息栈，出错文件及行号、期望输出类型和实际类型等，提升调试体验

## Bug修复

 - 修复while loop中存在slice操作时计算结果错误的bug
 - 修复inplace ops引起的transformer 模型性能下降问题
 - 通过完善cache key, 解决Ernie精度测试最后一个batch运行失败的问题
 - 修复fluid.dygraph.guard等context中出现异常时无法正确退出的问题

---

### Doc Survey Op Fusion Design (`doc/survey/op_fusion_design.md`)

# Operator fusion  
Fusing multiple operators together is an important method to optimize the program execution, particularly for GPU or other specialized accelerators. An obvious benefit is to avoid the overhead of saving the intermediate result back into global memory.   

There are generally two ways to fuse operators, fusing directly connected operators and fusing non directly connected operators. The first method is mainly used by [NNVM Compiler](https://github.com/dmlc/tvm/) and [XLA](https://www.tensorflow.org/performance/xla/). The second method is mainly used by Dynet and TensorFlow Fold to do auto-batching. The principle of fusing operator is according to some rules to combine multiple operations into one, for example, `Y = X * W` and `Z = Y + B` can be fused to `Z = X * W + B`, and `Y1 = X1 * W` and `Y2 = X2 * W` can be fused to `[Y1;Y2] = [X1;X2] * W`. In order to get a short-term profit, we decided to try to manually specify these rules.   

## Challenge
The challenge of fusing operators is:
  - how to make the rules.
  - how to implement these rules efficiently.

### How to make the rules?

The problem of determining the best single location for a fusion operator is an NP-hard combinatorial problem. After analysis the operators of the DL model, we found there are two group of operators can be fused explicitly, one is the simple and adjacent operations, for example, `tmp = x + y` and `z = Relu(tmp)`, and the other is the operators that have the same function, for example, a serials of `SGD` or `Momentum`. They usually appear in the model in a large number. So we should think about how to fuse them separately first.

### How to implement these rules efficiently?
#### How to fuse the adjacent operations efficiently?
Here we use a template function to represent the fused operations. The pros of using a template function are that it is simple and efficient, and the cons are that it is not easy to expand, and it can only be used to express some simple operations. So taking into account our current needs, the template function is more appropriate.

#### How to fuse the operators that have the same function efficiently?
We take SGD operator as an example, the training model may have hundreds of parameters and correspondingly have the same number of SGD operators. The expression(`w = w - lr*w_g`) of those operators is the same, so during of training, the executor will execute this expression hundreds time in CPU or other specialized accelerators. If we can fuse them and make the address of all `w` and all `w_g` continuous respectively, we only need execute one time. For some accelerators, the time of launching kernel is not neglected, so the time of hundreds of times of launching and executing kernel may be larger than launching and executing only once. There usually are many operators that similar to `SGD` in the DL model, such as `AllReduce` and `FC`.

---

### Api Guides Index En (`docs/api_guides/index_en.rst`)

=================
API Quick Search
=================

This section introduces the Fluid API structure and usage, to help you quickly get the full picture of the PaddlePaddle Fluid API. This section is divided into the following modules:

..  toctree::
    :maxdepth: 1

    low_level/program_en.rst
    low_level/layers/index_en.rst
    low_level/nets_en.rst
    low_level/optimizer_en.rst
    low_level/backward_en.rst
    low_level/metrics_en.rst
    low_level/model_save_reader_en.rst
    low_level/inference_en.rst
    low_level/memory_optimize_en.rst
    low_level/executor_en.rst
    low_level/parallel_executor_en.rst
    low_level/compiled_program_en.rst
    low_level/parameter_en.rst
    low_level/distributed/index_en.rst

---

### Design Index En (`docs/design/index_en.rst`)

Design
------------

.. toctree::
  :maxdepth: 1

  motivation/index_en.rst
  execution/index_en.rst
  concepts/index_en.rst
  data_type/index_en.rst
  memory/index_en.rst
  multi_devices/index_en.rst
  dynamic_rnn/index_en.rst
  concurrent/index_en.rst
  algorithm/index_en.rst
  network/index_en.rst
  modules/index_en.rst
  dist_train/index_en.rst

---

### Faq Install Cn (`docs/faq/install_cn.md`)

# 安装常见问题


##### 问题：使用过程中报找不到tensorrt库的日志

+ 问题描述:

> TensorRT dynamic library (libnvinfer.so) that Paddle depends on is not configured correctly. (error code is libnvinfer.so: cannot open shared object file: No such file or directory)  
> Suggestions:  
> Check if TensorRT is installed correctly and its version is matched with paddlepaddle you installed.  
> Configure TensorRT dynamic library environment variables as follows:  
> Linux: set LD_LIBRARY_PATH by export LD_LIBRARY_PATH=...  
> Windows: set PATH by `set PATH=XXX;  
+ 问题分析：

遇到该问题是因为使用的paddle默认开始了TensorRT，但是本地环境中没有找到TensorRT的库，该问题只影响使用[Paddle Inference](https://paddleinference.paddlepaddle.org.cn/master/product_introduction/inference_intro.html)开启TensorRT预测的场景，对其它方面均不造成影响。

+ 解决办法：

根据提示信息，在环境变量中加入TensorRT的库路径。

-----

##### 问题：Windows环境下，使用pip install时速度慢，如何解决？

+ 解决方案：

在pip后面加上参数`-i`指定pip源，使用国内源获取安装包。

+ 操作步骤：

1. Python2情况下，使用如下命令安装PaddlePaddle。

   ```bash
   pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple/
   ```

2. Python3情况下，使用如下命令安装PaddlePaddle。

   ```bash
   pip3 install paddlepaddle -i https://mirror.baidu.com/pypi/simple/
   ```

你还可以通过如下三个地址获取pip安装包，只需修改 `-i` 后网址即可：

1. https://pypi.tuna.tsinghua.edu.cn/simple
2. https://mirrors.aliyun.com/pypi/simple/
3. https://pypi.douban.com/simple/

------

##### 问题：使用pip install时报错，`PermissionError: [WinError 5]` ，如何解决？

+ 问题描述：

使用pip install时报错，`PermissionError: [WinError 5]` ，

`C:\\Program Files\\python35\\Lib\\site-packages\\graphviz`。

+ 报错分析：

用户权限问题导致，由于用户的Python安装到系统文件内（如`Program Files/`），任何的操作都需要管理员权限。

+ 解决方法：

选择“以管理员身份运行”运行CMD，重新执行安装过程, 使用命令`pip install paddlepaddle`。

------

##### 问题： 使用pip install时报错，`ERROR: No matching distribution found for paddlepaddle` ，如何解决？

+ 问题描述：

使用pip install时报错，`ERROR: Could not find a version that satisfies the requirement paddlepaddle (from versions: none)`

`ERROR: No matching distribution found for paddlepaddle`

<img src="https://agroup-bos-bj.cdn.bcebos.com/bj-febb18fb78004dc17f18d60a009dc6a8bd907251" alt="图片" />

+ 报错分析：

Python版本不匹配导致。用户使用的是32位Python，但是对应的32位pip没有PaddlePaddle源。

+ 解决方法：

请用户使用64位的Python进行PaddlePaddle安装。

------

##### 问题： 本地使用import paddle时报错，`ModuleNotFoundError:No module named ‘paddle’`，如何解决？

+ 报错分析：

原因在于用户的计算机上可能安装了多个版本的Python，而安装PaddlePaddle时的Python和`import paddle`时的Python版本不一致导致报错。如果用户熟悉PyCharm等常见的IDE配置包安装的方法，配置运行的方法，则可以避免此类问题。

+ 解决方法：

用户明确安装Paddle的python位置，并切换到该python进行安装。可能需要使用`python -m pip install paddlepaddle`命令确保paddle是安装到该python中。

------

##### 问题： 使用PaddlePaddle GPU的Docker镜像时报错， `Cuda Error: CUDA driver version is insufficient for CUDA runtime version`，如何解决？

+ 报错分析：

机器上的CUDA驱动偏低导致。

+ 解决方法：

需要升级CUDA驱动解决。

1. Ubuntu和CentOS环境，需要把相关的驱动和库映射到容器内部。如果使用GPU的docker环境，需要用nvidia-docker来运行，更多请参考[nvidia-docker](https://github.com/NVIDIA/nvidia-docker)。

2. Windows环境，需要升级CUDA驱动。

------

##### 问题： 使用PaddlePaddle时报错，`Error: no CUDA-capable device is detected`，如何解决？

+ 报错分析：

CUDA安装错误导致。

+ 解决方法：

查找“libcudart.so”所在目录，并将其添加到`LD_LIBRARY_PATH`中。

例如：执行`find / -name libcudart.so`, 发现libcudart.so在`/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudart.so`路径下， 使用如下命令添加即可。

```bash
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudart.so:${LD_LIBRARY_PATH}
```

------

##### 问题： 如何升级PaddlePaddle？

+ 答复：

1. GPU环境：

  ```bash
  pip install -U paddlepaddle-gpu
  ```

或者

  ```bash
  pip install paddlepaddle-gpu==需要安装的版本号（如2.0）
  ```

2. CPU环境：

  ```bash
  pip install -U paddlepaddle
  ```
或者

  ```bash
  pip install paddlepaddle==需要安装的版本号（如2.0）
  ```

------

##### 问题： 在GPU上如何选择PaddlePaddle版本？

+ 答复：

首先请确定你本机的CUDA、cuDNN版本，飞桨目前pip安装适配CUDA版本9.0/10.0/10.1/10.2/11.0，CUDA9.0/10.0/10.1/10.2 配合 cuDNN v7.6.5+，CUDA 工具包11.0配合cuDNN v8.0.4。请确定你安装的是适合的版本。更多安装信息见[官网安装文档](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/2.0/install/pip/windows-pip.html)

------

##### 问题： import paddle报错, dlopen: cannot load any more object with static TLS, 如何解决？

+ 答复：

glibc版本过低，建议使用官方提供的docker镜像或者将glibc升级到2.23+。

---

### Install Tables (`docs/install/Tables.md`)

<a name="third_party"></a>
# 附录

## **编译依赖表**

<p align="center">
<table>
    <thead>
    <tr>
        <th> 依赖包名称 </th>
        <th> 版本 </th>
        <th> 说明 </th>
        <th> 安装命令 </th>
    </tr>
    </thead>
    <tbody>
    <tr>
        <td> CMake </td>
        <td> 3.15, 3.16(推荐),3.17 </td>
        <td>  </td>
        <td>  </td>
    </tr>
    <tr>
        <td> GCC (Linux Only) </td>
        <td> 5.4 / 8.2 </td>
        <td>  推荐使用CentOS的devtools2 </td>
        <td>  </td>
    </tr>
    <tr>
        <td> Clang (MacOS Only) </td>
        <td> 9.0及以上 </td>
        <td> 通常使用MacOS 10.11及以上的系统对应的Clang版本即可 </td>
        <td>  </td>
    </tr>
        <tr>
        <td> Python（64 bit） </td>
        <td> 3.6+.x </td>
        <td> 依赖 libpython3.6+.so </td>
        <td> 请访问<a href="https://www.python.org">Python 官网</a></td>
    </tr>
    <tr>
        <td> SWIG </td>
        <td> 最低 2.0 </td>
        <td>  </td>
        <td> <code>apt install swig </code> 或 <code> yum install swig </code> </td>
    </tr>
    <tr>
        <td> wget </td>
        <td> any </td>
        <td>  </td>
        <td> <code> apt install wget </code>  或 <code> yum install wget </code> </td>
    </tr>
    <tr>
        <td> openblas </td>
        <td> any </td>
        <td> 可选 </td>
        <td>  </td>
    </tr>
    <tr>
        <td> pip </td>
        <td> >=20.2.2 </td>
        <td>  </td>
        <td> <code> apt install python-pip </code> 或 <code> yum install python-pip </code> </td>
    </tr>
    <tr>
        <td> numpy </td>
        <td> >=1.12.0 </td>
        <td>  </td>
        <td> <code> pip install numpy </code> </td>
    </tr>
    <tr>
        <td> protobuf </td>
        <td> >=3.1.0 </td>
        <td>  </td>
        <td> <code> pip install protobuf </code> </td>
    </tr>
    <tr>
        <td> wheel </td>
        <td> any </td>
        <td>  </td>
        <td> <code> pip install wheel </code> </td>
    </tr>
    <tr>
        <td> patchELF </td>
        <td> any </td>
        <td>  </td>
        <td> <code> apt install patchelf </code> 或参见github <a href="https://gist.github.com/ruario/80fefd174b3395d34c14">patchELF 官方文档</a></td>
    </tr>
    <tr>
        <td> go </td>
        <td> >=1.8 </td>
        <td> 可选 </td>
        <td>  </td>
    </tr>
    <tr>
        <td> setuptools </td>
        <td> >= 28.0.0 </td>
        <td> </td>
        <td>  </td>
    </tr>
    <tr>
        <td> unrar </td>
        <td>  </td>
        <td> </td>
        <td> brew install unrar (For MacOS), apt-get install unrar (For Ubuntu) </td>
    </tr>
    </tbody>
</table>
</p>


<a name="Compile"></a>
</br></br>
## **编译选项表**

<p align="center">
<table>
    <thead>
    <tr>
        <th> 选项 </th>
        <th> 说明 </th>
        <th> 默认值 </th>
    </tr>
    </thead>
    <tbody>
    <tr>
        <td> WITH_GPU </td>
        <td> 是否支持CUDA </td>
        <td> ON </td>
    </tr>
    <tr>
        <td> WITH_ROCM </td>
        <td> 是否支持ROCM </td>
        <td> OFF </td>
    </tr>
    <tr>
        <td> WITH_AVX </td>
        <td> 是否编译含有AVX指令集的PaddlePaddle二进制文件 </td>
        <td> ON </td>
    </tr>
    <tr>
        <td> WITH_PYTHON </td>
        <td> 是否内嵌PYTHON解释器 </td>
        <td> ON </td>
    </tr>
    <tr>
        <td> WITH_TESTING </td>
        <td> 是否开启单元测试 </td>
        <td> OFF </td>
    </tr>
    <tr>
        <td> WITH_MKL </td>
        <td> 是否使用MKL数学库，如果为否则是用OpenBLAS </td>
        <td> ON </td>
    </tr>
    <tr>
        <td> WITH_SYSTEM_BLAS </td>
        <td> 是否使用系统自带的BLAS </td>
        <td> OFF </td>
    </tr>
    <tr>
        <td> WITH_DISTRIBUTE </td>
        <td> 是否编译带有分布式的版本 </td>
        <td> OFF </td>
    </tr>
    <tr>
        <td> WITH_BRPC_RDMA </td>
        <td> 是否使用BRPC RDMA作为RPC协议 </td>
        <td> OFF </td>
    </tr>
        <tr>
        <td> ON_INFER </td>
        <td> 是否打开预测优化 </td>
        <td> OFF </td>
    </tr>
    <tr>
        <tr>
        <td> CUDA_ARCH_NAME </td>
        <td> 是否只针对当前CUDA架构编译 </td>
        <td> All:编译所有可支持的CUDA架构 可选：Auto 自动识别当前环境的架构编译 </td>
    </tr>
    <tr>
        <tr>
        <td> TENSORRT_ROOT </td>
        <td> 指定TensorRT路径 </td>
        <td> Windows下默认值为'/'，Linux下默认值为 '/usr/' </td>
    </tr>
   </tbody>
</table>
</p>





**BLAS**

PaddlePaddle支持 [MKL](https://software.intel.com/en-us/mkl) 和 [OpenBlAS](http://www.openblas.net) 两种BLAS库。默认使用MKL。如果使用MKL并且机器含有AVX2指令集，还会下载MKL-DNN数学库，详细参考[这里](https://github.com/PaddlePaddle/Paddle/tree/release/0.11.0/doc/design/mkldnn#cmake) 。

如果关闭MKL，则会使用OpenBLAS作为BLAS库。

**CUDA/cuDNN**

PaddlePaddle在编译时/运行时会自动找到系统中安装的CUDA和cuDNN库进行编译和执行。 使用参数 `-DCUDA_ARCH_NAME=Auto` 可以指定开启自动检测SM架构，加速编译。

PaddlePaddle可以使用cuDNN v5.1之后的任何一个版本来编译运行，但尽量请保持编译和运行使用的cuDNN是同一个版本。 我们推荐使用最新版本的cuDNN。

**编译选项的设置**

PaddePaddle通过编译时指定路径来实现引用各种BLAS/CUDA/cuDNN库。cmake编译时，首先在系统路径（ `/usr/lib` 和 `/usr/local/lib` ）中搜索这几个库，同时也会读取相关路径变量来进行搜索。 通过使用`-D`命令可以设置，例如：

> `cmake .. -DWITH_GPU=ON -DWITH_TESTING=OFF -DCUDNN_ROOT=/opt/cudnnv5`

**注意**：这几个编译选项的设置，只在第一次cmake的时候有效。如果之后想要重新设置，推荐清理整个编译目录（ rm -rf ）后，再指定。


<a name="whls"></a>
</br></br>
## **安装包列表**

<p align="center">
<table>
    <thead>
    <tr>
        <th> 版本号 </th>
        <th> 版本说明 </th>
    </tr>
    </thead>
    <tbody>
    <tr>
        <td> paddlepaddle==[版本号] 例如 paddlepaddle==2.1.1 </td>
        <td> 只支持CPU对应版本的PaddlePaddle，具体版本请参见<a href=https://pypi.org/project/paddlepaddle/#history>Pypi</a> </td>
    </tr>
    <tr>
        <td> paddlepaddle-gpu==[版本号] 例如 paddlepaddle-gpu==2.1.1 </td>
        <td> 默认安装支持CUDA 10.2和cuDNN 7的对应[版本号]的PaddlePaddle安装包 </td>
    </tr>
   </tbody>
</table>
</p>

您可以在 [Release History](https://pypi.org/project/paddlepaddle-gpu/#history) 中找到PaddlePaddle-gpu的各个发行版本。
> 其中`postXX` 对应的是CUDA和cuDNN的版本，`postXX`之前的数字代表Paddle的版本

需要注意的是，命令中<code> paddlepaddle-gpu==2.1.1 </code> 在windows环境下，会默认安装支持CUDA 10.2和cuDNN 7的对应[版本号]的PaddlePaddle安装包

<a name="ciwhls-release"></a>
</br></br>

## **多版本whl包列表-Release**

<p align="center">
<table>
    <thead>
    <tr>
        <th> 版本说明 </th>
        <th> cp36-cp36m    </th>
        <th> cp37-cp37m    </th>
        <th> cp38-cp38    </th>
        <th> cp39-cp39    </th>
    </tr>
    </thead>
    <tbody>
    <tr>
        <td> cpu-mkl-avx </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-cpu-avx-mkl/paddlepaddle-2.1.1-cp36-cp36m-linux_x86_64.whl"> paddlepaddle-2.1.1-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-cpu-avx-mkl/paddlepaddle-2.1.1-cp37-cp37m-linux_x86_64.whl"> paddlepaddle-2.1.1-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-cpu-avx-mkl/paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl"> paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-cpu-avx-mkl/paddlepaddle-2.1.1-cp39-cp39-linux_x86_64.whl"> paddlepaddle-2.1.1-cp39-cp39-linux_x86_64.whl</a></td>
    </tr>
    <tr>
        <td> cpu-openblas-avx </td>
        <td> - </td>
        <td> - </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-cpu-avx-openblas/paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl"> paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cpu-mkl-noavx </td>
        <td> - </td>
        <td> - </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-cpu-noavx-mkl/paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl"> paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cpu-openblas-noavx </td>
        <td> - </td>
        <td> - </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-cpu-noavx-openblas/paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl"> paddlepaddle-2.1.1-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda10.1-cudnn7-mkl-gcc5.4-avx </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc5.4/paddlepaddle_gpu-2.1.1.post101-cp36-cp36m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc5.4/paddlepaddle_gpu-2.1.1.post101-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc5.4/paddlepaddle_gpu-2.1.1.post101-cp38-cp38-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp38-cp38-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc5.4/paddlepaddle_gpu-2.1.1.post101-cp39-cp39-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp39-cp39-linux_x86_64.whl</a></td>
    </tr>
    <tr>
        <td> cuda10.1-cudnn7-mkl-gcc5.4-noavx </td>
        <td> - </td>
        <td> - </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.1-cudnn7-mkl-noavx-gcc5.4/paddlepaddle_gpu-2.1.1.post101-cp38-cp38-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda10.1-cudnn7.6-trt6.0.1.5 </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc5.4/paddlepaddle_gpu-2.1.1.post101-cp36-cp36m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc5.4/paddlepaddle_gpu-2.1.1.post101-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> - </td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda10.1-cudnn7-mkl-gcc8.2-avx-trt6.0.1.5 </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post101-cp36-cp36m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post101-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post101-cp38-cp38-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp38-cp38-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.1-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post101-cp39-cp39-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post101-cp39-cp39-linux_x86_64.whl</a></td>
    </tr>
    <tr>
        <td> cuda10.2-cudnn7-mkl-gcc8.2-avx </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.2-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1-cp36-cp36m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.2-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.2-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1-cp38-cp38-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1-cp38-cp38-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.2-cudnn7-mkl-gcc8.2/paddlepaddle_gpu-2.1.1-cp39-cp39-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1-cp39-cp39-linux_x86_64.whl</a></td>
    </tr>
    <tr>
        <td> cuda10.2-cudnn7-mkl-gcc8.2-noavx </td>
        <td> - </td>
        <td> - </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda10.2-cudnn7-mkl-noavx-gcc8.2/paddlepaddle_gpu-2.1.1-cp38-cp38-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda10.2-cudnn8-mkl-gcc8.2-trt7.1.3.4 </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.2-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1-cp36-cp36m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda10.2-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> - </td>
        <td> - </td>
    </tr>
    <tr>
        <td> jetpack4.4-cuda10.2-cudnn8-trt7.1.3 </td>
        <td> <a href="https://paddle-inference-lib.bj.bcebos.com/2.1.1-nv-jetson-jetpack4.4-all/paddlepaddle_gpu-2.1.1-cp36-cp36m-linux_aarch64.whl">
        paddlepaddle_gpu-2.1.1-cp36-cp36m-linux_aarch64.whl</a></td>
        <td> - </td>
        <td> <a href="https://paddle-inference-lib.bj.bcebos.com/2.1.1-nv-jetson-jetpack4.4-all/paddlepaddle_gpu-2.1.1-cp38-cp38-linux_aarch64.whl">
        paddlepaddle_gpu-2.1.1-cp38-cp38-linux_aarch64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda11.0-cudnn8-mkl-gcc8.2-avx-trt7.1.3 </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda11.0-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post110-cp36-cp36m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post110-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda11.0-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post110-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post110-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda11.0-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post110-cp38-cp38-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post110-cp38-cp38-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/with-trt/2.1.1-gpu-cuda11.0-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post110-cp39-cp39-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post110-cp39-cp39-linux_x86_64.whl</a></td>
    </tr>
    <tr>
        <td> cuda11.2-cudnn8-mkl-gcc8.2-avx </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda11.2-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post112-cp36-cp36m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post112-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda11.2-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post112-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post112-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda11.2-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post112-cp38-cp38-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post112-cp38-cp38-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-cuda11.2-cudnn8-mkl-gcc8.2/paddlepaddle_gpu-2.1.1.post112-cp39-cp39-linux_x86_64.whl">
        paddlepaddle_gpu-2.1.1.post112-cp39-cp39-linux_x86_64.whl</a></td>
    </tr>
    <tr>
        <td> rocm4.0.1-miopen2.11-mkl </td>
        <td> - </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/2.1.1-gpu-rocm4.0.1-miopen2.11-mkl/paddlepaddle_rocm-2.1.1.rocm401.miopen211-cp37-cp37m-linux_x86_64.whl">
        paddlepaddle_rocm-2.1.1.rocm401.miopen211-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> - </td>
        <td> - </td>
    </tr>
   </tbody>
</table>
</p>

### 表格说明

- 纵轴

cpu-mkl: 支持CPU训练和预测，使用Intel mkl数学库

cuda10_cudnn7-mkl: 支持GPU训练和预测，使用Intel mkl数学库


- 横轴

一般是类似于“cp37-cp37m”的形式，其中：

37:python tag,指python3.7，类似的还有“36”、“38”、“39”等

mu:指unicode版本python，若为m则指非unicode版本python

- 安装包命名规则

每个安装包都有一个专属的名字，它们是按照Python的官方规则 来命名的，形式如下：

{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl

其中build tag可以缺少，其他部分不能缺少

distribution: wheel名称version: 版本，例如0.14.0 (要求必须是数字格式)

python tag: 类似'py36', 'py37', 'py38', 'py39'，用于标明对应的python版本

abi tag:  类似'cp33m', 'abi3', 'none'

platform tag: 类似 'linux_x86_64', 'any'

<a name="ciwhls"></a>
</br></br>
## **多版本whl包列表-develop**
<p align="center">
<table>
    <thead>
    <tr>
        <th> 版本说明 </th>
        <th> cp36-cp36m    </th>
        <th> cp37-cp37m    </th>
        <th> cp38-cp38    </th>
        <th> cp39-cp39    </th>
    </tr>
    </thead>
    <tbody>
    <tr>
        <td> cpu-mkl </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-mkl/paddlepaddle-0.0.0-cp36-cp36m-linux_x86_64.whl"> paddlepaddle-latest-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-mkl/paddlepaddle-0.0.0-cp37-cp37m-linux_x86_64.whl"> paddlepaddle-latest-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-mkl/paddlepaddle-0.0.0-cp38-cp38-linux_x86_64.whl"> paddlepaddle-latest-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cpu-openblas </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-openblas/paddlepaddle-0.0.0-cp36-cp36m-linux_x86_64.whl"> paddlepaddle-latest-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-openblas/paddlepaddle-0.0.0-cp37-cp37m-linux_x86_64.whl"> paddlepaddle-latest-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-openblas/paddlepaddle-0.0.0-cp38-cp38-linux_x86_64.whl"> paddlepaddle-latest-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda10.1-cudnn7-mkl </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda10.1-cudnn7-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post101-cp36-cp36m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda10.1-cudnn7-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post101-cp37-cp37m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda10.1-cudnn7-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post101-cp38-cp38-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda10.2-cudnn7-mkl </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda10.2-cudnn7-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post102-cp36-cp36m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda10.2-cudnn7-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post102-cp37-cp37m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda10.2-cudnn7-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post102-cp38-cp38-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda11.0-cudnn8-mkl </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda11.0-cudnn8-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post110-cp36-cp36m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda11.0-cudnn8-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post110-cp37-cp37m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda11.0-cudnn8-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post110-cp38-cp38-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp38-cp38-linux_x86_64.whl</a></td>
        <td> - </td>
    </tr>
    <tr>
        <td> cuda11.2-cudnn8-mkl </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda11.2-cudnn8-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post112-cp36-cp36m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp36-cp36m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda11.2-cudnn8-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post112-cp37-cp37m-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp37-cp37m-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda11.2-cudnn8-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post112-cp38-cp38-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp38-cp38-linux_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-gpu-cuda11.2-cudnn8-mkl_gcc8.2/paddlepaddle_gpu-0.0.0.post112-cp39-cp39-linux_x86_64.whl"> paddlepaddle_gpu-latest-cp38-cp38-linux_x86_64.whl</a></td>
    </tr>
    <tr>
        <td> mac-cpu </td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-mac/paddlepaddle-0.0.0-cp36-cp36m-macosx_10_6_intel.whl"> paddlepaddle-cp36-cp36m-macosx_10_6_intel.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-mac/paddlepaddle-0.0.0-cp37-cp37m-macosx_10_6_intel.whl"> paddlepaddle-cp37-cp37m-macosx_10_6_intel.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-mac/paddlepaddle-0.0.0-cp38-cp38-macosx_10_14_x86_64.whl"> paddlepaddle-cp38-cp38-macosx_10_14_x86_64.whl</a></td>
        <td> <a href="https://paddle-wheel.bj.bcebos.com/develop-cpu-mac/paddlepaddle-0.0.0-cp39-cp39-macosx_10_14_x86_64.whl"> paddlepaddle-cp39-cp39-macosx_10_14_x86_64.whl</a></td>
    </tr>
   </tbody>
</table>
</p>






</br></br>

## 在Docker中执行PaddlePaddle训练程序


假设您已经在当前目录（比如在/home/work）编写了一个PaddlePaddle的程序: `train.py` （可以参考
[PaddlePaddleBook](https://github.com/PaddlePaddle/book/blob/develop/01.fit_a_line/README.cn.md)
编写），就可以使用下面的命令开始执行训练：

```
cd /home/work
```
```
docker run -it -v $PWD:/work registry.baidubce.com/paddlepaddle/paddle /work/train.py
```

上述命令中，`-it` 参数说明容器已交互式运行；`-v $PWD:/work`
指定将当前路径（Linux中PWD变量会展开为当前路径的绝对路径）挂载到容器内部的:`/work`
目录: `registry.baidubce.com/paddlepaddle/paddle` 指定需要使用的容器； 最后`/work/train.py`为容器内执行的命令，即运行训练程序。

当然，您也可以进入到Docker容器中，以交互式的方式执行或调试您的代码：

```
docker run -it -v $PWD:/work registry.baidubce.com/paddlepaddle/paddle /bin/bash
```
```
cd /work
```
```
python train.py
```

**注：PaddlePaddle Docker镜像为了减小体积，默认没有安装vim，您可以在容器中执行** `apt-get install -y vim` **安装后，在容器中编辑代码。**

</br></br>

## 使用Docker启动PaddlePaddle Book教程


使用Docker可以快速在本地启动一个包含了PaddlePaddle官方Book教程的Jupyter Notebook，可以通过网页浏览。
PaddlePaddle Book是为用户和开发者制作的一个交互式的Jupyter Notebook。
如果您想要更深入了解deep learning，可以参考PaddlePaddle Book。
大家可以通过它阅读教程，或者制作和分享带有代码、公式、图表、文字的交互式文档。

我们提供可以直接运行PaddlePaddle Book的Docker镜像，直接运行：

```
docker run -p 8888:8888 registry.baidubce.com/paddlepaddle/book
```

国内用户可以使用下面的镜像源来加速访问：

```
docker run -p 8888:8888 registry.baidubce.com/paddlepaddle/book
```

然后在浏览器中输入以下网址：

```
http://localhost:8888/
```


</br></br>
## 使用Docker执行GPU训练


为了保证GPU驱动能够在镜像里面正常运行，我们推荐使用
[nvidia-docker](https://github.com/NVIDIA/nvidia-docker)来运行镜像。
请不要忘记提前在物理机上安装GPU最新驱动。

```
nvidia-docker run -it -v $PWD:/work registry.baidubce.com/paddlepaddle/paddle:latest-gpu /bin/bash
```

**注: 如果没有安装nvidia-docker，可以尝试以下的方法，将CUDA库和Linux设备挂载到Docker容器内：**

```
export CUDA_SO="$(\ls /usr/lib64/libcuda* | xargs -I{} echo '-v {}:{}') \
$(\ls /usr/lib64/libnvidia* | xargs -I{} echo '-v {}:{}')"
export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}')
docker run ${CUDA_SO} \
${DEVICES} -it registry.baidubce.com/paddlepaddle/paddle:latest-gpu
```

---

### Install Install Kunlun Zh (`docs/install/install_Kunlun_zh.md`)

# 昆仑XPU芯片安装及运行飞桨

百度昆仑AI计算处理器（Baidu KUNLUN AI Computing Processor）是百度集十年AI产业技术实践于2019年推出的全功能AI芯片。基于自主研发的先进XPU架构，为云端和边缘端的人工智能业务而设计。 百度昆仑与飞桨及其他国产软硬件强强组合，打造一个全面领先的国产化AI技术生态，部署和应用于诸多 “人工智能+“的行业领域，包括智能云和高性能计算，智慧制造、智慧城市和安防等。更多昆仑XPU芯片详情及技术指标请 [点击这里](https://cloud.baidu.com/product/kunlun.html) 。
参考以下内容可快速了解和体验昆仑XPU芯片上运行飞桨：

- [飞桨对昆仑XPU芯片的支持](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/09_hardware_support/xpu_docs/paddle_2.0_xpu_cn.html)
- [飞桨框架昆仑XPU版安装说明](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/09_hardware_support/xpu_docs/paddle_install_cn.html)
- [飞桨框架昆仑XPU版训练示例](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/09_hardware_support/xpu_docs/train_example_cn.html)
- [飞桨预测库昆仑XPU版安装及使用示例](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/09_hardware_support/xpu_docs/inference_install_example_cn.html)

---

--- METRICS ---
- Files Extracted: 20
- Estimated Token Budget: ~71506 tokens
- Canonical Reference: https://codewiki.google/github.com/lyuwenyu/docs