Paddle Documentation

1. Project Overview & Quickstart (PaddlePaddle/Paddle) Copy Chapter

File: README.md Copy Chapter

English | 简体中文 | 日本語

Welcome to the PaddlePaddle GitHub.

PaddlePaddle, as the first independent R&D deep learning platform in China, has been officially open-sourced to professional communities since 2016. It is an industrial platform with advanced technologies and rich features that cover core deep learning frameworks, basic model libraries, end-to-end development kits, tools & components as well as service platforms.
PaddlePaddle originates from industrial practices with dedication and commitments to industrialization. It has been widely adopted by a wide range of sectors including manufacturing, agriculture, enterprise service, and so on while serving more than 23.33 million developers, 760,000 companies and generating 1,100,000 models. With such advantages, PaddlePaddle has helped an increasing number of partners commercialize AI.

Installation Copy Chapter

Latest PaddlePaddle Release: 3.3 Copy Section

Our vision is to enable deep learning for everyone via PaddlePaddle.
Please refer to our release announcement to track the latest features of PaddlePaddle.

Install Latest Stable Release or Nightly Release Copy Section

For detailed information about installation, please view Quick Install

**PaddlePaddle New Generation Framework 3.2** Copy Chapter

• Unified Dynamic/Static Graphs and Automatic Parallelism

  By requiring only minimal tensor partitioning annotations based on a single-card configuration, PaddlePaddle automatically discovers the most efficient distributed parallel strategy. This significantly reduces the costs of industrial development and training, enabling developers to focus more intently on model and algorithm innovation.

• Integrated Training and Inference for Large Models

  The same framework supports both training and inference, achieving code reuse and seamless integration between these stages. This provides a unified development experience and maximum training efficiency for the entire large model workflow, offering the industry a superior development experience.

• High-Order Differentiation for Scientific Computing

  Provides capabilities such as high-order automatic differentiation, complex number operations, Fourier transforms, compilation optimization, and distributed training support. It facilitates scientific exploration in fields including mathematics, mechanics, materials science, meteorology, and biology, substantially improving the speed of solving differential equations.

• Neural Network Compiler

  Adopting an integrated framework design, it supports efficient training and flexible inference for diverse models, including generative and scientific computing models. It achieves an effective balance between computational flexibility and high performance, significantly lowering performance optimization costs.

• Heterogeneous Multi-Chip Adaptation
  Features a mature and complete unified adaptation solution for multiple hardware types. Through standardized interfaces, it abstracts the variations in development interfaces across different chip software stacks, realizing a pluggable architecture.

Documentation Copy Chapter

We provide English and
Chinese documentation.

• Guides

  You might want to start from how to implement deep learning basics with PaddlePaddle.

• Practice

  So far you have already been familiar with Fluid. And the next step should be building a more efficient model or inventing your original Operator.

• API Reference

  Our new API enables much shorter programs.

• How to Contribute

  We appreciate your contributions!

Open Source Community Copy Chapter

• Github Issues: bug reports, feature requests, install issues, usage issues, etc.
• Many of our contribution events offer varying levels of mentorship from experienced community members, please check the events in the pinned issues, and consider attending.
• Community Blog: https://pfcc.blog/
• See more details about PaddlePaddle community at community.

Copyright and License Copy Chapter

PaddlePaddle is provided under the Apache-2.0 license.

File: benchmark/fluid/README.md Copy Chapter

Fluid Benchmark

This directory contains several models configurations and tools that used to run
Fluid benchmarks for local and distributed training.

Run the Benchmark Copy Chapter

To start, run the following command to get the full help message:

python fluid_benchmark.py --help

Currently supported --model argument include:

• mnist

• resnet

  • you can chose to use different dataset using --data_set cifar10 or
    --data_set flowers.

• vgg

• stacked_dynamic_lstm

• machine_translation

• Run the following command to start a benchmark job locally:

  bash Copy

  python fluid_benchmark.py --model mnist  --device GPU

  You can choose to use GPU/CPU training. With GPU training, you can specify
  --gpus <gpu_num> to run multi GPU training.

• Run distributed training with parameter servers:

  • start parameter servers:
    bash Copy

    PADDLE_TRAINING_ROLE=PSERVER PADDLE_PSERVER_PORT=7164 PADDLE_PSERVER_IPS=127.0.0.1 PADDLE_TRAINERS=1 PADDLE_CURRENT_IP=127.0.0.1 PADDLE_TRAINER_ID=0 python fluid_benchmark.py --model mnist  --device GPU --update_method pserver

  • start trainers:
    bash Copy

    PADDLE_TRAINING_ROLE=TRAINER PADDLE_PSERVER_PORT=7164 PADDLE_PSERVER_IPS=127.0.0.1 PADDLE_TRAINERS=1 PADDLE_CURRENT_IP=127.0.0.1 PADDLE_TRAINER_ID=0 python fluid_benchmark.py --model mnist  --device GPU --update_method pserver

• Run distributed training using NCCL2

  bash Copy

  PADDLE_PSERVER_PORT=7164 PADDLE_TRAINER_IPS=192.168.0.2,192.168.0.3  PADDLE_CURRENT_IP=127.0.0.1 PADDLE_TRAINER_ID=0 python fluid_benchmark.py --model mnist --device GPU --update_method nccl2

Run Distributed Benchmark on Kubernetes Cluster Copy Chapter

We provide a script kube_gen_job.py to generate Kubernetes yaml files to submit
distributed benchmark jobs to your cluster. To generate a job yaml, just run:

python kube_gen_job.py --jobname myjob --pscpu 4 --cpu 8 --gpu 8 --psmemory 20 --memory 40 --pservers 4 --trainers 4 --entry "python fluid_benchmark.py --model mnist --parallel 1 --device GPU --update_method pserver " --disttype pserver

Then the yaml files are generated under directory myjob, you can run:

kubectl create -f myjob/

The job shall start.

Notes for Run Fluid Distributed with NCCL2 and RDMA Copy Chapter

Before running NCCL2 distributed jobs, please check that whether your node has multiple network
interfaces, try to add the environment variable export NCCL_SOCKET_IFNAME=eth0 to use your actual
network device.

To run high-performance distributed training, you must prepare your hardware environment to be
able to run RDMA enabled network communication, please check out this
note for details.

File: benchmark/README.md Copy Chapter

Benchmark

Machine:

• CPU: 12-core Intel(R) Xeon(R) CPU E5-2620 v2 @2.10GHz
• GPU: Tesla K40m
• cuDNN: v5.1
• system: Docker 1.12.1, all platforms are tested in docker environment.

Platforms:

• PaddlePaddle: paddledev/paddle:gpu-devel-v0.9.0a0
• Tensorflow: gcr.io/tensorflow/tensorflow:0.11.0rc0-gpu
• Caffe: kaixhin/cuda-caffe

Several convolutional neural networks and recurrent neural networks are used to test.

Image Copy Chapter

Benchmark Model Copy Section

AlexNet, GoogleNet and a small network used in Caffe.

• AlexNet: but the group size is one.

• GoogleNet: but remove loss1 and loss2 when testing benchmark.

• SmallNet

Single-GPU Copy Section

• AlexNet: input - 3 * 227 * 227, Time: ms/batch

Notation

All platforms use cuDNN-v5.1. We see that caffe is slower in this experiment, because its workspace limit size of cuDNN-conv interface is 8 * 1024 * 1024, which is smaller in PaddlePaddle and TensorFlow. Note that Caffe will be faster if increasing the workspace limit size.

• GoogletNet: input - 3 * 224 * 224, Time: ms/batch

• SmallNet: input - 3 * 32 * 32, Time ms/batch

Notation

All the single-GPU experiments in caffe use caffe time to calculate elapsed time, which does not include parameter updating time. However, both PaddlePaddle and TensorFlow experiments contain the parameter updating time. As compared with the total time, this part is relatively little on single machine, we can ignore it.

In Tensorflow, they implement algorithm searching method instead of using the algorithm searching interface in cuDNN.

Multi-GPU: 4 GPUs Copy Section

• AlexNet, ms / batch

For example, if total-BatchSize = 128 * 4, the speedup ratio is calculated by

time_at_1gpu_batch_128 * 4 / time_at_4gpu_total_batch_512 
= (334 * 4)/347 
= 3.85

• GoogleNet, ms / batch

RNN Copy Chapter

We use lstm network for text classfication to test benchmark.

Dataset Copy Section

• IMDB
• Sequence length is 100. In fact, PaddlePaddle supports training with variable-length sequence, but TensorFlow needs to pad. Thus, we also pad sequence length to 100 in PaddlePaddle in order to compare.
• Dictionary size=30000
• Peephole connection is used in lstmemory by default in PaddlePaddle. It is also configured in TensorFlow.

Single-GPU Copy Section

LSTM in Text Classification

Testing 2 lstm layer + fc network with different hidden size and batch size.

• Batch size = 64, ms / batch

• Batch size = 128, ms / batch

• Batch size = 256, ms / batch

Seq2Seq

The benchmark of sequence-to-sequence network will be added later.

Multi GPU: 4 GPUs Copy Section

LSTM in Text Classification

• hidden_size = 256, ms / batch

• hidden_size = 512, ms / batch

Seq2Seq

The benchmark of sequence-to-sequence network will be added later.

File: doc/fluid/api/data/data_reader.rst Copy Chapter

===================== Data Reader Interface

DataTypes

.. autofunction:: paddle.v2.data_type.dense_array
:noindex:

.. autofunction:: paddle.v2.data_type.integer_value
:noindex:

.. autofunction:: paddle.v2.data_type.integer_value_sequence
:noindex:

.. autofunction:: paddle.v2.data_type.integer_value_sub_sequence
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_binary_vector
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_binary_vector_sequence
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_binary_vector_sub_sequence
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_float_vector
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_float_vector_sequence
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_float_vector_sub_sequence
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_non_value_slot
:noindex:

.. autofunction:: paddle.v2.data_type.sparse_value_slot
:noindex:

.. autoclass:: paddle.v2.data_type.InputType
:members:
:noindex:

DataFeeder

.. automodule:: paddle.v2.data_feeder
:members:
:noindex:

Reader

.. automodule:: paddle.reader
:members:
:noindex:

.. automodule:: paddle.reader.creator
:members:
:noindex:

minibatch

.. automodule:: paddle.v2.minibatch
:members:
:noindex:

File: doc/fluid/api/data/dataset.rst Copy Chapter

Dataset

.. automodule:: paddle.dataset
:members:
:noindex:

mnist
+++++

.. automodule:: paddle.dataset.mnist
:members:
:noindex:

cifar
+++++

.. automodule:: paddle.dataset.cifar
:members:
:noindex:

conll05
+++++++

.. automodule:: paddle.dataset.conll05
:members: get_dict,get_embedding,test
:noindex:

imdb
++++

.. automodule:: paddle.dataset.imdb
:members:
:noindex:

imikolov
++++++++

.. automodule:: paddle.dataset.imikolov
:members:
:noindex:

movielens
+++++++++

.. automodule:: paddle.dataset.movielens
:members:
:noindex:

.. autoclass:: paddle.dataset.movielens.MovieInfo
:noindex:

.. autoclass:: paddle.dataset.movielens.UserInfo
:noindex:

sentiment
+++++++++

.. automodule:: paddle.dataset.sentiment
:members:
:noindex:

uci_housing
+++++++++++

.. automodule:: paddle.dataset.uci_housing
:members:
:noindex:

wmt14
+++++

.. automodule:: paddle.dataset.wmt14
:members:
:noindex:

wmt16
+++++

.. automodule:: paddle.dataset.wmt16
:members:
:noindex:

File: doc/fluid/api/clip.rst Copy Chapter

.. THIS FILE IS GENERATED BY gen_doc.{py|sh}
!DO NOT EDIT THIS FILE MANUALLY!

==== clip

ErrorClipByValue Copy Chapter

.. autoclass:: paddle.fluid.clip.ErrorClipByValue
:members:
:noindex:

GradientClipByValue Copy Chapter

.. autoclass:: paddle.fluid.clip.GradientClipByValue
:members:
:noindex:

GradientClipByNorm Copy Chapter

.. autoclass:: paddle.fluid.clip.GradientClipByNorm
:members:
:noindex:

GradientClipByGlobalNorm Copy Chapter

.. autoclass:: paddle.fluid.clip.GradientClipByGlobalNorm
:members:
:noindex:

append_gradient_clip_ops Copy Chapter

.. autofunction:: paddle.fluid.clip.append_gradient_clip_ops
:noindex:

error_clip_callback Copy Chapter

.. autofunction:: paddle.fluid.clip.error_clip_callback
:noindex:

File: doc/fluid/api/data.rst Copy Chapter

================================== Data Reader Interface and DataSets

.. toctree::
:maxdepth: 1

data/data_reader.rst
data/image.rst
data/dataset.rst

File: doc/fluid/api/data_feeder.rst Copy Chapter

.. THIS FILE IS GENERATED BY gen_doc.{py|sh}
!DO NOT EDIT THIS FILE MANUALLY!

=========== data_feeder

DataFeeder Copy Chapter

.. autoclass:: paddle.fluid.data_feeder.DataFeeder
:members:
:noindex:

2. Official Technical Reference & Guides (PaddlePaddle/docs) Copy Chapter

PaddlePaddle docs

English | 简体中文 | 日本語

Source files for contents presented at PaddlePaddle documentation.

Note: English version API docs are generally docstrings in PaddlePaddle/Paddle, documents for other PaddlePaddle projects are being managed in their respective ways.

Codebase structure Copy Chapter

• docs: PaddlePaddle 2.0 & above docs source file.
• docs/api: PaddlePaddle API docs.
• docs/guides: PaddlePaddle guides docs.
• docs/tutorial: PaddlePaddle tutorial docs.
• ci_scripts: docs CI scripts.

How to build Copy Chapter

• pre-requirements
  • docker
• Instructions
  • step1: clone docs
    text Copy

    git clone https://github.com/PaddlePaddle/docs

  • step2: build docs
    text Copy

    cd docs
    mkdir output
    bash docs-build.sh -f absolute_path_docs

  • step3: preview docs
    The output of docs will be generated in docs/output.

How to contribute Copy Chapter

PaddlePaddle welcomes documentation contributions, please see CONTRIBUTING.md for details.

License Copy Chapter

Apache License 2.0