## File: docs/source/internal/file_utils.rst .. Copyright 2021 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. General Utilities ----------------------------------------------------------------------------------------------------------------------- This page lists all of Transformers general utility functions that are found in the file ``file_utils.py``. Most of those are only useful if you are studying the general code in the library. Enums and namedtuples ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.file_utils.ExplicitEnum .. autoclass:: transformers.file_utils.PaddingStrategy .. autoclass:: transformers.file_utils.TensorType Special Decorators ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autofunction:: transformers.file_utils.add_start_docstrings .. autofunction:: transformers.file_utils.add_start_docstrings_to_model_forward .. autofunction:: transformers.file_utils.add_end_docstrings .. autofunction:: transformers.file_utils.add_code_sample_docstrings .. autofunction:: transformers.file_utils.replace_return_docstrings Special Properties ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.file_utils.cached_property Other Utilities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.file_utils._BaseLazyModule --- ## File: docs/source/internal/generation_utils.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Utilities for Generation ----------------------------------------------------------------------------------------------------------------------- This page lists all the utility functions used by :meth:`~transformers.PreTrainedModel.generate`, :meth:`~transformers.PreTrainedModel.greedy_search`, :meth:`~transformers.PreTrainedModel.sample`, :meth:`~transformers.PreTrainedModel.beam_search`, :meth:`~transformers.PreTrainedModel.beam_sample`, and :meth:`~transformers.PreTrainedModel.group_beam_search`. Most of those are only useful if you are studying the code of the generate methods in the library. Generate Outputs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The output of :meth:`~transformers.PreTrainedModel.generate` is an instance of a subclass of :class:`~transformers.file_utils.ModelOutput`. This output is a data structure containing all the information returned by :meth:`~transformers.PreTrainedModel.generate`, but that can also be used as tuple or dictionary. Here's an example: .. code-block:: from transformers import GPT2Tokenizer, GPT2LMHeadModel tokenizer = GPT2Tokenizer.from_pretrained('gpt2') model = GPT2LMHeadModel.from_pretrained('gpt2') inputs = tokenizer("Hello, my dog is cute and ", return_tensors="pt") generation_output = model.generate(**inputs, return_dict_in_generate=True, output_scores=True) The ``generation_output`` object is a :class:`~transformers.generation_utils.GreedySearchDecoderOnlyOutput`, as we can see in the documentation of that class below, it means it has the following attributes: - ``sequences``: the generated sequences of tokens - ``scores`` (optional): the prediction scores of the language modelling head, for each generation step - ``hidden_states`` (optional): the hidden states of the model, for each generation step - ``attentions`` (optional): the attention weights of the model, for each generation step Here we have the ``scores`` since we passed along ``output_scores=True``, but we don't have ``hidden_states`` and ``attentions`` because we didn't pass ``output_hidden_states=True`` or ``output_attentions=True``. You can access each attribute as you would usually do, and if that attribute has not been returned by the model, you will get ``None``. Here for instance ``generation_output.scores`` are all the generated prediction scores of the language modeling head, and ``generation_output.attentions`` is ``None``. When using our ``generation_output`` object as a tuple, it only keeps the attributes that don't have ``None`` values. Here, for instance, it has two elements, ``loss`` then ``logits``, so .. code-block:: generation_output[:2] will return the tuple ``(generation_output.sequences, generation_output.scores)`` for instance. When using our ``generation_output`` object as a dictionary, it only keeps the attributes that don't have ``None`` values. Here, for instance, it has two keys that are ``sequences`` and ``scores``. We document here all output types. GreedySearchOutput ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. autoclass:: transformers.generation_utils.GreedySearchDecoderOnlyOutput :members: .. autoclass:: transformers.generation_utils.GreedySearchEncoderDecoderOutput :members: SampleOutput ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. autoclass:: transformers.generation_utils.SampleDecoderOnlyOutput :members: .. autoclass:: transformers.generation_utils.SampleEncoderDecoderOutput :members: BeamSearchOutput ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. autoclass:: transformers.generation_utils.BeamSearchDecoderOnlyOutput :members: .. autoclass:: transformers.generation_utils.BeamSearchEncoderDecoderOutput :members: BeamSampleOutput ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. autoclass:: transformers.generation_utils.BeamSampleDecoderOnlyOutput :members: .. autoclass:: transformers.generation_utils.BeamSampleEncoderDecoderOutput :members: LogitsProcessor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A :class:`~transformers.LogitsProcessor` can be used to modify the prediction scores of a language model head for generation. .. autoclass:: transformers.LogitsProcessor :members: __call__ .. autoclass:: transformers.LogitsProcessorList :members: __call__ .. autoclass:: transformers.LogitsWarper :members: __call__ .. autoclass:: transformers.MinLengthLogitsProcessor :members: __call__ .. autoclass:: transformers.TemperatureLogitsWarper :members: __call__ .. autoclass:: transformers.RepetitionPenaltyLogitsProcessor :members: __call__ .. autoclass:: transformers.TopPLogitsWarper :members: __call__ .. autoclass:: transformers.TopKLogitsWarper :members: __call__ .. autoclass:: transformers.NoRepeatNGramLogitsProcessor :members: __call__ .. autoclass:: transformers.NoBadWordsLogitsProcessor :members: __call__ .. autoclass:: transformers.PrefixConstrainedLogitsProcessor :members: __call__ .. autoclass:: transformers.HammingDiversityLogitsProcessor :members: __call__ .. autoclass:: transformers.ForcedBOSTokenLogitsProcessor :members: __call__ .. autoclass:: transformers.ForcedEOSTokenLogitsProcessor :members: __call__ .. autoclass:: transformers.InfNanRemoveLogitsProcessor :members: __call__ StoppingCriteria ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A :class:`~transformers.StoppingCriteria` can be used to change when to stop generation (other than EOS token). .. autoclass:: transformers.StoppingCriteria :members: __call__ .. autoclass:: transformers.StoppingCriteriaList :members: __call__ .. autoclass:: transformers.MaxLengthCriteria :members: __call__ .. autoclass:: transformers.MaxTimeCriteria :members: __call__ BeamSearch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.BeamScorer :members: process, finalize .. autoclass:: transformers.BeamSearchScorer :members: process, finalize Utilities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autofunction:: transformers.top_k_top_p_filtering .. autofunction:: transformers.tf_top_k_top_p_filtering --- ## File: docs/source/internal/modeling_utils.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Custom Layers and Utilities ----------------------------------------------------------------------------------------------------------------------- This page lists all the custom layers used by the library, as well as the utility functions it provides for modeling. Most of those are only useful if you are studying the code of the models in the library. Pytorch custom modules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.modeling_utils.Conv1D .. autoclass:: transformers.modeling_utils.PoolerStartLogits :members: forward .. autoclass:: transformers.modeling_utils.PoolerEndLogits :members: forward .. autoclass:: transformers.modeling_utils.PoolerAnswerClass :members: forward .. autoclass:: transformers.modeling_utils.SquadHeadOutput .. autoclass:: transformers.modeling_utils.SQuADHead :members: forward .. autoclass:: transformers.modeling_utils.SequenceSummary :members: forward PyTorch Helper Functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autofunction:: transformers.apply_chunking_to_forward .. autofunction:: transformers.modeling_utils.find_pruneable_heads_and_indices .. autofunction:: transformers.modeling_utils.prune_layer .. autofunction:: transformers.modeling_utils.prune_conv1d_layer .. autofunction:: transformers.modeling_utils.prune_linear_layer TensorFlow custom layers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.modeling_tf_utils.TFConv1D .. autoclass:: transformers.modeling_tf_utils.TFSharedEmbeddings :members: call .. autoclass:: transformers.modeling_tf_utils.TFSequenceSummary :members: call TensorFlow loss functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.modeling_tf_utils.TFCausalLanguageModelingLoss :members: .. autoclass:: transformers.modeling_tf_utils.TFMaskedLanguageModelingLoss :members: .. autoclass:: transformers.modeling_tf_utils.TFMultipleChoiceLoss :members: .. autoclass:: transformers.modeling_tf_utils.TFQuestionAnsweringLoss :members: .. autoclass:: transformers.modeling_tf_utils.TFSequenceClassificationLoss :members: .. autoclass:: transformers.modeling_tf_utils.TFTokenClassificationLoss :members: TensorFlow Helper Functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autofunction:: transformers.modeling_tf_utils.get_initializer .. autofunction:: transformers.modeling_tf_utils.keras_serializable .. autofunction:: transformers.modeling_tf_utils.shape_list --- ## File: docs/source/internal/pipelines_utils.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Utilities for pipelines ----------------------------------------------------------------------------------------------------------------------- This page lists all the utility functions the library provides for pipelines. Most of those are only useful if you are studying the code of the models in the library. Argument handling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.pipelines.ArgumentHandler .. autoclass:: transformers.pipelines.ZeroShotClassificationArgumentHandler .. autoclass:: transformers.pipelines.QuestionAnsweringArgumentHandler Data format ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.pipelines.PipelineDataFormat :members: .. autoclass:: transformers.pipelines.CsvPipelineDataFormat :members: .. autoclass:: transformers.pipelines.JsonPipelineDataFormat :members: .. autoclass:: transformers.pipelines.PipedPipelineDataFormat :members: Utilities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.pipelines.PipelineException --- ## File: docs/source/internal/tokenization_utils.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Utilities for Tokenizers ----------------------------------------------------------------------------------------------------------------------- This page lists all the utility functions used by the tokenizers, mainly the class :class:`~transformers.tokenization_utils_base.PreTrainedTokenizerBase` that implements the common methods between :class:`~transformers.PreTrainedTokenizer` and :class:`~transformers.PreTrainedTokenizerFast` and the mixin :class:`~transformers.tokenization_utils_base.SpecialTokensMixin`. Most of those are only useful if you are studying the code of the tokenizers in the library. PreTrainedTokenizerBase ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.tokenization_utils_base.PreTrainedTokenizerBase :special-members: __call__ :members: SpecialTokensMixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.tokenization_utils_base.SpecialTokensMixin :members: Enums and namedtuples ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.tokenization_utils_base.TruncationStrategy .. autoclass:: transformers.tokenization_utils_base.CharSpan .. autoclass:: transformers.tokenization_utils_base.TokenSpan --- ## File: docs/source/internal/trainer_utils.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Utilities for Trainer ----------------------------------------------------------------------------------------------------------------------- This page lists all the utility functions used by :class:`~transformers.Trainer`. Most of those are only useful if you are studying the code of the Trainer in the library. Utilities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.EvalPrediction .. autoclass:: transformers.IntervalStrategy .. autofunction:: transformers.set_seed .. autofunction:: transformers.torch_distributed_zero_first Callbacks internals ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.trainer_callback.CallbackHandler Distributed Evaluation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.trainer_pt_utils.DistributedTensorGatherer :members: Distributed Evaluation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.HfArgumentParser Debug Utilities ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.debug_utils.DebugUnderflowOverflow --- ## File: docs/source/main_classes/callback.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Callbacks ----------------------------------------------------------------------------------------------------------------------- Callbacks are objects that can customize the behavior of the training loop in the PyTorch :class:`~transformers.Trainer` (this feature is not yet implemented in TensorFlow) that can inspect the training loop state (for progress reporting, logging on TensorBoard or other ML platforms...) and take decisions (like early stopping). Callbacks are "read only" pieces of code, apart from the :class:`~transformers.TrainerControl` object they return, they cannot change anything in the training loop. For customizations that require changes in the training loop, you should subclass :class:`~transformers.Trainer` and override the methods you need (see :doc:`trainer` for examples). By default a :class:`~transformers.Trainer` will use the following callbacks: - :class:`~transformers.DefaultFlowCallback` which handles the default behavior for logging, saving and evaluation. - :class:`~transformers.PrinterCallback` or :class:`~transformers.ProgressCallback` to display progress and print the logs (the first one is used if you deactivate tqdm through the :class:`~transformers.TrainingArguments`, otherwise it's the second one). - :class:`~transformers.integrations.TensorBoardCallback` if tensorboard is accessible (either through PyTorch >= 1.4 or tensorboardX). - :class:`~transformers.integrations.WandbCallback` if `wandb `__ is installed. - :class:`~transformers.integrations.CometCallback` if `comet_ml `__ is installed. - :class:`~transformers.integrations.MLflowCallback` if `mlflow `__ is installed. - :class:`~transformers.integrations.AzureMLCallback` if `azureml-sdk `__ is installed. The main class that implements callbacks is :class:`~transformers.TrainerCallback`. It gets the :class:`~transformers.TrainingArguments` used to instantiate the :class:`~transformers.Trainer`, can access that Trainer's internal state via :class:`~transformers.TrainerState`, and can take some actions on the training loop via :class:`~transformers.TrainerControl`. Available Callbacks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here is the list of the available :class:`~transformers.TrainerCallback` in the library: .. autoclass:: transformers.integrations.CometCallback :members: setup .. autoclass:: transformers.DefaultFlowCallback .. autoclass:: transformers.PrinterCallback .. autoclass:: transformers.ProgressCallback .. autoclass:: transformers.EarlyStoppingCallback .. autoclass:: transformers.integrations.TensorBoardCallback .. autoclass:: transformers.integrations.WandbCallback :members: setup .. autoclass:: transformers.integrations.MLflowCallback :members: setup .. autoclass:: transformers.integrations.AzureMLCallback TrainerCallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TrainerCallback :members: Here is an example of how to register a custom callback with the PyTorch :class:`~transformers.Trainer`: .. code-block:: python class MyCallback(TrainerCallback): "A callback that prints a message at the beginning of training" def on_train_begin(self, args, state, control, **kwargs): print("Starting training") trainer = Trainer( model, args, train_dataset=train_dataset, eval_dataset=eval_dataset, callbacks=[MyCallback] # We can either pass the callback class this way or an instance of it (MyCallback()) ) Another way to register a callback is to call ``trainer.add_callback()`` as follows: .. code-block:: python trainer = Trainer(...) trainer.add_callback(MyCallback) # Alternatively, we can pass an instance of the callback class trainer.add_callback(MyCallback()) TrainerState ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TrainerState :members: TrainerControl ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TrainerControl :members: --- ## File: docs/source/main_classes/configuration.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Configuration ----------------------------------------------------------------------------------------------------------------------- The base class :class:`~transformers.PretrainedConfig` implements the common methods for loading/saving a configuration either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace's AWS S3 repository). PretrainedConfig ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.PretrainedConfig :members: --- ## File: docs/source/main_classes/data_collator.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Data Collator ----------------------------------------------------------------------------------------------------------------------- Data collators are objects that will form a batch by using a list of dataset elements as input. These elements are of the same type as the elements of :obj:`train_dataset` or :obj:`eval_dataset`. To be able to build batches, data collators may apply some processing (like padding). Some of them (like :class:`~transformers.DataCollatorForLanguageModeling`) also apply some random data augmentation (like random masking) oin the formed batch. Examples of use can be found in the :doc:`example scripts <../examples>` or :doc:`example notebooks <../notebooks>`. Default data collator ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autofunction:: transformers.data.data_collator.default_data_collator DataCollatorWithPadding ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.data.data_collator.DataCollatorWithPadding :members: DataCollatorForTokenClassification ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.data.data_collator.DataCollatorForTokenClassification :members: DataCollatorForSeq2Seq ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.data.data_collator.DataCollatorForSeq2Seq :members: DataCollatorForLanguageModeling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.data.data_collator.DataCollatorForLanguageModeling :members: mask_tokens DataCollatorForWholeWordMask ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.data.data_collator.DataCollatorForWholeWordMask :members: mask_tokens DataCollatorForPermutationLanguageModeling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.data.data_collator.DataCollatorForPermutationLanguageModeling :members: mask_tokens --- ## File: docs/source/main_classes/feature_extractor.rst .. Copyright 2021 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Feature Extractor ----------------------------------------------------------------------------------------------------------------------- A feature extractor is in charge of preparing input features for a multi-modal model. This includes feature extraction from sequences, *e.g.*, pre-processing audio files to Log-Mel Spectrogram features, feature extraction from images *e.g.* cropping image image files, but also padding, normalization, and conversion to Numpy, PyTorch, and TensorFlow tensors. FeatureExtractionMixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.feature_extraction_utils.FeatureExtractionMixin :members: from_pretrained, save_pretrained SequenceFeatureExtractor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.SequenceFeatureExtractor :members: pad BatchFeature ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.BatchFeature :members: ImageFeatureExtractionMixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.image_utils.ImageFeatureExtractionMixin :members: --- ## File: docs/source/main_classes/logging.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Logging ----------------------------------------------------------------------------------------------------------------------- 🤗 Transformers has a centralized logging system, so that you can setup the verbosity of the library easily. Currently the default verbosity of the library is ``WARNING``. To change the level of verbosity, just use one of the direct setters. For instance, here is how to change the verbosity to the INFO level. .. code-block:: python import transformers transformers.logging.set_verbosity_info() You can also use the environment variable ``TRANSFORMERS_VERBOSITY`` to override the default verbosity. You can set it to one of the following: ``debug``, ``info``, ``warning``, ``error``, ``critical``. For example: .. code-block:: bash TRANSFORMERS_VERBOSITY=error ./myprogram.py All the methods of this logging module are documented below, the main ones are :func:`transformers.logging.get_verbosity` to get the current level of verbosity in the logger and :func:`transformers.logging.set_verbosity` to set the verbosity to the level of your choice. In order (from the least verbose to the most verbose), those levels (with their corresponding int values in parenthesis) are: - :obj:`transformers.logging.CRITICAL` or :obj:`transformers.logging.FATAL` (int value, 50): only report the most critical errors. - :obj:`transformers.logging.ERROR` (int value, 40): only report errors. - :obj:`transformers.logging.WARNING` or :obj:`transformers.logging.WARN` (int value, 30): only reports error and warnings. This the default level used by the library. - :obj:`transformers.logging.INFO` (int value, 20): reports error, warnings and basic information. - :obj:`transformers.logging.DEBUG` (int value, 10): report all information. Base setters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autofunction:: transformers.logging.set_verbosity_error .. autofunction:: transformers.logging.set_verbosity_warning .. autofunction:: transformers.logging.set_verbosity_info .. autofunction:: transformers.logging.set_verbosity_debug Other functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autofunction:: transformers.logging.get_verbosity .. autofunction:: transformers.logging.set_verbosity .. autofunction:: transformers.logging.get_logger .. autofunction:: transformers.logging.enable_default_handler .. autofunction:: transformers.logging.disable_default_handler .. autofunction:: transformers.logging.enable_explicit_format .. autofunction:: transformers.logging.reset_format --- ## File: docs/source/main_classes/model.rst .. Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Models ----------------------------------------------------------------------------------------------------------------------- The base classes :class:`~transformers.PreTrainedModel`, :class:`~transformers.TFPreTrainedModel`, and :class:`~transformers.FlaxPreTrainedModel` implement the common methods for loading/saving a model either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace's AWS S3 repository). :class:`~transformers.PreTrainedModel` and :class:`~transformers.TFPreTrainedModel` also implement a few methods which are common among all the models to: - resize the input token embeddings when new tokens are added to the vocabulary - prune the attention heads of the model. The other methods that are common to each model are defined in :class:`~transformers.modeling_utils.ModuleUtilsMixin` (for the PyTorch models) and :class:`~transformers.modeling_tf_utils.TFModuleUtilsMixin` (for the TensorFlow models) or for text generation, :class:`~transformers.generation_utils.GenerationMixin` (for the PyTorch models) and :class:`~transformers.generation_tf_utils.TFGenerationMixin` (for the TensorFlow models) PreTrainedModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.PreTrainedModel :members: ModuleUtilsMixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.modeling_utils.ModuleUtilsMixin :members: TFPreTrainedModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.TFPreTrainedModel :members: TFModelUtilsMixin ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.modeling_tf_utils.TFModelUtilsMixin :members: FlaxPreTrainedModel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.FlaxPreTrainedModel :members: Generation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.generation_utils.GenerationMixin :members: .. autoclass:: transformers.generation_tf_utils.TFGenerationMixin :members: Pushing to the Hub ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: transformers.file_utils.PushToHubMixin :members: