## 1. Project Overview & Quickstart (ludwig-ai/ludwig) ## File: README.md ______________________________________________________________________ ## What is Ludwig? Ludwig is a **declarative deep learning framework** that lets you train, fine-tune, and deploy AI models — from LLM fine-tuning to tabular classification — using a YAML config file and zero boilerplate Python. ```yaml # Fine-tune Llama-3.1 with LoRA in one config file model_type: llm base_model: meta-llama/Llama-3.1-8B adapter: type: lora trainer: type: finetune epochs: 3 input_features: - name: instruction type: text output_features: - name: response type: text ``` ```bash ludwig train --config model.yaml --dataset my_data.csv ``` **Tech stack:** Python 3.12 · PyTorch 2.7+ · Pydantic 2 · Transformers 5 · Ray 2.54 Ludwig is hosted by the [Linux Foundation AI & Data](https://lfaidata.foundation/). ______________________________________________________________________ ## What's New in Ludwig 0.16 | Feature | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------ | | **PatchTST & N-BEATS encoders** | State-of-the-art timeseries forecasting encoders with MASE/sMAPE metrics | | **Advanced PEFT adapters** | PiSSA, EVA, CorDA/LoftQ initializers; TinyLoRA, OFT, HRA, WaveFT, LN-Tuning, VBLoRA, C3A adapter types | | **VLM fine-tuning** | Train LLaVA, Qwen2-VL, InternVL via `is_multimodal: true` with gated cross-attention | | **HyperNetwork combiner** | Conditioning-based feature fusion — one feature generates weights for others | | **Nash-MTL & Pareto-MTL** | Game-theoretic and preference-based multi-task loss balancing | | **LLM config generation** | `ludwig generate_config "describe your task"` — LLM writes the YAML for you | | **ModelInspector** | Architecture analysis, weight collection, feature importance proxy | | **Ray Serve & KServe** | Distributed and Kubernetes-native model deployment shims | | **GRPO alignment** | Reward-model-free RLHF via Group Relative Policy Optimization | | **torchao quantization + QAT** | PyTorch-native `int4/int8/float8` with Quantization-Aware Training | | **Multi-adapter PEFT** | Multiple named LoRA adapters with weighted merging (TIES, DARE, SVD) | | **Native Optuna executor** | GPT/TPE/CMA-ES samplers, pruning, resumable SQLite/PostgreSQL storage | | **Timeseries forecasting** | `model.forecast(dataset, horizon=N)` API with `TimeseriesOutputFeature` | | **Muon & ScheduleFreeAdamW** | New optimizers for large-scale pretraining and fine-tuning | | **Image segmentation decoders** | UNet, SegFormer, FPN decoders for semantic segmentation | ______________________________________________________________________ ## Installation ```bash pip install ludwig # core pip install ludwig[full] # all optional dependencies pip install ludwig[llm] # LLM fine-tuning only ``` Requires Python 3.12+. See [contributing](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md) for a full dependency matrix. ______________________________________________________________________ ## Quick Start ### Fine-tune an LLM (instruction tuning) [](https://colab.research.google.com/drive/1c3AO8l_H6V_x37RwQ8V7M6A-RmcBf2tG?usp=sharing) Ludwig supports the full LLM fine-tuning spectrum: | Technique | Config key | | --------------------------------- | ------------------------------------------------------------------------ | | Supervised fine-tuning (SFT) | `trainer.type: finetune` | | DPO / KTO / ORPO / GRPO alignment | `trainer.type: dpo` (or `kto`, `orpo`, `grpo`) | | LoRA / DoRA / VeRA / PiSSA | `adapter.type: lora` (or `dora`, `vera`, `lora` + `init_weights: pissa`) | | 4-bit QLoRA (bitsandbytes) | `quantization.bits: 4` | | torchao + QAT | `quantization.backend: torchao` | | Multi-adapter with merging | `adapters:` dict + `merge:` block | | VLM (vision-language) | `is_multimodal: true` | ```yaml model_type: llm base_model: meta-llama/Llama-3.1-8B quantization: bits: 4 adapter: type: lora prompt: template: | ### Instruction: {instruction} ### Input: {input} ### Response: input_features: - name: prompt type: text output_features: - name: output type: text trainer: type: finetune learning_rate: 0.0001 batch_size: 1 gradient_accumulation_steps: 16 epochs: 3 learning_rate_scheduler: decay: cosine warmup_fraction: 0.01 backend: type: local ``` ```bash export HUGGING_FACE_HUB_TOKEN="" ludwig train --config model.yaml --dataset "ludwig://alpaca" ``` ### Train a multimodal classifier ```yaml input_features: - name: review_text type: text encoder: type: bert - name: star_rating type: number - name: product_image type: image encoder: type: dinov2 output_features: - name: recommended type: binary ``` ```bash ludwig train --config model.yaml --dataset reviews.csv ``` ### Generate a config from natural language ```bash ludwig generate_config "I have a CSV with age, income, education level, and I want to predict loan default" ``` ### Make predictions ```bash ludwig predict --model_path results/experiment_run/model --dataset new_data.csv ``` ### Launch a REST API ```bash ludwig serve --model_path results/experiment_run/model # POST http://localhost:8000/predict ``` ______________________________________________________________________ ## Capabilities **LLM Fine-Tuning** - **Supervised fine-tuning (SFT)** on instruction/response pairs - **Alignment training**: DPO, KTO, ORPO, GRPO (reward-model-free RLHF) - **PEFT adapters**: LoRA, DoRA, VeRA, LoRA+, TinyLoRA, OFT, HRA, WaveFT, LN-Tuning, VBLoRA, C3A - **LoRA initializers**: PiSSA, EVA, CorDA, LoftQ for improved convergence - **Multi-adapter PEFT**: multiple named adapters on one base model, switchable at runtime; merge with TIES, DARE, SVD, magnitude pruning - **Quantization**: 4-bit/8-bit QLoRA (bitsandbytes), torchao int4/int8/float8 with QAT - **VLM fine-tuning**: LLaVA, Qwen2-VL, InternVL via `is_multimodal: true` - **Sequence packing** for efficient training on variable-length inputs - **Paged and 8-bit optimizers** for memory-efficient training **Multimodal & Tabular Models** - **Input modalities**: text, numbers, categories, binary, sets, bags, sequences, images, audio, timeseries, vectors, dates - **Text encoders**: any HuggingFace Transformer (BERT, RoBERTa, ModernBERT, Qwen3, Llama-3.1, etc.), plus Mamba-2, Jamba - **Image encoders**: DINOv2, ConvNeXt, EfficientNet, ViT, CAFormer, ConvFormer, PoolFormer, TIMM (1000+ models) - **Timeseries encoders**: PatchTST, N-BEATS, CNN, RNN, Transformer; MASE and sMAPE metrics; `model.forecast()` API - **Combiners**: concat, transformer, tab_transformer, FT-Transformer, TabNet, TabPFN v2, HyperNetwork, ProjectAggregate, GatedFusion, Perceiver - **Multi-task learning**: multiple output features in a single model; Nash-MTL, Pareto-MTL, FAMO, GradNorm, uncertainty loss balancing - **Image segmentation**: UNet, SegFormer, FPN decoders **Training Infrastructure** - **Distributed training**: HuggingFace Accelerate with DDP, FSDP, DeepSpeed (zero-code changes) - **Ray backend**: training across a Ray cluster, larger-than-memory datasets via Ray Data - **Automatic batch size selection** and learning rate range test - **Mixed precision** (fp16/bf16), gradient checkpointing, gradient accumulation - **Optimizers**: AdamW, Adafactor, SGD, Muon, ScheduleFreeAdamW, Lion, paged/8-bit variants - **Learning rate schedulers**: cosine, linear, polynomial, reduce-on-plateau, OneCycleLR - **Model Soup**: uniform and greedy checkpoint averaging for better generalization at zero inference cost - **Modality dropout** for robust multimodal models **Hyperparameter Optimization** - **Executors**: Ray Tune (ASHA, PBT, Bayesian) and native Optuna (auto/GP/TPE/CMA-ES) - **Optuna persistence**: SQLite or PostgreSQL for resumable HPO runs - **Pruning** with Optuna's MedianPruner and HyperbandPruner - **Search spaces**: uniform, log-uniform, choice, randint, quantized - **Full Ludwig config** is searchable — any nested parameter can be a hyperparameter **Production & Deployment** - **REST API**: FastAPI server with Prometheus metrics and structured logging (`ludwig serve`) - **vLLM serving**: OpenAI-compatible API with PagedAttention and continuous batching - **Ray Serve**: distributed deployment with auto-scaling and traffic splitting - **KServe**: Kubernetes-native deployment with Open Inference Protocol v2 - **Model export**: SafeTensors (default), `torch.export` `.pt2` bundles, ONNX - **HuggingFace Hub**: `ludwig upload hf_hub` — push model + auto-generated model card - **Docker**: prebuilt containers at [ludwigai/ludwig](https://hub.docker.com/u/ludwigai) **Tooling & Integrations** - **Experiment tracking**: TensorBoard, Weights & Biases, Comet ML, MLflow, Aim Stack - **Model inspection**: `ModelInspector` — weight enumeration, architecture summary, feature importance proxy - **Visualizations**: learning curves, confusion matrices, calibration plots, ROC curves, hyperopt analysis - **AutoML**: `ludwig.automl.auto_train()` — give it a dataset and a time budget; the YAML-driven search space samples encoder/combiner/decoder combinations and validates them before training - **Dataset quality checks**: `from ludwig.utils.dataset_quality import check_dataset_quality` — validates a DataFrame before training (missing values, class imbalance, near-duplicate columns, ID leakage, …) - **OpenML integration**: load any OpenML task directly — `OpenMLLoader` fetches by task ID and caches locally as Parquet - **LLM config generation**: `ludwig generate_config "describe your task"` — LLM writes the YAML - **K-fold cross-validation**: `ludwig experiment --k_fold N` - **Dataset Zoo**: 70+ built-in benchmark datasets (`ludwig://mnist`, `ludwig://alpaca`, …) ______________________________________________________________________ ## Examples ### LLM & Alignment | Use Case | Link | | ------------------------------------- | ----------------------------------------------------------------------------------- | | LLM instruction tuning (LoRA + QLoRA) | [examples/llm](https://ludwig.ai/latest/examples/llm/llm_finetuning) | | DPO / GRPO alignment | [examples/llm/alignment](https://ludwig.ai/latest/examples/llm/alignment) | | Advanced PEFT (PiSSA, OFT, VBLoRA, …) | [examples/llms/peft_advanced](https://ludwig.ai/latest/examples/llms/peft_advanced) | | VLM fine-tuning (LLaVA, Qwen2-VL) | [examples/vlm](https://github.com/ludwig-ai/ludwig/tree/main/examples/vlm) | ### Tabular & Multimodal | Use Case | Link | | -------------------------------------- | ------------------------------------------------------------------------------------------------- | | Binary classification (Titanic) | [examples/titanic](https://ludwig.ai/latest/examples/titanic) | | Tabular classification (census income) | [examples/adult_census_income](https://ludwig.ai/latest/examples/adult_census_income) | | Multimodal classification | [examples/multimodal_classification](https://ludwig.ai/latest/examples/multimodal_classification) | | Multi-task learning | [examples/multi_task](https://ludwig.ai/latest/examples/multi_task) | ### Timeseries & Vision | Use Case | Link | | ------------------------------------------ | ----------------------------------------------------------------------------------------- | | Timeseries forecasting (PatchTST, N-BEATS) | [examples/forecasting](https://ludwig.ai/latest/examples/forecasting) | | Weather forecasting | [examples/weather](https://ludwig.ai/latest/examples/weather) | | Image classification (MNIST) | [examples/mnist](https://ludwig.ai/latest/examples/mnist) | | Semantic segmentation | [examples/semantic_segmentation](https://ludwig.ai/latest/examples/semantic_segmentation) | ### NLP & Audio | Use Case | Link | | ------------------------ | --------------------------------------------------------------------------------------- | | Text classification | [examples/text_classification](https://ludwig.ai/latest/examples/text_classification) | | Named entity recognition | [examples/ner_tagging](https://ludwig.ai/latest/examples/ner_tagging) | | Machine translation | [examples/machine_translation](https://ludwig.ai/latest/examples/machine_translation) | | Speech recognition | [examples/speech_recognition](https://ludwig.ai/latest/examples/speech_recognition) | | Speaker verification | [examples/speaker_verification](https://ludwig.ai/latest/examples/speaker_verification) | ______________________________________________________________________ ## Why Ludwig? - **Zero boilerplate** — no training loop, no data pipeline, no evaluation code. The YAML config is the entire program. - **Best-in-class LLM support** — full spectrum from LoRA to GRPO alignment, torchao QAT, and VLM fine-tuning, all in config. - **Multimodal out of the box** — mix text, images, numbers, audio, and timeseries with one config change. - **Scale without code changes** — go from laptop → multi-GPU → Ray cluster by changing `backend.type`. - **Expert control when you need it** — every activation function, scheduler, and optimizer is configurable. - **Reproducible research** — every run is logged and the full config is saved. Compare experiments with `ludwig visualize`. ______________________________________________________________________ ## Publications - [Ludwig: A Type-Based Declarative Deep Learning Toolbox](https://arxiv.org/pdf/1909.07930.pdf) (2019) - [Declarative Machine Learning Systems](https://arxiv.org/pdf/2107.08148.pdf) (2021) - [Ludwig's State-of-the-Art Benchmarks](https://openreview.net/pdf?id=hwjnu6qW7E4) ______________________________________________________________________ ## Community [](https://discord.gg/CBgdrGnZjy) - [Discord](https://discord.gg/CBgdrGnZjy) — ask questions, share what you've built - [GitHub Issues](https://github.com/ludwig-ai/ludwig/issues) — bugs and feature requests - [X / Twitter](https://twitter.com/ludwig_ai) — announcements - [Medium](https://medium.com/ludwig-ai) — tutorials and deep-dives --- ## File: docker/README.md # Ludwig Docker Images These images provide Ludwig, a toolbox to train and evaluate deep learning models without the need to write code. Ludwig Docker images contain the full set of pre-requisite packages to support these capabilities - text features - image features - audio features - visualizations - hyperparameter optimization - distributed training - model serving ## Publishing images Images are normally published automatically by CI (`.github/workflows/docker.yml`) when a release tag is pushed. To publish manually or backfill a release, use the script in this directory: ```bash # Requires: docker login to a ludwigai Docker Hub account ./docker/build_and_push.sh [--latest] # Examples ./docker/build_and_push.sh 0.14.0 --latest # new latest release ./docker/build_and_push.sh 0.13.0 # backfill without updating :latest ``` See `RELEASES.md` for the full release procedure. ## Repositories These four repositories contain a version of Ludwig with full features built from the project's `master` branch. - `ludwigai/ludwig` Ludwig packaged with PyTorch - `ludwigai/ludwig-gpu` Ludwig packaged with gpu-enabled version of PyTorch - `ludwigai/ludwig-ray` Ludwig packaged with PyTorch and Ray 2.3.1 (https://github.com/ray-project/ray) - `ludwigai/ludwig-ray-gpu` Ludwig packaged with gpu-enabled versions of PyTorch and Ray 2.3.1 (https://github.com/ray-project/ray) ## Image Tags - `master` - built from Ludwig's `master` branch - `nightly` - nightly build of Ludwig's software. - `sha-` - version of Ludwig software at designated git sha1 7-character commit point. ## Running Containers Examples of using the `ludwigai/ludwig:master` image to: - run the `ludwig cli` command or - run Python program containing Ludwig api or - view Ludwig results with Tensorboard For purposes of the examples assume this host directory structure ``` /top/level/directory/path/ data/ train.csv src/ config.yaml ludwig_api_program.py ``` ### Run Ludwig CLI ``` # set shell variable to parent directory parent_path=/top/level/directory/path # invoke docker run command to execute the ludwig cli # map host directory ${parent_path}/data to container /data directory # map host directory ${parent_path}/src to container /src directory docker run -v ${parent_path}/data:/data \ -v ${parent_path}/src:/src \ ludwigai/ludwig:master \ experiment --config /src/config.yaml \ --dataset /data/train.csv \ --output_directory /src/results ``` Experiment results can be found in host directory `/top/level/directory/path/src/results` ### Run Python program using Ludwig APIs ``` # set shell variable to parent directory parent_path=/top/level/directory/path # invoke docker run command to execute Python interpreter # map host directory ${parent_path}/data to container /data directory # map host directory ${parent_path}/src to container /src directory # set current working directory to container /src directory # change default entrypoint from ludwig to python docker run -v ${parent_path}/data:/data \ -v ${parent_path}/src:/src \ -w /src \ --entrypoint python \ ludwigai/ludwig:master /src/ludwig_api_program.py ``` Ludwig results can be found in host directory `/top/level/directory/path/src/results` ### View Ludwig Tensorboard results ``` # set shell variable to parent directory parent_path=/top/level/directory/path # invoke docker run command to execute Tensorboard # map host directory ${parent_path}/src to container /src directory # set up mapping from localhost port 6006 to container port 6006 # change default entrypoint from ludwig to tensorboard # --logdir container location of tenorboard logs /src/results/_/model/logs # --bind_all Tensorboard serves on all public container interfaces docker run -v ${parent_path}/src:/src \ -p 6006:6006 \ --entrypoint tensorboard \ ludwigai/ludwig:master \ --logdir /src/results/experiment_run/model/logs \ --bind_all ``` Point browser to `http://localhost:6006` to see Tensorboard dashboard. ### Devcontainer If you want to contribute to Ludwig, you can setup a Docker container with all the dependencies installed as a full featured development environment. This can be done using devcontainers with VS Code: https://code.visualstudio.com/docs/devcontainers/containers You can find the `devcontainer.json` file within the top level `.devcontainer` folder. --- ## File: docs/developer_guide/adding_a_feature_type.md # Adding a New Feature Type to Ludwig This guide walks through every file you need to touch when adding a brand-new feature type (e.g. a hypothetical `"widget"` type). Use `ludwig/features/binary_feature.py` and `ludwig/schema/features/binary_feature.py` as living reference implementations — they are among the simplest complete examples. ______________________________________________________________________ ## Conceptual overview Each feature type lives in two parallel places: | Layer | Location | Purpose | | ------------------ | ------------------------------------------ | ----------------------------------------------------------------------------- | | **Schema** | `ludwig/schema/features/_feature.py` | Pydantic-backed config classes; declares hyperparameters and their defaults | | **Feature module** | `ludwig/features/_feature.py` | PyTorch modules; implements preprocessing, encoding, decoding, postprocessing | The schema classes are used for config validation and serialization. The feature module classes are instantiated at model-build time using those configs. Neither layer knows the other exists at import time — they are wired together through the feature registry. ______________________________________________________________________ ## Step 1 ��� Define the constant Add the type string to `ludwig/constants.py`: ```python WIDGET = "widget" ``` ______________________________________________________________________ ## Step 2 — Write the schema file Create `ludwig/schema/features/widget_feature.py`. The minimal required structure is: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` **Key rules:** - `type` must be a `ProtectedString` with your constant — this prevents accidental overwrite via user YAML. - `@input_mixin_registry.register` / `@output_mixin_registry.register` make the preprocessing config available to `global_defaults` in Ludwig configs. - `@ecd_input_config_registry.register` / `@ecd_output_config_registry.register` wire the schema into the ECD model config builder. ______________________________________________________________________ ## Step 3 — Write the preprocessing config Create `ludwig/schema/features/preprocessing/widget_feature_preprocessing.py` if your feature needs non-default preprocessing parameters, or register your type against an existing one (e.g. `number_feature` for scalars). For a new type, create the file: ```python from ludwig.schema.features.preprocessing.base import BasePreprocessingConfig from ludwig.schema.features.preprocessing.utils import register_preprocessor from ludwig.constants import WIDGET @register_preprocessor(WIDGET) class WidgetPreprocessingConfig(BasePreprocessingConfig): # add preprocessing hyperparameters here pass ``` ______________________________________________________________________ ## Step 4 — Write the feature module Create `ludwig/features/widget_feature.py`. The required classes are: ### Inner preprocessing module ```python import torch from ludwig.features.base_feature import BasePreprocessingModule, FeaturePreprocessingMixin, InputFeature, OutputFeature class _WidgetPreprocessing(BasePreprocessingModule): """Runs inside the model graph during inference to preprocess raw input.""" def __init__(self, metadata: dict, preprocessing_config, is_input_feature: bool = True): super().__init__() # store everything needed to preprocess at inference time def forward(self, v): # v is the raw column value; return a tensor raise NotImplementedError ``` ### FeatureMixin (shared preprocessing logic) `FeaturePreprocessingMixin` provides the Python-side preprocessing used during dataset preparation (not inside the model graph). You must implement `add_feature_data` and `get_preprocessing_module`: ```python class WidgetFeatureMixin(FeaturePreprocessingMixin): @staticmethod def type(): return WIDGET @staticmethod def cast_column(column, backend): """Cast the raw DataFrame column to the expected dtype.""" return column @staticmethod def add_feature_data( feature_config, input_df, proc_df, metadata, preprocessing_parameters, backend, skip_save_processed_input, ): """Populate proc_df[feature_config[PROC_COLUMN]] with preprocessed values.""" proc_df[feature_config[PROC_COLUMN]] = input_df[feature_config[COLUMN]].values return proc_df @staticmethod def fill_missing_values(feature_config, input_df, backend): """Replace NaN/None with a fill value appropriate for this type.""" return input_df @staticmethod def feature_meta(column, preprocessing_parameters, backend): """Compute and return the training-set-level metadata dict for this feature.""" return {} @staticmethod def get_preprocessing_module(feature_config, metadata): """Return the _WidgetPreprocessing module for use during inference.""" return _WidgetPreprocessing(metadata, feature_config.preprocessing) ``` ### InputFeature class ```python from ludwig.schema.features.widget_feature import WidgetInputFeatureConfig class WidgetInputFeature(WidgetFeatureMixin, InputFeature): def __init__(self, input_feature_config: WidgetInputFeatureConfig, encoder_obj=None, **kwargs): super().__init__(input_feature_config, **kwargs) self._input_shape = torch.Size([1]) # set to actual encoded shape if encoder_obj: self.encoder_obj = encoder_obj else: self.encoder_obj = self.initialize_encoder(input_feature_config.encoder) def forward(self, inputs, mask=None): assert inputs.dtype == torch.float32 encoder_output = self.encoder_obj(inputs, mask=mask) return {"encoder_output": encoder_output} @property def input_dtype(self): return torch.float32 @property def input_shape(self): return self._input_shape @property def output_shape(self): return self.encoder_obj.output_shape @staticmethod def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs): pass @staticmethod def create_sample_input(batch_size=2): return torch.zeros(batch_size, 1) @staticmethod def get_schema_cls(): return WidgetInputFeatureConfig ``` ### OutputFeature class (only if this type can be a target) ```python from ludwig.schema.features.widget_feature import WidgetOutputFeatureConfig class WidgetOutputFeature(WidgetFeatureMixin, OutputFeature): def __init__(self, output_feature_config: WidgetOutputFeatureConfig, output_features: dict, **kwargs): super().__init__(output_feature_config, output_features, **kwargs) self._input_shape = torch.Size([output_feature_config.input_size]) self.decoder_obj = self.initialize_decoder(output_feature_config.decoder) self._setup_loss() self._setup_metrics() def logits(self, inputs, target=None): return self.decoder_obj(inputs) def create_predict_module(self): return _WidgetPredict() # see PredictModule below def get_prediction_set(self): return {LOGITS, PREDICTIONS, PROBABILITIES} @classmethod def update_config_with_metadata(cls, feature_config, feature_metadata, *args, **kwargs): feature_config.input_size = feature_metadata["input_size"] @staticmethod def get_schema_cls(): return WidgetOutputFeatureConfig ``` ### PredictModule (for output features) ```python from ludwig.features.base_feature import PredictModule class _WidgetPredict(PredictModule): def forward(self, inputs, feature_name): logits = inputs[f"{feature_name}_{LOGITS}"] predictions = (logits > 0.5).float() return {PREDICTIONS: predictions, LOGITS: logits} ``` ______________________________________________________________________ ## Step 5 — Register in the feature registries Open `ludwig/features/feature_registries.py` and add your classes to all relevant registry functions: ```python # at the top — add import from ludwig.features.widget_feature import WidgetFeatureMixin, WidgetInputFeature # in get_base_type_registry(), inside the returned dict: # WIDGET: WidgetFeatureMixin, # # in get_input_type_registry(), inside the returned dict: # WIDGET: WidgetInputFeature, # # in get_output_type_registry() if applicable, inside the returned dict: # WIDGET: WidgetOutputFeature, ``` The model builder uses `get_input_type_registry()` and `get_output_type_registry()` to instantiate feature objects from config at training time. ______________________________________________________________________ ## Step 6 — Register the constant in constants.py (feature sets) If the feature appears in `FEATURE_TYPES`, `INPUT_FEATURE_TYPES`, or similar sets, add `WIDGET` there too. ______________________________________________________________________ ## Step 7 — Write tests Create `tests/ludwig/features/test_widget_feature.py`. At minimum test: 1. `WidgetFeatureMixin.add_feature_data` — correct column values written to `proc_df` 1. `_WidgetPreprocessing.forward` — correct tensor shape for a known input 1. `WidgetInputFeature.forward` — correct output keys and shapes with a random input 1. Encoder round-trip via `create_sample_input` ```python import torch import pytest from tests.integration_tests.utils import generate_data, run_api_test def test_widget_preprocessing_forward(): meta = {} module = _WidgetPreprocessing(meta, preprocessing_config=None) out = module(torch.zeros(4)) assert out.shape == (4, 1) ``` ______________________________________________________________________ ## Checklist - [ ] `ludwig/constants.py` — add `WIDGET = "widget"` - [ ] `ludwig/schema/features/widget_feature.py` — schema classes + registry decorators - [ ] `ludwig/schema/features/preprocessing/` — preprocessing config class (or reuse existing) - [ ] `ludwig/features/widget_feature.py` — preprocessing module, mixin, input/output feature classes - [ ] `ludwig/features/feature_registries.py` — add to `get_base_type_registry`, `get_input_type_registry`, optionally `get_output_type_registry` - [ ] `tests/ludwig/features/test_widget_feature.py` — unit tests for preprocessing and forward pass ______________________________________________________________________ ## Common pitfalls **`proc_df[PROC_COLUMN]` vs `proc_df[COLUMN]`** — always write to `PROC_COLUMN` (the internal column name), not `COLUMN` (the raw user column name). They can differ when the user renames features. **`get_preprocessing_module` vs `add_feature_data`** — `add_feature_data` runs in Python at dataset preparation time (CPU, pandas). `get_preprocessing_module` returns a `torch.nn.Module` that runs inside the model graph at inference time. Both must produce compatible representations. **`input_shape` vs `output_shape`** — `InputFeature.input_shape` is the shape of the *raw preprocessed* tensor going into the encoder. `InputFeature.output_shape` is the encoder's output shape that feeds into the combiner. Return `self.encoder_obj.output_shape` for the latter. **Registry order matters** — the registry in `feature_registries.py` is read at import time. If you import your feature class before `feature_registries.py` is loaded, the registry will be empty. The correct order is always: define constants → define schema → define feature → add to registry. **Schema `type` field** — always use `schema_utils.ProtectedString(WIDGET)` not `str = WIDGET`. The protected string raises an error if a user tries to override it in their config YAML, which prevents subtle type mismatches. --- ## File: examples/ray/kubernetes/README.md ## Running on Kubernetes ### Connect to k8s cluster with a Ray operator You should now be pointing to your cluster with `kubectl`. Check the nodes to make sure you're connected correctly: ``` kubectl get nodes ``` We recommend using the [Kuberay](https://github.com/ray-project/kuberay) implementation of the Ray Operator to launch Ray clusters. ### Configure the Ray cluster First choose your preferred cluster template from `clusters`, for example: ``` export CLUSTER_NAME=ludwig-ray-cpu-cluster ``` ### Start the cluster ``` ./utils/ray_up.sh $CLUSTER_NAME ``` ### Submit a script for execution ``` ./utils/submit.sh $CLUSTER_NAME scripts/train.py ``` ### SSH into the head node ``` ./utils/attach.sh $CLUSTER_NAME ``` ### Run the Ray Dashboard ``` ./utils/dashboard.sh $CLUSTER_NAME ``` Navigate to http://localhost:8267 ### (For Ludwig Developers) Sync local Ludwig repo ``` ./utils/rsync_up.sh $CLUSTER_NAME ~/repos/ludwig ``` ### Shutdown the cluster ``` ./utils/ray_down.sh $CLUSTER_NAME ``` ### Connecting to remote filesystems (S3, GCS, etc.) Build a custom Docker image deriving from `ludwig-ray` or `ludwig-ray-gpu` containing the library needed for your data: - `s3fs` - `adlfs` - `gcsfs` Set environment variables into the cluster YAML definition with your credentials. For example, you can connect to S3 using the environment variables described in the [boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables). You could also include the credentials directly into the Docker image if they don't need to be configured at runtime. --- ## File: examples/ray/job_submission/README.md # Ray Job Submission for Ludwig Run Ludwig training on a remote Ray cluster using Ray Job Submission instead of Ray Client. ## Why Ray Job Submission? Ray Client mode (`ray.init("ray://head:10001")`) has known issues with `ray.data` operations ([ray-project/ray#47759](https://github.com/ray-project/ray/issues/47759)), causing `OwnerDiedError` and similar failures during distributed training. Ray Job Submission avoids this entirely by running the training script directly on the cluster head node. ## How it works ``` Your machine Ray Cluster +------------------+ +------------------+ | submit_job.py | --- uploads ---> | train_on_cluster.py | config.yaml | config + | (runs on head node) | | script | ray.init() is local | | <-- streams --- | ludwig.train() | | logs back | saves to S3/NFS +------------------+ +------------------+ ``` 1. `submit_job.py` runs on your machine and uploads the config + training script 1. `train_on_cluster.py` runs on the cluster head node 1. `ray.init()` connects locally (no Client mode) 1. Ludwig distributes training across workers normally 1. Model is saved to shared storage (S3/GCS/NFS) ## Prerequisites **Your machine:** ```bash pip install "ray[default]" ``` **Ray cluster:** ```bash pip install "ludwig[distributed]" ``` Or install at job start with `--pip ludwig[distributed]` (adds cold start time). ## Usage ```bash # Basic usage python submit_job.py \ --ray-address http://ray-head:8265 \ --config config.yaml \ --dataset s3://my-bucket/data/train.csv \ --output-dir s3://my-bucket/results/ # KubeRay cluster python submit_job.py \ --ray-address http://ray-head.ray.svc:8265 \ --config config.yaml \ --dataset s3://my-bucket/data/train.csv \ --output-dir s3://my-bucket/results/ # Install Ludwig on the fly (no pre-install on cluster) python submit_job.py \ --ray-address http://ray-head:8265 \ --config config.yaml \ --dataset s3://my-bucket/data/train.csv \ --output-dir /shared/nfs/results/ \ --pip "ludwig[distributed]" # Submit without waiting for results python submit_job.py \ --ray-address http://ray-head:8265 \ --config config.yaml \ --dataset s3://my-bucket/data/train.csv \ --output-dir s3://my-bucket/results/ \ --no-follow ``` ## Data access The dataset must be accessible **from the cluster**, not from your machine: | Storage | Example path | Notes | | ------- | -------------------------- | ----------------------------- | | S3 | `s3://bucket/data.csv` | Cluster needs AWS credentials | | GCS | `gs://bucket/data.csv` | Cluster needs GCP credentials | | NFS | `/shared/data/train.csv` | Must be mounted on all nodes | | HDFS | `hdfs://namenode/data.csv` | Hadoop cluster | If your data is local, upload it first: ```bash aws s3 cp my_data.csv s3://my-bucket/data/my_data.csv ``` ## Files - `submit_job.py` -- runs on your machine, submits the job - `train_on_cluster.py` -- runs on the cluster, does the actual training - `config.yaml` -- sample Ludwig config (customize for your task) ## Customization **Using your own config**: Replace `config.yaml` with your Ludwig config. Any valid Ludwig config works. **Requesting GPUs for the driver**: Use `--num-gpus 1` if your training script needs GPU access on the head node. **Custom runtime environment**: Edit `submit_job.py` to add `runtime_env` options like `conda`, `container`, or `env_vars`. --- ## File: examples/peft_advanced/README.md # Advanced PEFT Adapters in Ludwig This directory contains examples demonstrating Ludwig's extended PEFT (Parameter-Efficient Fine-Tuning) adapter support, including: - **PiSSA / EVA / CorDA / LoftQ** — advanced LoRA initializers - **rsLoRA** — rank-stabilized LoRA scaling - **TinyLoRA** — extreme low-rank fine-tuning (LoRA-XS variant) - **C3A** — contextual/conditional/compositional adapters - **OFT / HRA** — orthogonal fine-tuning methods - **WaveFT** — wavelet-domain fine-tuning - **LN-Tuning** — layer normalization only - **VBLoRA** — vector bank LoRA ## Files | File | Description | | --------------------- | ------------------------------------------------------------ | | `pissa_lora.yaml` | PiSSA initialization (faster convergence than standard LoRA) | | `eva_lora.yaml` | EVA initialization (data-driven, SOTA performance) | | `corda_lora.yaml` | CorDA initialization (combines PiSSA + context signals) | | `loftq_lora.yaml` | LoftQ (quantization-aware LoRA init) | | `rslora_dora.yaml` | rsLoRA + DoRA combination | | `tinylora_llm.yaml` | TinyLoRA for LLM fine-tuning on minimal hardware | | `c3a_llm.yaml` | C3A adapter for multi-task scenarios | | `oft_llm.yaml` | OFT adapter (orthogonal, preserves pretrained knowledge) | | `hra_llm.yaml` | HRA adapter (Householder reflections) | | `waveft_llm.yaml` | WaveFT adapter (frequency-domain updates) | | `ln_tuning_llm.yaml` | LN-Tuning (ultra-lightweight: only LayerNorm weights) | | `vblora_llm.yaml` | VBLoRA (shared vector bank for extreme compression) | | `compare_adapters.py` | Script comparing adapters by parameter count | | `train_example.py` | Full training example with adapter selection | ## Quick Start ```bash # Train with PiSSA (recommended for most tasks — faster convergence) ludwig train --config pissa_lora.yaml --dataset ludwig://imdb # Ultra-low memory: TinyLoRA ludwig train --config tinylora_llm.yaml --dataset ludwig://imdb # Orthogonal fine-tuning (preserves pretrained knowledge) ludwig train --config oft_llm.yaml --dataset ludwig://imdb ``` ## Adapter Selection Guide | Hardware constraint | Recommended adapter | Params (7B model) | | ------------------- | ------------------------ | ----------------- | | 80 GB GPU | `lora` r=16 + PiSSA init | ~100M | | 24 GB GPU | `lora` r=8 + rsLoRA | ~50M | | 16 GB GPU | `tinylora` r=2 | ~1M | | 8 GB GPU | `ln_tuning` | ~0.1M | | Edge / CPU | `tinylora` r=2, u=13 | \<100K | --- ## File: examples/optimizers/README.md # Optimizer Comparison: Schedule-Free, Muon, Adafactor, and More [](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/optimizers/optimizer_comparison.ipynb) ## Why optimizer choice matters The optimizer is more than a training detail — it controls how fast gradients are translated into weight updates, whether training is stable in early epochs, how much memory the optimizer state consumes, and whether you need to tune a separate learning-rate schedule at all. Ludwig 0.11 added five production-ready optimizers beyond the classic Adam/SGD family: **RAdam**, **Adafactor**, **Schedule-Free AdamW**, **Muon**, and **SOAP**. This example shows how to configure each one and compares them on a real dataset. ## What this example shows - How to set `trainer.optimizer.type` in a Ludwig YAML config - The one rule for Schedule-Free AdamW: no `learning_rate_scheduler` - Side-by-side training curves (validation loss + accuracy) for all optimizers - A summary table of final metrics and wall-clock training time ## Prerequisites ```bash pip install ludwig ``` No GPU required. The notebook runs on CPU in a few minutes. ## Quick start ### Run the notebook (recommended) Open [`optimizer_comparison.ipynb`](optimizer_comparison.ipynb) in Jupyter or click the Colab badge above. ### Run the script ```bash python optimizer_comparison.py ``` This downloads the UCI Wine Quality dataset, trains all five configs, and prints a comparison table. ### Use a standalone YAML config Each optimizer has its own config file you can use directly with the Ludwig CLI: ```bash ludwig train --config config_schedule_free_adamw.yaml --dataset winequality-red.csv ``` | File | Optimizer | | --------------------------------- | ------------------- | | `config_adamw.yaml` | AdamW (baseline) | | `config_radam.yaml` | RAdam | | `config_adafactor.yaml` | Adafactor | | `config_schedule_free_adamw.yaml` | Schedule-Free AdamW | | `config_muon.yaml` | Muon | ## Key insight: Schedule-Free AdamW needs no LR scheduler ```yaml trainer: optimizer: type: schedule_free_adamw lr: 0.001 # Do NOT add learning_rate_scheduler here ``` Adding a `learning_rate_scheduler` on top of `schedule_free_adamw` fights the built-in schedule and hurts convergence. See the notebook for a detailed explanation. --- ## File: examples/open_set_recognition/README.md # Open-Set Recognition with Agnostophobia Losses ## MNIST Tutorial [](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/open_set_recognition/open_set_mnist.ipynb) The notebook `open_set_mnist.ipynb` walks through the full open-set recognition workflow on a real image dataset: - **Dataset**: MNIST digits — classes 0–7 are *known*, classes 8–9 act as *unknown/background* - **Models**: three Ludwig image classifiers using `stacked_cnn` encoder and `category` output - CE Baseline (`softmax_cross_entropy`) — trained on known classes only - Entropic Open-Set (`entropic_open_set`) — entropy maximisation on background samples - Objectosphere (`objectosphere`) — norm push on known + norm suppression on background - **Evaluation**: confidence histograms and ROC curves for unknown detection The notebook is Colab-compatible — it installs Ludwig and torchvision, downloads MNIST, saves images to disk, builds `train.csv`/`test.csv`, trains all three models, and plots results. YAML configs for standalone use: - `config_baseline_mnist.yaml` - `config_entropic_mnist.yaml` - `config_objectosphere_mnist.yaml` ______________________________________________________________________ ## Quick Validation Script This example reproduces the key findings from: > Dhamija, A. R., Günther, M., & Boult, T. (2018). > **Reducing Network Agnostophobia.** > *NeurIPS 2018.* https://arxiv.org/abs/1811.04110 Standard classifiers are trained to output high-confidence predictions for every input — even inputs from classes never seen during training. This is called *network agnostophobia*: the network is incapable of expressing "I don't know." The paper proposes two loss functions that address this: | Loss | Description | | --------------------- | -------------------------------------------------------------------------- | | **Entropic Open-Set** | CE on known samples + entropy maximisation on background samples | | **Objectosphere** | CE + logit-norm push for known + entropy + norm suppression for background | Both are available in Ludwig's category and binary output features. ### Quick start ```bash pip install ludwig python train_open_set.py ``` The script generates a synthetic two-class-family dataset (four known Gaussian clusters + two unknown clusters), trains three classifiers, and prints a comparison table showing mean max probability on unknowns — lower is better for open-set recognition. Expected output (approximate): ``` Model | Max-prob (known) | Max-prob (unknown) | Norm known | Norm unknown -----------------------|-----------------|-------------------|------------|------------- CE Baseline | 0.998 | 0.741 | 8.828 | 5.375 Entropic Open-Set | 0.974 | 0.273 | 6.254 | 0.637 Objectosphere | 0.874 | 0.363 | 13.843 | 2.361 ``` ### Ludwig configuration #### Entropic Open-Set Loss ```yaml output_features: - name: label type: category loss: type: entropic_open_set background_class: 4 # integer index of the background/unknown class ``` #### Objectosphere Loss ```yaml output_features: - name: label type: category loss: type: objectosphere background_class: 4 xi: 10.0 # minimum logit norm for known-class samples zeta: 0.1 # weight for unknown-class magnitude suppression ``` `background_class` is the **integer index** of the background/unknown class in Ludwig's vocabulary for that feature. You can discover it by inspecting the saved model's `training_set_metadata.json` file after a training run — look for the `str2idx` field of the relevant output feature. ### Inference-time unknown detection For **Objectosphere** models, unknown inputs can be detected using a simple threshold on the logit L2 norm: ```python predictions = model.predict(dataset=df) # Retrieve raw logits via the API (requires model.collect_activations) import torch norms = logit_tensor.norm(dim=-1) is_unknown = norms < threshold # choose threshold from validation set ``` For both loss types, you can also use the **maximum softmax probability** as a simpler threshold: samples with max-prob below some value (e.g. 0.5) are flagged as unknown. --- ## File: examples/multi_task/README.md # Multi-Task Learning with Nash-MTL Loss Balancing [](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/multi_task/multi_task.ipynb) > **Note:** Nash-MTL requires PR #4092 (`future-capabilities` branch) and is not yet available in the main Ludwig release. The FAMO and uncertainty weighting methods shown here are available now. ## Overview This example demonstrates multi-task learning with Ludwig: training a single model to predict multiple outputs simultaneously, and using **loss balancing** to prevent one task from dominating training. The dataset is the [UCI Wine Quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality). We predict two outputs at once: - `quality_score` — the raw 0–10 quality score (regression) - `quality_binary` — whether the wine is good (quality ≥ 7, binary classification) These two tasks have different loss magnitudes. Without balancing, the regression loss typically dominates and the classifier under-trains. ## What You Will Learn 1. How to define multiple output features in a Ludwig config 1. Why loss magnitudes differ between regression and classification tasks 1. How FAMO and uncertainty weighting improve multi-task training (available now) 1. What Nash-MTL does and how it compares to heuristic methods (requires PR #4092) 1. How to read a comparison table and choose the right balancing strategy ## Loss Balancing Methods Compared | Method | Status | When to use | | --------------- | ----------------- | ---------------------------------------------- | | `none` | Available | Baseline; tasks have similar loss scales | | `log_transform` | Available | Quick improvement with no hyperparameters | | `uncertainty` | Available | Tasks have stable, learnable scale differences | | `famo` | Available | General purpose; good default choice | | `gradnorm` | Available | Gradient-level balancing; more expensive | | `nash_mtl` | Requires PR #4092 | Most principled; best when tasks conflict | ## Quick Start ```bash pip install ludwig # Baseline ludwig train --config config_no_balancing.yaml --dataset wine_quality_dual.csv # FAMO (available now) ludwig train --config config_famo.yaml --dataset wine_quality_dual.csv # Uncertainty weighting (available now) ludwig train --config config_uncertainty.yaml --dataset wine_quality_dual.csv # Nash-MTL (requires PR #4092) ludwig train --config config_nash_mtl.yaml --dataset wine_quality_dual.csv ``` Or run the full comparison script: ```bash python train_multi_task.py ``` ## Files | File | Description | | -------------------------- | ------------------------------------------ | | `multi_task.ipynb` | Interactive notebook with full walkthrough | | `train_multi_task.py` | Standalone Python script | | `config_no_balancing.yaml` | Baseline config — no loss balancing | | `config_famo.yaml` | FAMO balancing (available now) | | `config_uncertainty.yaml` | Uncertainty weighting (available now) | | `config_nash_mtl.yaml` | Nash-MTL balancing (requires PR #4092) | ## Prerequisites - Python 3.9+ - Ludwig installed (`pip install ludwig`) - Internet access to download the UCI Wine Quality dataset (~80 KB) Optional: GPU for faster training (not required). ## Background ### Multi-Task Learning Multi-task learning trains a shared model to predict several outputs simultaneously. The shared representation encourages the model to learn features useful across tasks, often improving generalisation compared to separate single-task models — especially when training data is limited. ### The Loss Balancing Problem When tasks have different loss scales (e.g., MSE for regression vs. cross-entropy for binary classification), their gradients have different magnitudes. During backpropagation, the task with larger gradients dominates parameter updates and the other task effectively under-trains. Loss balancing methods assign adaptive weights to each task's loss so that all tasks contribute proportionately to the total gradient. ### Nash-MTL Nash-MTL (Navon et al., ICML 2022) frames loss balancing as a Nash bargaining game. Rather than using heuristic rules or hand-tuned weights, it finds the unique solution where no task can improve its loss without worsening another task's loss. This makes it the most principled approach, particularly valuable when tasks genuinely conflict. See [Navon et al., 2022](https://arxiv.org/abs/2202.01017) for the theoretical grounding. --- ## File: examples/mnist/README.md # MNIST Hand-written Digit Classification This API example is based on [Ludwig's MNIST Hand-written Digit image classification example](https://ludwig-ai.github.io/ludwig-docs/examples/#image-classification-mnist). ### Examples | File | Description | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | simple_model_training.py | Demonstrates using Ludwig api for training a model. | | advance_model_training.py | Demonstrates a method to assess alternative model architectures. | | assess_model_performance.py | Assess model performance on hold-out test data set. This shows how to load a previously trained model to make predictions. | | visualize_model_test_results.ipynb | Example for extracting training statistics and generate custom visualizations. | ## 2. Official Technical Reference & Guides (ludwig-ai/ludwig-docs) ## File: README.md # Ludwig documentation Website: [ludwig.ai](ludwig.ai) Ludwig's documentation is built using [MkDocs](https://www.mkdocs.org/) and the beautiful [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/) theme, deployed using [Mike](https://github.com/jimporter/mike). ## Edit-refresh development 1. Install requirements. ``` pip install -r requirements.txt ``` 2. If the content of [contributing guide](https://github.com/ludwig-ai/ludwig/blob/master/CONTRIBUTING.md) in the Ludwig code repository has changed, then run ``` python code_doc_autogen.py ``` in order to download it into `docs/developer_guide/contributing.md` in this repository (since all documentation content is served from local files). Be sure to commit the new version of the contributor guide into the repository. If the contributor guide source did not change, then skip this step. 3. In terminal, keep a window running: ``` mkdocs serve ``` 4. Navigate to to view your changes. > :bulb: No need to return to your terminal -- changes will be automatically > reflected as you save changes to files. ## Versioned docs The full `ludwig.ai` website is deployed using `mike`, a wrapper around `mkdocs`, which deploys includes previous snapshots of documentation for older versions of Ludwig. To see how the fully rendered `ludwig.ai` website with multiple versions looks: 1. Export the ludwig version to an environment variable: ``` export LUDWIG_VERSION=$(python -c "import ludwig; print('.'.join(ludwig.__version__.split('.')[:2]))") ``` 2. Run `mike deploy` ``` mike deploy --update-aliases $LUDWIG_VERSION latest --ignore ``` 3. In a separate tab, run the `mike` web server: ``` mike serve --ignore ``` 4. Navigate to to view. > :warning: `mike serve` is **not** edit-refreshable. In order to see changes reflected, re-run `mike deploy` and `mike serve`. > > ``` > mike deploy --update-aliases $LUDWIG_VERSION latest --ignore > mike serve --ignore > ``` ## Updating docs for older Ludwig versions The CI system will by default publish new docs for the latest version every day. Updating docs for an older version of Ludwig needs to be done manually. Create a new branch: ``` git checkout -b $VERSION ``` Install the relevant version of Ludwig and generate documentation: ``` pip install ludwig==$VERSION python code_doc_autogen.py ``` Use the `--push` option to publish the changes to the remote repo. Be sure to only include the major and minor version (e.g., `0.5` instead of `0.5.1`): ``` mike deploy --push $MAJOR_MINOR_VERSION ``` ## Regenerating API documentation Markdown files under `docs/user_guide/api/` are generated automatically. To regenerate these files, run: ``` python code_doc_autogen.py ``` --- ## File: docs/configuration/features/audio_features.md {% from './macros/includes.md' import render_fields, render_yaml %} {% set mv_details = "See [Missing Value Strategy](./input_features.md#missing-value-strategy) for details." %} {% set type = "See explanations for each type [here](audio_features.md#input-features)." %} {% set details = {"missing_value_strategy": mv_details, "type": type} %} # Preprocessing Example of a preprocessing specification (assuming the audio files have a sample rate of 16000): {% set preprocessing = get_feature_preprocessing_schema("audio") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Ludwig supports reading audio files using PyTorch's [Torchaudio](https://pytorch.org/audio/stable/index.html) library. This library supports `WAV`, `AMB`, `MP3`, `FLAC`, `OGG/VORBIS`, `OPUS`, `SPHERE`, and `AMR-NB` formats. Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} Preprocessing parameters can also be defined once and applied to all audio input features using the [Type-Global Preprocessing](../defaults.md#type-global-preprocessing) section. ## Preprocessing Modes Ludwig supports three preprocessing modes for audio features, controlled by the `mode` parameter: | Mode | Preprocessing memory | Training epoch 1 | Training epoch 2+ | Best for | |------|----------------------|------------------|---------------------|----------| | `eager` | High (O(N×tensor)) | Fast | Fast | Small datasets that fit in RAM | | `lazy` (default) | Low (O(batch)) | Slower (decode-bound) | Slower | Large datasets | | `lazy_cached` | Low (O(batch)) | Fast (GPU pipelined) | Very fast (memmap) | Large datasets, any GPU speed | ### `mode: lazy` (default) Ludwig stores file paths in the processed dataset and decodes audio clips on-the-fly, one batch at a time, during training. Decoding runs in a `ThreadPoolExecutor` that overlaps with the GPU forward pass. For FBANK features, the thread-pool size is automatically capped to avoid CPU over-subscription (each decode already uses PyTorch's internal thread pool). ### `mode: lazy_cached` On the first training epoch, audio is decoded per batch (same as `lazy`) and written to a numpy memmap alongside the Parquet cache. From epoch 2 onward, the memmap is read directly (~0.1 ms/batch), eliminating decode overhead entirely. ### `mode: eager` All audio files are decoded during preprocessing and stored as tensors in the Parquet cache. Use this only when the full decoded dataset fits comfortably in memory. ### Configuration Examples ```yaml input_features: - name: audio type: audio preprocessing: mode: lazy # default prefetch_size: null # auto (4 for lazy/lazy_cached, 0 for eager) lazy_cache_dir: null # default: ~/.cache/ludwig/lazy_media// audio_file_length_limit_in_s: 7.5 type: fbank num_filter_bands: 80 ``` ```yaml input_features: - name: audio type: audio preprocessing: mode: lazy_cached # decode+cache on epoch 1; memmap from epoch 2+ lazy_cache_dir: /fast/nvme/audio_cache ``` ```yaml input_features: - name: audio type: audio preprocessing: mode: eager # decode everything upfront ``` See [Choosing a Preprocessing Mode](../../user_guide/datasets/data_preprocessing.md#choosing-a-preprocessing-mode) for a full comparison. ### Lazy Preprocessing with HuggingFace Datasets When loading a HuggingFace dataset (e.g. `datasets.load_dataset(...)`), audio columns are delivered as Python dicts — not file paths: ```python { "array": np.ndarray, # decoded waveform, shape (samples,) "sampling_rate": 16000, # sample rate in Hz "path": "/path/to/cache.wav", # optional: HF's local cache path } ``` Ludwig handles this transparently: 1. **If `path` points to an existing file on disk** (HuggingFace's local cache), Ludwig reuses that file directly — no copy is made. 2. **Otherwise**, Ludwig writes the waveform to a WAV file in `lazy_cache_dir` and uses that path. The cache is persistent and idempotent — subsequent runs skip the write step entirely. ### Controlling the Cache Directory `lazy_cache_dir` controls where WAV files are written for in-memory sources (HuggingFace datasets). The decoded memmap for `lazy_cached` mode is placed next to the Parquet cache, not inside `lazy_cache_dir`. ```yaml input_features: - name: speech type: audio preprocessing: mode: lazy_cached lazy_cache_dir: /fast/nvme/my_project/audio_cache ``` The per-feature subdirectory is created automatically. ### Bare Tensor Inputs If your dataset delivers bare `torch.Tensor` objects (shape `(channels, samples)` or `(samples,)`) instead of dicts, Ludwig treats them the same as the in-memory dict case: tensors are written to WAV files in `lazy_cache_dir` using the sample rate recorded in the feature metadata. # Input Features Audio files are transformed into one of the following types according to `type` under the `preprocessing` configuration. - **`raw`**: Audio file is transformed into a float valued tensor of size `N x L x W` (where `N` is the size of the dataset and `L` corresponds to `audio_file_length_limit_in_s * sample_rate` and `W = 1`). - **`stft`**: Audio is transformed to the `stft` magnitude. Audio file is transformed into a float valued tensor of size `N x L x W` (where `N` is the size of the dataset, `L` corresponds to `ceil(audio_file_length_limit_in_s * sample_rate - window_length_in_s * sample_rate + 1/ window_shift_in_s * sample_rate) + 1` and `W` corresponds to `num_fft_points / 2`). - **`fbank`**: Audio file is transformed to FBANK features (also called log Mel-filter bank values). FBANK features are implemented according to their definition in the [HTK Book](http://www.inf.u-szeged.hu/~tothl/speech/htkbook.pdf): Raw Signal -> Preemphasis -> DC mean removal -> `stft` magnitude -> Power spectrum: `stft^2` -> mel-filter bank values: triangular filters equally spaced on a Mel-scale are applied -> log-compression: `log()`. Overall the audio file is transformed into a float valued tensor of size `N x L x W` with `N,L` being equal to the ones in `stft` and `W` being equal to `num_filter_bands`. - **`stft_phase`**: The phase information for each stft bin is appended to the `stft` magnitude so that the audio file is transformed into a float valued tensor of size `N x L x 2W` with `N,L,W` being equal to the ones in `stft`. - **`group_delay`**: Audio is transformed to group delay features according to Equation (23) in this [paper](https://www.ias.ac.in/article/fullyext/sadh/036/05/0745-0782). Group_delay features has the same tensor size as `stft`. The encoder parameters specified at the feature level are: - **`tied`** (default `null`): name of another input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. Example audio feature entry in the input features list: ```yaml name: audio_column_name type: audio tied: null encoder: type: parallel_cnn ``` ## Encoders Audio feature encoders include all [Sequence Features](sequence_features.md#input-features) encoders as well as the pretrained audio encoders described below. Encoder type and encoder parameters can also be defined once and applied to all audio input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ### Wav2Vec2 Encoder The Wav2Vec2 encoder (Baevski et al., "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations", NeurIPS 2020) processes raw audio waveforms using self-supervised contrastive learning over masked latent representations. It produces contextualized speech features suitable for speech recognition, audio classification, and speaker identification. Wav2Vec2 expects raw waveform input at 16kHz sample rate. Default pretrained model: `facebook/wav2vec2-base` {% set wav2vec2_encoder = get_encoder_schema("audio", "wav2vec2") %} {{ render_yaml(wav2vec2_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(wav2vec2_encoder, exclude=["type"])) }} ### Whisper Encoder The Whisper encoder (Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision", ICML 2023) is the encoder portion of OpenAI's Whisper model, trained on 680,000 hours of multilingual audio data. It excels at noisy and multilingual speech tasks. Whisper expects log-mel spectrogram input (80 mel bins). Default pretrained model: `openai/whisper-base` {% set whisper_encoder = get_encoder_schema("audio", "whisper") %} {{ render_yaml(whisper_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(whisper_encoder, exclude=["type"])) }} ### HuBERT Encoder The HuBERT encoder (Hsu et al., "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", IEEE/ACM TASLP 2021) uses self-supervised masked prediction to learn speech representations. It is particularly effective for speaker verification, emotion recognition, and audio classification tasks. HuBERT expects raw waveform input at 16kHz sample rate. Default pretrained model: `facebook/hubert-base-ls960` {% set hubert_encoder = get_encoder_schema("audio", "hubert") %} {{ render_yaml(hubert_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(hubert_encoder, exclude=["type"])) }} # Output Features There are no audio decoders at the moment. If this unlocks an interesting use case for your application, please file a GitHub Issue or ping the [Community Discord](https://discord.gg/CBgdrGnZjy). --- ## File: docs/configuration/features/bag_features.md {% from './macros/includes.md' import render_fields, render_yaml %} {% set mv_details = "See [Missing Value Strategy](./input_features.md#missing-value-strategy) for details." %} {% set norm_details = "See [Normalization](../combiner.md#normalization) for details." %} {% set details = {"missing_value_strategy": mv_details, "norm": norm_details} %} # Preprocessing Bag features are expected to be provided as a string of elements separated by whitespace, e.g. "elem5 elem0 elem5 elem1". Bags are similar to [set features](set_features.md), the only difference being that elements may appear multiple times. The bag feature encoder outputs a matrix, similar to a set encoder, except each element of the matrix is a float value representing the frequency of the respective element in the bag. Embeddings are aggregated by summation, weighted by the frequency of each element. {% set preprocessing = get_feature_preprocessing_schema("bag") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} # Input Features Bag features have only one encoder type available: `embed`. The encoder parameters specified at the feature level are: - **`tied`** (default `null`): name of another input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. Example bag feature entry in the input features list: ```yaml name: bag_column_name type: bag tied: null encoder: type: embed ``` Encoder type and encoder parameters can also be defined once and applied to all bag input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ## Encoders ### Embed Weighted Encoder ``` mermaid graph LR A["0.0\n1.0\n1.0\n0.0\n0.0\n2.0\n0.0"] --> B["0\n1\n5"]; B --> C["emb 0\nemb 1\nemb 5"]; C --> D["Weighted\n Sum\n Operation"]; ``` { data-search-exclude } The embed weighted encoder first transforms the element frequency vector to sparse integer lists, which are then mapped to either dense or sparse embeddings (one-hot encodings). Lastly, embeddings are aggregated as a weighted sum where each embedding is multiplied by its respective element's frequency. Inputs are of size `b` while outputs are of size `b x h` where `b` is the batch size and `h` is the dimensionality of the embeddings. The parameters are the same used for [set input features](set_features.md#input-features) except for `reduce_output` which should not be used because the weighted sum already acts as a reducer. {% set encoder = get_encoder_schema("bag", "embed") %} {{ render_yaml(encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder, exclude=["type"]), details=details) }} # Output Features Bag types are not supported as output features at this time. --- ## File: docs/configuration/features/binary_features.md {% from './macros/includes.md' import render_fields, render_yaml %} {% set mv_details = "See [Missing Value Strategy](./input_features.md#missing-value-strategy) for details." %} {% set details = {"missing_value_strategy": mv_details} %} # Preprocessing Binary features are directly transformed into a binary valued vector of length `n` (where `n` is the size of the dataset) and added to the HDF5 with a key that reflects the name of column in the dataset. {% set preprocessing = get_feature_preprocessing_schema("binary") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} Preprocessing parameters can also be defined once and applied to all binary input features using the [Type-Global Preprocessing](../defaults.md#type-global-preprocessing) section. # Input Features Binary features have two encoders, `passthrough` and `dense`. The available encoder can be specified using the `type` parameter: - **`type`** (default `passthrough`): the possible values are `passthrough` and `dense`. `passthrough` outputs the raw integer values unaltered. `dense` randomly initializes a trainable embedding matrix. The encoder parameters specified at the feature level are: - **`tied`** (default `null`): name of another input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. Example binary feature entry in the input features list: ```yaml name: binary_column_name type: binary tied: null encoder: type: dense ``` Encoder type and encoder parameters can also be defined once and applied to all binary input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ## Encoders ### Passthrough Encoder The `passthrough` encoder passes through raw binary values without any transformations. Inputs of size `b` are transformed to outputs of size `b x 1` where `b` is the batch size. {% set encoder_passthrough = get_encoder_schema("binary", "passthrough") %} {{ render_yaml(encoder_passthrough, parent="encoder") }} There are no additional parameters for the `passthrough` encoder. ### Dense Encoder The `dense` encoder passes the raw binary values through a fully connected layer. Inputs of size `b` are transformed to size `b x h`. {% set encoder_dense = get_encoder_schema("binary", "dense") %} {{ render_yaml(encoder_dense, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder_dense, exclude=["type"]), details=details) }} # Output Features Binary output features can be used when a binary classification needs to be performed or when the output is a single probability. There is only one decoder available: `regressor`. Example binary output feature using default parameters: ```yaml name: binary_column_name type: binary reduce_input: sum dependencies: [] calibration: false reduce_dependencies: sum threshold: 0.5 decoder: type: regressor fc_layers: null num_fc_layers: 0 fc_output_size: 256 fc_use_bias: true fc_weights_initializer: xavier_uniform fc_bias_initializer: zeros fc_norm: null fc_norm_params: null fc_activation: relu fc_dropout: 0.0 input_size: null use_bias: true weights_initializer: xavier_uniform bias_initializer: zeros loss: type: binary_weighted_cross_entropy weight: 1.0 positive_class_weight: null robust_lambda: 0 confidence_penalty: 0 ``` Parameters: - **`reduce_input`** (default `sum`): defines how to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`dependencies`** (default `[]`): the output features this one is dependent on. For a detailed explanation refer to [Output Features Dependencies](output_features.md#output-feature-dependencies). - **`calibration`** (default `false`): if true, performs calibration by temperature scaling after training is complete. Calibration uses the validation set to find a scale factor (temperature) which is multiplied with the logits to shift output probabilities closer to true likelihoods. - **`reduce_dependencies`** (default `sum`): defines how to reduce the output of a dependent feature that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`threshold`** (defaults `0.5`): The threshold above (greater or equal) which the predicted output of the sigmoid function will be mapped to 1. - **`loss`** (default `{"type": "binary_weighted_cross_entropy"}`): is a dictionary containing a loss `type`. `binary_weighted_cross_entropy` is the only supported loss type for binary output features. See [Loss](#loss) for details. - **`decoder`** (default: `{"type": "regressor"}`): Decoder for the desired task. Options: `regressor`. See [Decoder](#decoders) for details. Decoder type and decoder parameters can also be defined once and applied to all binary output features using the [Type-Global Decoder](../defaults.md#type-global-decoder) section. ## Decoders ### Regressor ``` mermaid graph LR A["Combiner\n Output"] --> B["Fully\n Connected\n Layers"]; B --> C["Projection into\n Output Space"]; C --> D["Sigmoid"]; subgraph DEC["DECODER.."] B C D end ``` { data-search-exclude } The regressor decoder is a (potentially empty) stack of fully connected layers, followed by a projection into a single number followed by a sigmoid function. {% set decoder = get_decoder_schema("binary", "regressor") %} {{ render_yaml(decoder, parent="decoder") }} Parameters: {{ render_fields(schema_class_to_fields(decoder, exclude=["type"]), details=details) }} ## Loss ### Binary Weighted Cross Entropy {% set loss = get_loss_schema("binary_weighted_cross_entropy") %} {{ render_yaml(loss, parent="loss") }} Parameters: {{ render_fields(schema_class_to_fields(loss, exclude=["type"]), details=details) }} Loss and loss related parameters can also be defined once and applied to all binary output features using the [Type-Global Loss](../defaults.md#type-global-loss) section. ## Metrics The metrics that are calculated every epoch and are available for binary features are the `accuracy`, `loss`, `precision`, `recall`, `roc_auc` and `specificity`. You can set any of these to be the `validation_metric` in the `training` section of the configuration if the `validation_field` is set as the name of a binary feature. --- ## File: docs/configuration/features/category_features.md {% from './macros/includes.md' import render_fields, render_yaml %} {% set mv_details = "See [Missing Value Strategy](./input_features.md#missing-value-strategy) for details." %} {% set norm_details = "See [Normalization](../combiner.md#normalization) for details." %} {% set details = {"missing_value_strategy": mv_details, "fc_norm": norm_details} %} # Preprocessing Category features are transformed into integer valued vectors of size `n` (where `n` is the size of the dataset) and added to the HDF5 with a key that reflects the name of column in the dataset. Categories are mapped to integers by first collecting a dictionary of all unique category strings present in the column of the dataset, ranking them descending by frequency and assigning a sequential integer ID from the most frequent to the most rare (with 0 assigned to the special unknown placeholder token ``). The column name is added to the JSON file, with an associated dictionary containing 1. the mapping from integer to string (`idx2str`) 2. the mapping from string to id (`str2idx`) 3. the mapping from string to frequency (`str2freq`) 4. the size of the set of all tokens (`vocab_size`) 5. additional preprocessing information (by default how to fill missing values and what token to use to fill missing values) {% set preprocessing = get_feature_preprocessing_schema("category") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} Preprocessing parameters can also be defined once and applied to all category input features using the [Type-Global Preprocessing](../defaults.md#type-global-preprocessing) section. # Input Features Category features have three encoders. The `passthrough` encoder passes the raw integer values coming from the input placeholders to outputs of size `b x 1`. The other two encoders map to either `dense` or `sparse` embeddings (one-hot encodings) and returned as outputs of size `b x h`, where `b` is the batch size and `h` is the dimensionality of the embeddings. The encoder parameters specified at the feature level are: - **`tied`** (default `null`): name of another input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. Example category feature entry in the input features list: ```yaml name: category_column_name type: category tied: null encoder: type: dense ``` The available encoder parameters are: - **`type`** (default `dense`): the possible values are `passthrough`, `dense`, `sparse`, `onehot`, `target` and `hash`. `passthrough` outputs the raw integer values unaltered. `dense` randomly initializes a trainable embedding matrix, `sparse` uses one-hot encoding, `onehot` produces a one-hot vector, `target` uses mean target encoding, and `hash` uses feature hashing for fixed-memory encoding of high-cardinality categories. Encoder type and encoder parameters can also be defined once and applied to all category input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ## Encoders ### Dense Encoder {% set encoder = get_encoder_schema("category", "dense") %} {{ render_yaml(encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder, exclude=["type"]), details=details) }} ### Sparse Encoder {% set encoder = get_encoder_schema("category", "sparse") %} {{ render_yaml(encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder, exclude=["type"]), details=details) }} ### OneHot Encoder The `onehot` encoder produces a one-hot vector representation of the category. Each category is mapped to a binary vector of size equal to the vocabulary size, with a single 1 at the position corresponding to the category's index. {% set onehot_encoder = get_encoder_schema("category", "onehot") %} {{ render_yaml(onehot_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(onehot_encoder, exclude=["type"]), details=details) }} ### Target Encoder The `target` encoder replaces each category with the mean of the target variable for that category (also known as mean target encoding). This is effective for high-cardinality categorical features where a standard embedding table would be very large. Ludwig handles target leakage internally using cross-fitting on the training data. {% set target_encoder = get_encoder_schema("category", "target") %} {{ render_yaml(target_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(target_encoder, exclude=["type"]), details=details) }} ### Hash Encoder The `hash` encoder uses feature hashing (the "hashing trick") to map categories to a fixed-size embedding vector. This provides a constant memory footprint regardless of vocabulary size, handles unseen categories gracefully, and works well for extremely high-cardinality features or streaming data where the full vocabulary is not known in advance. {% set hash_encoder = get_encoder_schema("category", "hash") %} {{ render_yaml(hash_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(hash_encoder, exclude=["type"]), details=details) }} # Output Features ``` mermaid graph LR A["Combiner\n Output"] --> B["Fully\n Connected\n Layers"]; B --> C["Projection into\n Output Space"]; C --> D["Softmax"]; subgraph DEC["DECODER.."] B C D end ``` { data-search-exclude } Category features can be used when a multi-class classification needs to be performed. There is only one decoder available for category features: a (potentially empty) stack of fully connected layers, followed by a projection into a vector of size of the number of available classes, followed by a softmax. Example category output feature using default parameters: ```yaml name: category_column_name type: category reduce_input: sum dependencies: [] calibration: false reduce_dependencies: sum loss: type: softmax_cross_entropy confidence_penalty: 0 robust_lambda: 0 class_weights: null class_similarities: null class_similarities_temperature: 0 decoder: type: classifier ``` Parameters: - **`reduce_input`** (default `sum`): defines how to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`calibration`** (default `false`): if true, performs calibration by temperature scaling after training is complete. Calibration uses the validation set to find a scale factor (temperature) which is multiplied with the logits to shift output probabilities closer to true likelihoods. - **`dependencies`** (default `[]`): the output features this one is dependent on. For a detailed explanation refer to [Output Features Dependencies](output_features.md#output-feature-dependencies). - **`reduce_dependencies`** (default `sum`): defines how to reduce the output of a dependent feature that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`loss`** (default `{type: softmax_cross_entropy}`): is a dictionary containing a loss `type`. `softmax_cross_entropy` is the only supported loss type for category output features. See [Loss](#loss) for details. - **`top_k`** (default `3`): determines the parameter `k`, the number of categories to consider when computing the `top_k` measure. It computes accuracy but considering as a match if the true category appears in the first `k` predicted categories ranked by decoder's confidence. - **`decoder`** (default: `{"type": "classifier"}`): Decoder for the desired task. Options: `classifier`, `mlp_classifier`. See [Decoder](#decoders) for details. Decoder type and decoder parameters can also be defined once and applied to all category output features using the [Type-Global Decoder](../defaults.md#type-global-decoder) section. ## Decoders ### Classifier {% set decoder = get_decoder_schema("category", "classifier") %} {{ render_yaml(decoder, parent="decoder") }} Parameters: {{ render_fields(schema_class_to_fields(decoder, exclude=["type"]), details=details) }} ### MLP Classifier `mlp_classifier` adds an explicit stack of hidden layers between the combiner output and the final projection. It is useful when the combiner itself is shallow (for example the `concat` combiner over tabular features) and the classification head needs more non-linear capacity. ```yaml decoder: type: mlp_classifier num_fc_layers: 2 output_size: 512 activation: relu dropout: 0.2 ``` Setting `num_fc_layers: 0` is equivalent to the plain `classifier`. {% set decoder = get_decoder_schema("category", "mlp_classifier") %} {{ render_yaml(decoder, parent="decoder") }} Parameters: {{ render_fields(schema_class_to_fields(decoder, exclude=["type"]), details=details) }} ### Probability calibration (temperature scaling) Both `classifier` and `mlp_classifier` expose a `calibration` parameter. Set it to `temperature_scaling` to learn a single scalar temperature on the validation set that is applied as `logits / T` ([Guo et al., ICML 2017](https://arxiv.org/abs/1706.04599)). Temperature scaling never changes the argmax prediction; it only reshapes the probability distribution so that predicted confidences track empirical accuracy more closely. ```yaml decoder: type: classifier calibration: temperature_scaling ``` Temperature scaling is run automatically at the end of training using the validation split. The learned temperature is serialized with the model and applied at inference. ### Monte Carlo dropout uncertainty Both decoders also expose `mc_dropout_samples` ([Gal & Ghahramani, ICML 2016](https://arxiv.org/abs/1506.02142)). When set to a positive integer, Ludwig runs the decoder that many times at inference *with dropout enabled* and returns: - the mean predicted probabilities, - an additional `uncertainty` tensor that captures the variance across samples. ```yaml decoder: type: mlp_classifier num_fc_layers: 2 dropout: 0.2 mc_dropout_samples: 20 ``` MC dropout requires `dropout > 0` in the decoder; otherwise the repeated forward passes are identical and the uncertainty estimate is always zero. ## Loss ### Softmax Cross Entropy {% set loss = get_loss_schema("softmax_cross_entropy") %} {{ render_yaml(loss, parent="loss") }} Parameters: {{ render_fields(schema_class_to_fields(loss, exclude=["type"]), details=details) }} Loss and loss related parameters can also be defined once and applied to all category output features using the [Type-Global Loss](../defaults.md#type-global-loss) section. ## Metrics The measures that are calculated every epoch and are available for category features are `accuracy`, `hits_at_k` (computes accuracy considering as a match if the true category appears in the first `k` predicted categories ranked by decoder's confidence) and the `loss` itself. You can set either of them as `validation_metric` in the `training` section of the configuration if you set the `validation_field` to be the name of a category feature. --- # Category Distribution Output Feature `category_distribution` is a specialised output feature type for datasets where each target row is a **soft probability distribution over categories** rather than a single hard label. Typical use cases include: - Label smoothing targets generated from an ensemble or teacher model - Crowd-sourced annotations summarised as empirical agreement rates - Reading comprehension tasks with partial-credit labels The feature type is `category_distribution`. Because the targets are probability vectors, the column values must be parseable as JSON arrays of floats whose length equals the number of classes. ## Configuration ```yaml output_features: - name: label_distribution type: category_distribution vocab: [class_a, class_b, class_c] preprocessing: missing_value_strategy: drop_row decoder: type: classifier loss: type: softmax_cross_entropy ``` ### Required parameters | Parameter | Type | Description | |-----------|------|-------------| | `vocab` | list[str] | Ordered list of class names. **Required** — `category_distribution` has no way to infer the vocabulary from the soft-label vectors. | ### Preprocessing {% set preprocessing = get_feature_preprocessing_schema("category_distribution_output") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} ### Decoders `category_distribution` reuses the same `classifier` and `mlp_classifier` decoders as the standard `category` output feature. See [Decoders](#decoders) above for parameter details. ### Loss `category_distribution` targets are treated as probability distributions, so the recommended loss is `softmax_cross_entropy` with label smoothing disabled (smoothing is already encoded in the targets). ### Metrics The same metrics as `category` are reported: `accuracy` (argmax of the predicted distribution vs. argmax of the target distribution), `hits_at_k`, and `loss`. --- ## File: docs/configuration/features/date_features.md {% from './macros/includes.md' import render_fields, render_yaml %} {% set mv_details = "See [Missing Value Strategy](./input_features.md#missing-value-strategy) for details." %} {% set norm_details = "See [Normalization](../combiner.md#normalization) for details." %} {% set details = {"missing_value_strategy": mv_details, "norm": norm_details, "norm_params": norm_details} %} Date features are like `2023-06-25 15:00:00`, `2023-06-25`, `6-25-2023`, or `6/25/2023`. # Preprocessing Ludwig will try to infer the date format automatically, but a specific format can be provided. The date string spec is the same as the one described in python's [datetime](https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior). {% set preprocessing = get_feature_preprocessing_schema("date") %} {{ render_yaml(preprocessing, parent="preprocessing") }} ```yaml name: date_feature_name type: date preprocessing: missing_value_strategy: fill_with_const fill_value: '' datetime_format: "%d %b %Y" ``` Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} Preprocessing parameters can also be defined once and applied to all date input features using the [Type-Global Preprocessing](../defaults.md#type-global-preprocessing) section. # Input Features Input date features are transformed into a int tensors of size `N x 9` (where `N` is the size of the dataset and the 9 dimensions contain year, month, day, weekday, yearday, hour, minute, second, and second of day). For example, the date `2022-06-25 09:30:59` would be deconstructed into: ```python [ 2022, # Year 6, # June 25, # 25th day of the month 5, # Weekday: Saturday 176, # 176th day of the year 9, # Hour 30, # Minute 59, # Seconds 34259, # 34259th second of the day ] ``` The encoder parameters specified at the feature level are: - **`tied`** (default `null`): name of another input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. Currently there are two encoders supported for dates: `DateEmbed` (default) and `DateWave`. The encoder can be set by specifying `embed` or `wave` in the feature's `encoder` parameter in the input feature's configuration. Example date feature entry in the input features list: ```yaml name: date_feature_name type: date encoder: type: embed ``` Encoder type and encoder parameters can also be defined once and applied to all date input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ## Encoders ### Embed Encoder This encoder passes the year through a fully connected layer of one neuron and embeds all other elements for the date, concatenates them and passes the concatenated representation through fully connected layers. {% set encoder_embed = get_encoder_schema("date", "embed") %} {{ render_yaml(encoder_embed, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder_embed, exclude=["type"]), details=details) }} ### Wave Encoder This encoder passes the year through a fully connected layer of one neuron and represents all other elements for the date by taking the cosine of their value with a different period (12 for months, 31 for days, etc.), concatenates them and passes the concatenated representation through fully connected layers. {% set encoder_wave = get_encoder_schema("date", "wave") %} {{ render_yaml(encoder_wave, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder_wave, exclude=["type"]), details=details) }} # Output Features There is currently no support for date as an output feature. Consider using the [`TEXT` type](text_features.md). --- ## File: docs/configuration/features/h3_features.md {% from './macros/includes.md' import render_fields, render_yaml %} {% set mv_details = "See [Missing Value Strategy](./input_features.md#missing-value-strategy) for details." %} {% set norm_details = "See [Normalization](../combiner.md#normalization) for details." %} {% set details = {"missing_value_strategy": mv_details, "norm": norm_details} %} H3 is a indexing system for representing geospatial data. For more details about it refer to . # Preprocessing Ludwig will parse the H3 64bit encoded format automatically. {% set preprocessing = get_feature_preprocessing_schema("h3") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} Preprocessing parameters can also be defined once and applied to all H3 input features using the [Type-Global Preprocessing](../defaults.md#type-global-preprocessing) section. # Input Features Input H3 features are transformed into a int valued tensors of size `N x 19` (where `N` is the size of the dataset and the 19 dimensions represent 4 H3 resolution parameters (4) - mode, edge, resolution, base cell - and 15 cell coordinate values. The encoder parameters specified at the feature level are: - **`tied`** (default `null`): name of another input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. Example H3 feature entry in the input features list: ```yaml name: h3_feature_name type: h3 tied: null encoder: type: embed ``` The available encoder parameters are: - **`type`** (default ``embed``): the possible values are `embed`, `weighted_sum`, and `rnn`. Encoder type and encoder parameters can also be defined once and applied to all H3 input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ## Encoders ### Embed Encoder This encoder encodes each component of the H3 representation (mode, edge, resolution, base cell and children cells) with embeddings. Children cells with value `0` will be masked out. After the embedding, all embeddings are summed and optionally passed through a stack of fully connected layers. {% set encoder_embed = get_encoder_schema("h3", "embed") %} {{ render_yaml(encoder_embed, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder_embed, exclude=["type"]), details=details) }} ### Weighted Sum Embed Encoder This encoder encodes each component of the H3 representation (mode, edge, resolution, base cell and children cells) with embeddings. Children cells with value `0` will be masked out. After the embedding, all embeddings are summed with a weighted sum (with learned weights) and optionally passed through a stack of fully connected layers. {% set encoder_weighted_sum = get_encoder_schema("h3", "weighted_sum") %} {{ render_yaml(encoder_weighted_sum, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder_weighted_sum, exclude=["type"]), details=details) }} ### RNN Encoder This encoder encodes each component of the H3 representation (mode, edge, resolution, base cell and children cells) with embeddings. Children cells with value `0` will be masked out. After the embedding, all embeddings are passed through an RNN encoder. The intuition behind this is that, starting from the base cell, the sequence of children cells can be seen as a sequence encoding the path in the tree of all H3 hexes. {% set encoder_rnn = get_encoder_schema("h3", "rnn") %} {{ render_yaml(encoder_rnn, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder_rnn, exclude=["type"]), details=details) }} # Output Features There is currently no support for H3 as an output feature. Consider using the [`TEXT` type](text_features.md). --- ## File: docs/configuration/features/image_features.md {% from './macros/includes.md' import render_fields, render_yaml %} Input image features are transformed into a float valued tensors of size `N x C x H x W` (where `N` is the size of the dataset, `C` is the number of channels, and `H x W` is the height and width of the image (can be specified by the user). These tensors are added to HDF5 with a key that reflects the name of column in the dataset. The column name is added to the JSON file, with an associated dictionary containing preprocessing information about the sizes of the resizing. # Supported Image Formats The number of channels in the image is determined by the image format. The following table lists the supported image formats and the number of channels. | Format | Number of channels | | -------------------- | ------------------ | | Grayscale | 1 | | Grayscale with Alpha | 2 | | RGB | 3 | | RGB with Alpha | 4 | # Preprocessing During preprocessing, raw image files are transformed into numpy arrays and saved in the hdf5 format. !!! note Images passed to an image encoder are expected to have the same size. If images are different sizes, by default they will be resized to the dimensions of the first image in the dataset. Optionally, a `resize_method` together with a target `width` and `height` can be specified in the feature preprocessing parameters, in which case all images will be resized to the specified target size. {% set preprocessing = get_feature_preprocessing_schema("image") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Parameters: {{ render_fields(schema_class_to_fields(preprocessing)) }} Preprocessing parameters can also be defined once and applied to all image input features using the [Type-Global Preprocessing](../defaults.md#type-global-preprocessing) section. ## Preprocessing Modes Ludwig supports three preprocessing modes for image features, controlled by the `mode` parameter: | Mode | Preprocessing memory | Training epoch 1 | Training epoch 2+ | Best for | |------|----------------------|------------------|---------------------|----------| | `eager` | High (O(N×tensor)) | Fast | Fast | Small datasets that fit in RAM | | `lazy` (default) | Low (O(batch)) | Slower (decode-bound) | Slower | Large datasets | | `lazy_cached` | Low (O(batch)) | Fast (GPU pipelined) | Very fast (memmap) | Large datasets, any GPU speed | ### `mode: lazy` (default) Ludwig stores file paths in the processed dataset and decodes images on-the-fly, one batch at a time, during training. Decoding runs in a `ThreadPoolExecutor` that overlaps with the GPU forward pass, matching the throughput of the eager decode path. ### `mode: lazy_cached` On the first training epoch, images are decoded per batch (same as `lazy`) and written to a numpy memmap alongside the Parquet cache. From epoch 2 onward, the memmap is read directly (~0.1 ms/batch), eliminating decode overhead entirely. ### `mode: eager` All images are decoded during preprocessing and stored as tensors in the Parquet cache. Use this only when the full decoded dataset fits comfortably in memory. ### Configuration Examples ```yaml input_features: - name: image type: image preprocessing: mode: lazy # default prefetch_size: null # auto (4 for lazy/lazy_cached, 0 for eager) lazy_cache_dir: null # default: ~/.cache/ludwig/lazy_media// height: 224 width: 224 num_channels: 3 resize_method: interpolate ``` ```yaml input_features: - name: image type: image preprocessing: mode: lazy_cached # decode+cache on epoch 1; memmap from epoch 2+ lazy_cache_dir: /fast/nvme/image_cache ``` ```yaml input_features: - name: image type: image preprocessing: mode: eager # decode everything upfront ``` !!! note Lazy preprocessing is automatically **disabled** when using a TorchVision pretrained encoder (e.g. `resnet`, `efficientnet`, `vit`). Those encoders apply their own normalization pipeline which requires images to be decoded upfront. See [Choosing a Preprocessing Mode](../../user_guide/datasets/data_preprocessing.md#choosing-a-preprocessing-mode) for a full comparison. ### Lazy Preprocessing with HuggingFace Datasets When loading a HuggingFace dataset, image columns are delivered as `PIL.Image.Image` objects — not file paths. Ludwig handles this transparently based on what the PIL Image carries: 1. **PIL Image opened from disk** — PIL sets a `.filename` attribute pointing to the source file. Ludwig detects this and reuses that path directly (no copy). 2. **In-memory PIL Image** (no `.filename`) — Ludwig saves the image as a PNG file in `lazy_cache_dir` and uses that path going forward. HuggingFace may also deliver images as dicts: ```python {"bytes": b"...", "path": "/path/to/cached.jpg"} # HF Image column format ``` Ludwig will reuse `"path"` if the file exists, otherwise decode `"bytes"` and save to cache. Raw `bytes` and `numpy.ndarray` inputs (both HWC and CHW channel orderings) are also supported. The cache is persistent and idempotent: subsequent runs with the same dataset skip the write step entirely. ### Controlling the Cache Directory `lazy_cache_dir` controls where PNG files are written for in-memory sources (HuggingFace datasets). The decoded memmap for `lazy_cached` mode is placed next to the Parquet cache, not inside `lazy_cache_dir`. ```yaml input_features: - name: photo type: image preprocessing: mode: lazy_cached lazy_cache_dir: /fast/nvme/my_project/image_cache ``` The per-feature subdirectory is created automatically. Multiple image features each get their own subdirectory named after the feature, even if they share the same `lazy_cache_dir`. # Input Features The encoder parameters specified at the feature level are: - `tied` (default `null`): name of another input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. - `augmentation` (default `False`): specifies image data augmentation operations to generate synthetic training data. More details on image augmentation can be found [here](#image-augmentation). Example image feature entry in the input features list: ```yaml name: image_column_name type: image tied: null encoder: type: stacked_cnn ``` The available encoder parameters are: - `type` (default `stacked_cnn`): the possible values are `stacked_cnn`, `resnet`, `mlp_mixer`, `vit`, `clip`, `dinov2`, `siglip`, `convnextv2`, and [TorchVision Pretrained Image Classification models](#torchvision-pretrained-model-encoders). Encoder type and encoder parameters can also be defined once and applied to all image input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ## Encoders ### Convolutional Stack Encoder (`stacked_cnn`) Stack of 2D convolutional layers with optional normalization, dropout, and down-sampling pooling layers, followed by an optional stack of fully connected layers. Convolutional Stack Encoder takes the following optional parameters: {% set image_encoder = get_encoder_schema("image", "stacked_cnn") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} ### MLP-Mixer Encoder Encodes images using MLP-Mixer, as described in [MLP-Mixer: An all-MLP Architecture for Vision](https://arxiv.org/abs/2105.01601). MLP-Mixer divides the image into equal-sized patches, applying fully connected layers to each patch to compute per-patch representations (tokens) and combining the representations with fully-connected mixer layers. The MLP-Mixer Encoder takes the following optional parameters: {% set image_encoder = get_encoder_schema("image", "mlp_mixer") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} ### TorchVision Pretrained Model Encoders Twenty TorchVision pretrained image classification models are available as Ludwig image encoders. The available models are: - `AlexNet` - `ConvNeXt` - `DenseNet` - `EfficientNet` - `EfficientNetV2` - `GoogLeNet` - `Inception V3` - `MaxVit` - `MNASNet` - `MobileNet V2` - `MobileNet V3` - `RegNet` - `ResNet` - `ResNeXt` - `ShuffleNet V2` - `SqueezeNet` - `SwinTransformer` - `VGG` - `VisionTransformer` - `Wide ResNet` See [TorchVison documentation](https://pytorch.org/vision/stable/models.html#classification) for more details. Ludwig encoders parameters for TorchVision pretrained models: #### AlexNet {% set image_encoder = get_encoder_schema("image", "alexnet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### ConvNeXt {% set image_encoder = get_encoder_schema("image", "convnext") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### DenseNet {% set image_encoder = get_encoder_schema("image", "densenet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### EfficientNet {% set image_encoder = get_encoder_schema("image", "efficientnet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### GoogLeNet {% set image_encoder = get_encoder_schema("image", "googlenet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### Inception V3 {% set image_encoder = get_encoder_schema("image", "inceptionv3") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### MaxVit {% set image_encoder = get_encoder_schema("image", "maxvit") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### MNASNet {% set image_encoder = get_encoder_schema("image", "mnasnet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### MobileNet V2 {% set image_encoder = get_encoder_schema("image", "mobilenetv2") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### MobileNet V3 {% set image_encoder = get_encoder_schema("image", "mobilenetv3") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### RegNet {% set image_encoder = get_encoder_schema("image", "regnet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### ResNet {% set image_encoder = get_encoder_schema("image", "resnet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### ResNeXt {% set image_encoder = get_encoder_schema("image", "resnext") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### ShuffleNet V2 {% set image_encoder = get_encoder_schema("image", "shufflenet_v2") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### SqueezeNet {% set image_encoder = get_encoder_schema("image", "squeezenet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### SwinTransformer {% set image_encoder = get_encoder_schema("image", "swin_transformer") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### VGG {% set image_encoder = get_encoder_schema("image", "vgg") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### VisionTransformer {% set image_encoder = get_encoder_schema("image", "vit") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### Wide ResNet {% set image_encoder = get_encoder_schema("image", "wide_resnet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} **Note**: - At this time Ludwig supports only the `DEFAULT` pretrained weights, which are the best available weights for a specific model. More details on `DEFAULT` weights can be found in this [blog post](https://pytorch.org/blog/introducing-torchvision-new-multi-weight-support-api/). - Some TorchVision pretrained models consume large amounts of memory. These `model_variant` required more than 12GB of memory: - `efficientnet_torch`: `b7` - `regnet_torch`: `y_128gf` - `vit_torch`: `h_14` ### U-Net Encoder The U-Net encoder is based on [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597). The encoder implements the contracting downsampling path of the U-Net stack. U-Net Encoder takes the following optional parameters: {% set image_encoder = get_encoder_schema("image", "unet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} ### CLIP Encoder The CLIP image encoder (Radford et al., "Learning Transferable Visual Models From Natural Language Supervision", ICML 2021) encodes images using CLIP's vision transformer. The resulting embeddings are aligned with text in a shared latent space, enabling zero-shot classification and multimodal tasks. Use CLIP when you need visual features that are semantically aligned with text -- for example, when combining image and text inputs for multimodal classification, or when you want zero-shot image classification without task-specific fine-tuning. Default pretrained model: `openai/clip-vit-base-patch32` {% set clip_encoder = get_encoder_schema("image", "clip") %} {{ render_yaml(clip_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(clip_encoder, exclude=["type"])) }} ### DINOv2 Encoder The DINOv2 encoder (Oquab et al., "DINOv2: Learning Robust Visual Features without Supervision", TMLR 2024) produces self-supervised visual features that work well as frozen backbones. Unlike CLIP, DINOv2 does not require text alignment -- it learns visual features purely from images using self-distillation. Use DINOv2 when you want a general-purpose frozen feature extractor, especially for dense prediction tasks (segmentation, depth estimation) or when you want to avoid fine-tuning the vision backbone. Default pretrained model: `facebook/dinov2-base` {% set dinov2_encoder = get_encoder_schema("image", "dinov2") %} {{ render_yaml(dinov2_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(dinov2_encoder, exclude=["type"])) }} ### SigLIP Encoder The SigLIP encoder (Zhai et al., "Sigmoid Loss for Language Image Pre-Training", ICCV 2023) uses sigmoid loss instead of softmax for image-text pre-training. This enables better scaling to larger batch sizes and more efficient training compared to CLIP, while maintaining similar zero-shot capabilities. Default pretrained model: `google/siglip-base-patch16-224` {% set siglip_encoder = get_encoder_schema("image", "siglip") %} {{ render_yaml(siglip_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(siglip_encoder, exclude=["type"])) }} ### ConvNeXt V2 Encoder The ConvNeXt V2 encoder (Woo et al., "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders", CVPR 2023) improves on ConvNeXt with Global Response Normalization (GRN) and fully convolutional masked autoencoder (FCMAE) pre-training. It is a pure-CNN architecture that matches or exceeds vision transformers on ImageNet. Available via TIMM with model variants from atto (3.7M params) to huge (660M params). {% set convnextv2_encoder = get_encoder_schema("image", "convnextv2") %} {{ render_yaml(convnextv2_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(convnextv2_encoder, exclude=["type"])) }} ### Generic TIMM Encoder The `timm` encoder exposes the full [pytorch-image-models](https://github.com/huggingface/pytorch-image-models) library as a single configurable encoder. Over 1,000 pretrained vision models are available, including all MetaFormer variants, EfficientFormer V2, DaViT, FastViT, and many more. Install TIMM before using this encoder: ```sh pip install timm ``` ```yaml encoder: type: timm model_name: caformer_s18.sail_in22k_ft_in1k use_pretrained: true trainable: true ``` Browse available model names at [timm.fast.ai](https://timm.fast.ai/). {% set timm_encoder = get_encoder_schema("image", "timm") %} {{ render_yaml(timm_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(timm_encoder, exclude=["type"])) }} ### CAFormer Encoder CAFormer (Yu et al., "MetaFormer Baselines for Vision", TPAMI 2024) is a **hybrid MetaFormer** that uses depthwise separable convolutions in the lower stages and self-attention in the upper stages. It achieves state-of-the-art accuracy/efficiency trade-offs on ImageNet. | Variant | Params | ImageNet top-1 | |---------|--------|----------------| | `caformer_s18` | 26 M | 83.6 % | | `caformer_s36` | 39 M | 84.5 % | | `caformer_m36` | 56 M | 85.2 % | | `caformer_b36` | 99 M | 85.5 % | ```yaml encoder: type: caformer model_name: caformer_s18.sail_in22k_ft_in1k use_pretrained: true ``` {% set caformer_encoder = get_encoder_schema("image", "caformer") %} {{ render_yaml(caformer_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(caformer_encoder, exclude=["type"])) }} ### ConvFormer Encoder ConvFormer replaces the attention token-mixer with a large-kernel depthwise convolution, making it a **pure-CNN MetaFormer** that outperforms ConvNeXt while being fully convolutional (no positional embeddings, any resolution input). | Variant | Params | ImageNet top-1 | |---------|--------|----------------| | `convformer_s18` | 27 M | 83.0 % | | `convformer_s36` | 40 M | 84.1 % | | `convformer_m36` | 57 M | 84.5 % | | `convformer_b36` | 100 M | 84.8 % | ```yaml encoder: type: convformer model_name: convformer_s18.sail_in22k_ft_in1k use_pretrained: true ``` {% set convformer_encoder = get_encoder_schema("image", "convformer") %} {{ render_yaml(convformer_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(convformer_encoder, exclude=["type"])) }} ### PoolFormer Encoder PoolFormer uses simple **average pooling** as the token mixer — proving that the MetaFormer architecture itself (not the specific mixer) is responsible for the strong performance of modern vision transformers. PoolFormerV2 adds grouped-normalization and extra depth to further close the gap with attention-based models. | Variant | Params | ImageNet top-1 | |---------|--------|----------------| | `poolformerv2_s12` | 12 M | 80.3 % | | `poolformerv2_s24` | 21 M | 82.0 % | | `poolformerv2_s36` | 31 M | 82.7 % | | `poolformerv2_m36` | 56 M | 83.5 % | | `poolformerv2_m48` | 73 M | 83.8 % | ```yaml encoder: type: poolformer model_name: poolformerv2_s12 use_pretrained: true ``` {% set poolformer_encoder = get_encoder_schema("image", "poolformer") %} {{ render_yaml(poolformer_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(poolformer_encoder, exclude=["type"])) }} ### Deprecated Encoders (planned to remove in v0.8) #### Legacy ResNet Encoder DEPRECATED: This encoder is deprecated and will be removed in a future release. Please use the equivalent TorchVision [ResNet](#resnet) encoder instead. Implements ResNet V2 as described in [Identity Mappings in Deep Residual Networks](https://arxiv.org/abs/1603.05027). The ResNet encoder takes the following optional parameters: {% set image_encoder = get_encoder_schema("image", "resnet") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} #### Legacy Vision Transformer Encoder DEPRECATED: This encoder is deprecated and will be removed in a future release. Please use the equivalent TorchVision [VisionTransformer](#visiontransformer) encoder instead. Encodes images using a Vision Transformer as described in [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929). Vision Transformer divides the image into equal-sized patches, uses a linear transformation to encode each flattened patch, then applies a deep transformer architecture to the sequence of encoded patches. The Vision Transformer Encoder takes the following optional parameters: {% set image_encoder = get_encoder_schema("image", "vit") %} {{ render_yaml(image_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(image_encoder, exclude=["type"])) }} ## Image Augmentation Image augmentation is a technique used to increase the diversity of a training dataset by applying random transformations to the images. The goal is to train a model that is robust to the variations in the training data. Augmentation is specified by the `augmentation` section in the image feature configuration and can be specified in one of the following ways: **Boolean: `False` (Default)** No augmentation is applied to the images. ```yaml augmentation: False ``` **Boolean: `True`** The following augmentation methods are applied to the images: `random_horizontal_flip` and `random_rotate`. ```yaml augmentation: True ``` **List of Augmentation Methods** One or more of the following augmentation methods are applied to the images in the order specified by the user: `random_horizontal_flip`, `random_vertical_flip`, `random_rotate`, `random_blur`, `random_brightness`, and `random_contrast`. The following is an illustrative example. ```yaml augmentation: - type: random_horizontal_flip - type: random_vertical_flip - type: random_rotate degree: 10 - type: random_blur kernel_size: 3 - type: random_brightness min: 0.5 max: 2.0 - type: random_contrast min: 0.5 max: 2.0 ``` Augmentation is applied to the batch of images in the training set only. The validation and test sets are not augmented. Following illustrates how augmentation affects an image: **Horizontal Flip**: Image is randomly flipped horizontally. {% set image_augmentation = get_augmentation_schema("image", "random_horizontal_flip") %} {{ render_yaml(image_augmentation) }} **Vertical Flip**: Image is randomly flipped vertically. {% set image_augmentation = get_augmentation_schema("image", "random_vertical_flip") %} {{ render_yaml(image_augmentation) }} **Rotate**: Image is randomly rotated by an amount in the range [-degree, +degree]. `degree` must be a positive integer. {% set image_augmentation = get_augmentation_schema("image", "random_rotate") %} {{ render_yaml(image_augmentation) }} Parameters: {{ render_fields(schema_class_to_fields(image_augmentation, exclude="type")) }} Following shows the effect of rotating an image: **Blur**: Image is randomly blurred using a Gaussian filter with kernel size specified by the user. The `kernel_size` must be a positive, odd integer. {% set image_augmentation = get_augmentation_schema("image", "random_blur") %} {{ render_yaml(image_augmentation) }} Parameters: {{ render_fields(schema_class_to_fields(image_augmentation, exclude="type")) }} Following shows the effect of blurring an image with various kernel sizes: **Adjust Brightness**: Image brightness is adjusted by a factor randomly selected in the range [min, max]. Both `min` and `max` must be a float greater than 0, with `min` less than `max`. {% set image_augmentation = get_augmentation_schema("image", "random_brightness") %} {{ render_yaml(image_augmentation) }} Parameters: {{ render_fields(schema_class_to_fields(image_augmentation, exclude="type")) }} Following shows the effect of brightness adjustment with various factors: **Adjust Contrast**: Image contrast is adjusted by a factor randomly selected in the range [min, max]. Both `min` and `max` must be a float greater than 0, with `min` less than `max`. {% set image_augmentation = get_augmentation_schema("image", "random_contrast") %} {{ render_yaml(image_augmentation) }} Parameters: {{ render_fields(schema_class_to_fields(image_augmentation, exclude="type")) }} Following shows the effect of contrast adjustment with various factors: **Illustrative Examples of Image Feature Configuration with Augmentation** ```yaml name: image_column_name type: image encoder: type: resnet model_variant: 18 use_pretrained: true pretrained_cache_dir: None trainable: true augmentation: false ``` ```yaml name: image_column_name type: image encoder: type: stacked_cnn augmentation: true ``` ```yaml name: image_column_name type: image encoder: type: alexnet augmentation: - type: random_horizontal_flip - type: random_rotate degree: 10 - type: random_blur kernel_size: 3 - type: random_brightness min: 0.5 max: 2.0 - type: random_contrast min: 0.5 max: 2.0 - type: random_vertical_flip ``` # Output Features Image features can be used when semantic segmentation needs to be performed. Ludwig 0.15 exposes three segmentation decoders: `unet`, `segformer`, and `fpn`. Example image output feature using default parameters: ```yaml name: image_column_name type: image reduce_input: sum dependencies: [] reduce_dependencies: sum loss: type: softmax_cross_entropy decoder: type: unet ``` Parameters: - **`reduce_input`** (default `sum`): defines how to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`dependencies`** (default `[]`): the output features this one is dependent on. For a detailed explanation refer to [Output Feature Dependencies](output_features.md#output-feature-dependencies). - **`reduce_dependencies`** (default `sum`): defines how to reduce the output of a dependent feature that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`loss`** (default `{type: softmax_cross_entropy}`): is a dictionary containing a loss `type`. `softmax_cross_entropy` is the only supported loss type for image output features. See [Loss](#loss) for details. - **`decoder`** (default: `{"type": "unet"}`): Decoder for the desired task. Options: `unet`, `segformer`, `fpn`. See [Decoder](#decoders) for details. ## Decoders ### U-Net Decoder The U-Net decoder is based on [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597). The decoder implements the expansive upsampling path of the U-Net stack. Semantic segmentation supports one input and one output feature. The `num_fc_layers` in the decoder and combiner sections must be set to 0 as U-Net does not have any fully connected layers. U-Net Decoder takes the following optional parameters: {% set decoder = get_decoder_schema("image", "unet") %} {{ render_yaml(decoder, parent="decoder") }} Parameters: {{ render_fields(schema_class_to_fields(decoder, exclude=["type"]), details=details) }} The decoder depth is configurable via `num_stages` (default `4`). Each stage doubles the spatial resolution during upsampling, so the combiner output height and width must be divisible by `2 ** num_stages`. Deeper stacks capture longer-range spatial context at the cost of more parameters. ### SegFormer Decoder `segformer` implements the lightweight all-MLP decoder from [Xie et al., NeurIPS 2021](https://arxiv.org/abs/2105.15203). Instead of transposed convolutions it projects all encoder stages into a shared hidden size and fuses them with a single MLP — much cheaper than U-Net while remaining competitive on standard segmentation benchmarks when paired with a pretrained hierarchical encoder such as Swin or ConvNeXt V2. ```yaml output_features: - name: mask type: image decoder: type: segformer hidden_size: 256 dropout: 0.1 num_classes: 21 ``` {% set decoder = get_decoder_schema("image", "segformer") %} {{ render_yaml(decoder, parent="decoder") }} Parameters: {{ render_fields(schema_class_to_fields(decoder, exclude=["type"]), details=details) }} ### FPN Decoder `fpn` implements the Feature Pyramid Network decoder from [Lin et al., CVPR 2017](https://arxiv.org/abs/1612.03144). It builds a top-down pyramid with lateral connections across multiple encoder stages, producing multi-scale features that are useful for dense prediction tasks and for segmenting objects at very different scales. ```yaml output_features: - name: mask type: image decoder: type: fpn num_channels: 256 num_levels: 4 num_classes: 80 ``` {% set decoder = get_decoder_schema("image", "fpn") %} {{ render_yaml(decoder, parent="decoder") }} Parameters: {{ render_fields(schema_class_to_fields(decoder, exclude=["type"]), details=details) }} Decoder type and decoder parameters can also be defined once and applied to all image output features using the [Type-Global Decoder](../defaults.md#type-global-decoder) section. ## Loss ### Softmax Cross Entropy {% set loss = get_loss_schema("softmax_cross_entropy") %} {{ render_yaml(loss, parent="loss") }} Parameters: {{ render_fields(schema_class_to_fields(loss, exclude=["type"]), details=details) }} Loss and loss related parameters can also be defined once and applied to all image output features using the [Type-Global Loss](../defaults.md#type-global-loss) section. ## Metrics The measures that are calculated every epoch and are available for image features are the `accuracy` and `loss`. You can set either of them as `validation_metric` in the `training` section of the configuration if you set the `validation_field` to be the name of a category feature. --- ## File: docs/configuration/features/input_features.md The `input_features` section is list of feature definitions. Each feature definition contains two required fields: `name` and `type`. === "YAML" ```yaml input_features: - name: Pclass type: category ``` === "Python Dict" ```python {"input_features": [{"name": "Pclass", "type": "category"}]} ``` `name` is the name of the feature in the dataset. `type` is one of the [supported data types](supported_data_types.md). # Preprocessing Recall Ludwig's butterfly framework. Each input feature can specify its own preprocessing via the `preprocessing` subsection. === "YAML" ```yaml input_features: - name: Fare type: number preprocessing: missing_value_strategy: fill_with_mean ``` === "Python Dict" ```python {"input_features": [{"name": "Fare", "type": "number", "preprocessing": {"missing_value_strategy": "fill_with_mean"}}]} ``` It's also possible to specify preprocessing rules for all features of a certain type. See [Type-Global Preprocessing](../defaults.md#type-global-preprocessing). ## Common Parameters ### Missing Value Strategy Ludwig allows any input feature to be "missing" in the dataset (at both training and prediction time). How missing values are handled by default varies depending on the feature type, but this handling strategy can be configured in `preprocessing` section of the config for each feature: ```yaml preprocessing: missing_value_strategy: fill_with_mean ``` Options: - **`fill_with_const`**: Replaces the missing value with a specific value specified with the `fill_value` parameter. - **`fill_with_mode`**: Replaces the missing values with the most frequent value in the column. - **`fill_with_mean`**: Replaces the missing values with the mean of the values in the column (`number` features only). - **`fill_with_false`** Replace the missing values with the `false` value in the column (`binary` features only). - **`bfill`**: Replaces the missing values with the next valid value from the subsequent rows of the input dataset. - **`ffill`**: Replaces the missing values with the previous valid value from the preceding rows of the input dataset. - **`drop_row`**: Removes the entire row from the dataset if this column is missing. For output features, the default strategy is always `drop_row`, as otherwise Ludwig would be forced to "make up" the ground truth values being predicted. However, this can also be overridden using the same `missing_value_stragegy` param if so desired. # Encoders Each input feature can configure a specific `encoder` to map input feature values into tensors. For instance, a user might want to encode a `sequence` feature using a `transformer` or an `image` feature using a `stacked_cnn`. Different data types support different encoders. Check the documentation for specific feature types to see what encoders are supported for that type. All the other parameters besides `name`, `type`, and `preprocessing`, will be passed as parameters to the encoder subsection. Note that each encoder can have different parameters, so extensive documentation for each of the encoders that can be used for a certain data type can be found in each data type's documentation. Here is an example of how to specify a specific encoder config for an input feature: === "YAML" ```yaml input_features: - name: text type: text preprocessing: tokenizer: space encoder: type: bert reduce_output: null trainable: true ``` === "Python Dict" ```python { "input_features": [ { "name": "text", "type": "text", "level": "word", "preprocessing": {"word_tokenizer": "space"}, "encoder": { "type": "bert", "reduce_output": None, "trainable": True, }, } ] } ``` Encoders map raw feature values into tensors. These are usually vectors in the case of data types without a temporal / sequential aspect, matrices for when there is a temporal / sequential aspect, or higher rank tensors for when there is a spatial or a spatiotemporal aspect to the input data. Different configurations of the same encoder may return a tensor with different rank, for instance a sequential encoder may return a vector of size `h` that is either the final vector of a sequence or the result of pooling over the sequence length, or it can return a matrix of size `l x h` where `l` is the length of the sequence and `h` is the hidden dimension if you specify the pooling reduce operation (`reduce_output`) to be `None`. For the sake of simplicity you can imagine the output to be a vector in most of the cases, but there is a `reduce_output` parameter one can specify to change the default behavior. For first-time users, we recommend starting with the defaults. # Tying encoder weights An additional feature that Ludwig provides is the option to have **tied weights** between different encoders. For instance, if my model takes two sentences as input and return the probability of their entailment, I may want to encode both sentences with the same encoder. This is done by specifying the `tied` parameter of one feature to be the name of another output feature. For example: === "YAML" ```yaml input_features: - name: sentence1 type: text - name: sentence2 type: text tied: sentence1 ``` === "Python Dict" ```python {"input_features": [{"name": "sentence1", "type": "text"}, {"name": "sentence2", "type": "text", "tied": "sentence1"}]} ``` Specifying a name of a non-existent input feature will result in an error. Also, in order to be able to have tied weights, all encoder parameters have to be identical between the two input features. It's also possible to specify encoder type and encoder related parameters for all features of a certain type. See [Type-Global Encoder](../defaults.md#type-global-encoder). --- ## File: docs/configuration/features/number_features.md {% from './macros/includes.md' import render_fields, render_yaml %} {% set mv_details = "See [Missing Value Strategy](./input_features.md#missing-value-strategy) for details." %} {% set nz_details = "See [Normalization](#normalization) for details." %} {% set norm_details = "See [Normalization](../combiner.md#normalization) for details." %} {% set details = {"missing_value_strategy": mv_details, "normalization": nz_details, "norm": norm_details, "fc_norm": norm_details} %} # Preprocessing Number features are directly transformed into a float valued vector of length `n` (where `n` is the size of the dataset) and added to the HDF5 with a key that reflects the name of column in the dataset. No additional information about them is available in the JSON metadata file. {% set preprocessing = get_feature_preprocessing_schema("number") %} {{ render_yaml(preprocessing, parent="preprocessing") }} Parameters: {{ render_fields(schema_class_to_fields(preprocessing), details=details) }} Preprocessing parameters can also be defined once and applied to all number input features using the [Type-Global Preprocessing](../defaults.md#type-global-preprocessing) section. ## Normalization Technique to be used when normalizing the number feature types. Options: - **`null`**: No normalization is performed. - **`zscore`**: The mean and standard deviation are computed so that values are shifted to have zero mean and 1 standard deviation. - **`minmax`**: The minimum is subtracted from values and the result is divided by difference between maximum and minimum. - **`log1p`**: The value returned is the natural log of 1 plus the original value. Note: `log1p` is defined only for positive values. - **`iq`**: The median is subtracted from values and the result is divided by the interquartile range (IQR), i.e., the 75th percentile value minus the 25th percentile value. The resulting data has a zero mean and median and a standard deviation of 1. This is useful if your feature has large outliers since the normalization won't be skewed by those values. The best normalization techniqe to use depends on the distribution of your data, but `zscore` is a good place to start in many cases. # Input Features Number features have four encoders. The `passthrough` encoder simply returns the raw numerical values. The `dense` encoder passes values through fully connected layers. The `ple` encoder (Piecewise Linear Encoding) computes quantile-based bin edges from training data and produces a piecewise-linear interpolation vector, which is the most impactful improvement for tabular deep learning accuracy (Gorishniy et al., NeurIPS 2022). The `periodic` encoder uses learned sinusoidal features for smooth numerical representations. The encoder parameters specified at the feature level are: - **`tied`** (default `null`): name of the input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters. Example number feature entry in the input features list: ```yaml name: number_column_name type: number tied: null encoder: type: dense ``` The available encoder parameters: - **`type`** (default `passthrough`): the possible values are `passthrough`, `dense`, `ple`, `periodic`, and `bins`. `passthrough` outputs the raw values unaltered. `dense` passes through fully connected layers. `ple` uses Piecewise Linear Encoding with quantile bin edges. `periodic` uses learned sinusoidal features. `bins` discretizes values into bins and produces an embedding. Encoder type and encoder parameters can also be defined once and applied to all number input features using the [Type-Global Encoder](../defaults.md#type-global-encoder) section. ## Encoders ### Passthrough Encoder {% set encoder = get_encoder_schema("number", "passthrough") %} {{ render_yaml(encoder, parent="encoder") }} There are no additional parameters for `passthrough` encoder. ### Dense Encoder {% set encoder = get_encoder_schema("number", "dense") %} {{ render_yaml(encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(encoder, exclude=["type"]), details=details) }} ### PLE Encoder Piecewise Linear Encoding computes quantile-based bin edges from training data, then for each input value produces a vector where each element is a piecewise-linear interpolation within that bin. A learned linear projection maps this to the output embedding space. Based on [Gorishniy et al., NeurIPS 2022](https://arxiv.org/abs/2203.05556). {% set ple_encoder = get_encoder_schema("number", "ple") %} {{ render_yaml(ple_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(ple_encoder, exclude=["type"]), details=details) }} ### Periodic Encoder Uses learned sinusoidal features: `sin(2*pi*f*x + phi)` where `f` and `phi` are learnable per-frequency parameters. A linear projection maps the periodic features to the output embedding space. Based on [Gorishniy et al., NeurIPS 2022](https://arxiv.org/abs/2203.05556). {% set periodic_encoder = get_encoder_schema("number", "periodic") %} {{ render_yaml(periodic_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(periodic_encoder, exclude=["type"]), details=details) }} ### Bins Encoder The `bins` encoder discretizes numerical values into fixed bins and maps each bin to a learned embedding. This is a simple approach to numerical feature encoding that converts continuous values into categorical-like representations. It works well as a fast baseline for tabular data. {% set bins_encoder = get_encoder_schema("number", "bins") %} {{ render_yaml(bins_encoder, parent="encoder") }} Parameters: {{ render_fields(schema_class_to_fields(bins_encoder, exclude=["type"]), details=details) }} # Output Features Number features can be used when a regression needs to be performed. There is only one decoder available for number features: a (potentially empty) stack of fully connected layers, followed by a projection to a single number. Example number output feature using default parameters: ```yaml name: number_column_name type: number reduce_input: sum dependencies: [] reduce_dependencies: sum loss: type: mean_squared_error decoder: type: regressor ``` Parameters: - **`reduce_input`** (default `sum`): defines how to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`dependencies`** (default `[]`): the output features this one is dependent on. For a detailed explanation refer to [Output Feature Dependencies](output_features.md#output-feature-dependencies). - **`reduce_dependencies`** (default `sum`): defines how to reduce the output of a dependent feature that is not a vector, but a matrix or a higher order tensor, on the first dimension (second if you count the batch dimension). Available values are: `sum`, `mean` or `avg`, `max`, `concat` (concatenates along the first dimension), `last` (returns the last vector of the first dimension). - **`loss`** (default `{type: mean_squared_error}`): is a dictionary containing a loss `type`. Options: `mean_squared_error`, `mean_absolute_error`, `root_mean_squared_error`, `root_mean_squared_percentage_error`. See [Loss](#loss) for details. - **`decoder`** (default: `{"type": "regressor"}`): Decoder for the desired task. Options: `regressor`. See [Decoder](#decoders) for details. ## Decoders ### Regressor {% set decoder = get_decoder_schema("number", "regressor") %} {{ render_yaml(decoder, parent="decoder") }} Parameters: {{ render_fields(schema_class_to_fields(decoder, exclude=["type"]), details=details) }} Decoder type and decoder parameters can also be defined once and applied to all number output features using the [Type-Global Decoder](../defaults.md#type-global-decoder) section. ## Loss {% set loss_classes = get_loss_schemas("number") %} {% for loss in loss_classes %} ### {{ loss.name() }} {{ render_yaml(loss, parent="loss") }} Parameters: {{ render_fields(schema_class_to_fields(loss, exclude=["type"]), details=details) }} {% endfor %} Loss and loss related parameters can also be defined once and applied to all number output features using the [Type-Global Loss](../defaults.md#type-global-loss) section. ## Metrics The metrics that are calculated every epoch and are available for number features are `mean_squared_error`, `mean_absolute_error`, `root_mean_squared_error`, `root_mean_squared_percentage_error` and the `loss` itself. You can set either of them as `validation_metric` in the `training` section of the configuration if you set the `validation_field` to be the name of a number feature.