aim Documentation

Technical Documentation: aimhubio/aim

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

[**About**](#-about) • [**Demos**](#-demos) • [**Ecosystem**](#-ecosystem) • [**Quick Start**](#-quick-start) • [**Examples**](https://github.com/aimhubio/aim/tree/main/examples) • [**Documentation**](https://aimstack.readthedocs.io/en/latest/) • [**Community**](#-community) • [**Blog**](https://aimstack.io/blog)

ℹ️ About

Aim is an open-source, self-hosted ML experiment tracking tool designed to handle 10,000s of training runs.

Aim provides a performant and beautiful UI for exploring and comparing training runs.
Additionally, its SDK enables programmatic access to tracked metadata — perfect for automations and Jupyter Notebook analysis.

🎬 Demos

Check out live Aim demos NOW to see it in action.

🌍 Ecosystem

Aim is not just an experiment tracker. It's a groundwork for an ecosystem.
Check out the two most famous Aim-based tools.

🏁 Quick start

Follow the steps below to get started with Aim.

1. Install Aim on your training environment Copy Chapter

pip3 install aim

2. Integrate Aim with your code Copy Chapter

from aim import Run

# Initialize a new run
run = Run()

# Log run parameters
run["hparams"] = {
    "learning_rate": 0.001,
    "batch_size": 32,
}

# Log metrics
for i in range(10):
    run.track(i, name='loss', step=i, context={ "subset":"train" })
    run.track(i, name='acc', step=i, context={ "subset":"train" })

See the full list of supported trackable objects(e.g. images, text, etc) here.

3. Run the training as usual and start Aim UI Copy Chapter

aim up

Learn more Copy Chapter

Migrate from other tools

Aim has built-in converters to easily migrate logs from other tools.
These migrations cover the most common usage scenarios.
In case of custom and complex scenarios you can use Aim SDK to implement your own conversion script.

• TensorBoard logs converter
• MLFlow logs converter
• Weights & Biases logs converter

Integrate Aim into an existing project

Aim easily integrates with a wide range of ML frameworks, providing built-in callbacks for most of them.

• Integration with Pytorch Ignite
• Integration with Pytorch Lightning
• Integration with Hugging Face
• Integration with Keras & tf.Keras
• Integration with Keras Tuner
• Integration with XGboost
• Integration with CatBoost
• Integration with LightGBM
• Integration with fastai
• Integration with MXNet
• Integration with Optuna
• Integration with PaddlePaddle
• Integration with Stable-Baselines3
• Integration with Acme
• Integration with Prophet

Query runs programmatically via SDK

Aim Python SDK empowers you to query and access any piece of tracked metadata with ease.

from aim import Repo

my_repo = Repo('/path/to/aim/repo')

query = "metric.name == 'loss'" # Example query

# Get collection of metrics
for run_metrics_collection in my_repo.query_metrics(query).iter_runs():
    for metric in run_metrics_collection:
        # Get run params
        params = metric.run[...]
        # Get metric values
        steps, metric_values = metric.values.sparse_numpy()

Set up a centralized tracking server

Aim remote tracking server allows running experiments in a multi-host environment and collect tracked data in a centralized location.

See the docs on how to set up the remote server.

Deploy Aim on kubernetes

• The official Aim docker image: https://hub.docker.com/r/aimstack/aim
• A guide on how to deploy Aim on kubernetes: https://aimstack.readthedocs.io/en/latest/using/k8s_deployment.html

Read the full documentation on aimstack.readthedocs.io 📖

🆚 Comparisons to familiar tools

TensorBoard vs Aim

Training run comparison

Order of magnitude faster training run comparison with Aim

• The tracked params are first class citizens at Aim. You can search, group, aggregate via params - deeply explore all the tracked data (metrics, params, images) on the UI.
• With tensorboard the users are forced to record those parameters in the training run name to be able to search and compare. This causes a super-tedius comparison experience and usability issues on the UI when there are many experiments and params. TensorBoard doesn't have features to group, aggregate the metrics

Scalability

• Aim is built to handle 1000s of training runs - both on the backend and on the UI.
• TensorBoard becomes really slow and hard to use when a few hundred training runs are queried / compared.

Beloved TB visualizations to be added on Aim

• Embedding projector.

• Neural network visualization.

  MLflow vs Aim

MLFlow is an end-to-end ML Lifecycle tool.
Aim is focused on training tracking.
The main differences of Aim and MLflow are around the UI scalability and run comparison features.

Aim and MLflow are a perfect match - check out the aimlflow - the tool that enables Aim superpowers on Mlflow.

Run comparison

• Aim treats tracked parameters as first-class citizens. Users can query runs, metrics, images and filter using the params.
• MLFlow does have a search by tracked config, but there are no grouping, aggregation, subplotting by hyparparams and other comparison features available.

UI Scalability

• Aim UI can handle several thousands of metrics at the same time smoothly with 1000s of steps. It may get shaky when you explore 1000s of metrics with 10000s of steps each. But we are constantly optimizing!

• MLflow UI becomes slow to use when there are a few hundreds of runs.

  Weights and Biases vs Aim

Hosted vs self-hosted

• Weights and Biases is a hosted closed-source MLOps platform.
• Aim is self-hosted, free and open-source experiment tracking tool.

🛣️ Roadmap

Detailed milestones Copy Chapter

The Aim product roadmap :sparkle:

• The Backlog contains the issues we are going to choose from and prioritize weekly
• The issues are mainly prioritized by the highly-requested features

High-level roadmap Copy Chapter

The high-level features we are going to work on the next few months:

In progress

• Aim SDK low-level interface

• Dashboards – customizable layouts with embedded explorers

• Ergonomic UI kit

• Text Explorer

  Next-up

Aim UI

• Runs management
  • Runs explorer – query and visualize runs data(images, audio, distributions, ...) in a central dashboard
• Explorers
  • Distributions Explorer

SDK and Storage

• Scalability
  • Smooth UI and SDK experience with over 10.000 runs
• Runs management
  • CLI commands
    • Reporting - runs summary and run details in a CLI compatible format
    • Manipulations – copy, move, delete runs, params and sequences
• Cloud storage support – store runs blob(e.g. images) data on the cloud
• Artifact storage – store files, model checkpoints, and beyond

Integrations

• ML Frameworks:

  • Shortlist: scikit-learn

• Resource management tools

  • Shortlist: Kubeflow, Slurm

• Workflow orchestration tools

  Done

  • Live updates (Shipped: Oct 18 2021)
  • Images tracking and visualization (Start: Oct 18 2021, Shipped: Nov 19 2021)
  • Distributions tracking and visualization (Start: Nov 10 2021, Shipped: Dec 3 2021)
  • Jupyter integration (Start: Nov 18 2021, Shipped: Dec 3 2021)
  • Audio tracking and visualization (Start: Dec 6 2021, Shipped: Dec 17 2021)
  • Transcripts tracking and visualization (Start: Dec 6 2021, Shipped: Dec 17 2021)
  • Plotly integration (Start: Dec 1 2021, Shipped: Dec 17 2021)
  • Colab integration (Start: Nov 18 2021, Shipped: Dec 17 2021)
  • Centralized tracking server (Start: Oct 18 2021, Shipped: Jan 22 2022)
  • Tensorboard adaptor - visualize TensorBoard logs with Aim (Start: Dec 17 2021, Shipped: Feb 3 2022)
  • Track git info, env vars, CLI arguments, dependencies (Start: Jan 17 2022, Shipped: Feb 3 2022)
  • MLFlow adaptor (visualize MLflow logs with Aim) (Start: Feb 14 2022, Shipped: Feb 22 2022)
  • Activeloop Hub integration (Start: Feb 14 2022, Shipped: Feb 22 2022)
  • PyTorch-Ignite integration (Start: Feb 14 2022, Shipped: Feb 22 2022)
  • Run summary and overview info(system params, CLI args, git info, ...) (Start: Feb 14 2022, Shipped: Mar 9 2022)
  • Add DVC related metadata into aim run (Start: Mar 7 2022, Shipped: Mar 26 2022)
  • Ability to attach notes to Run from UI (Start: Mar 7 2022, Shipped: Apr 29 2022)
  • Fairseq integration (Start: Mar 27 2022, Shipped: Mar 29 2022)
  • LightGBM integration (Start: Apr 14 2022, Shipped: May 17 2022)
  • CatBoost integration (Start: Apr 20 2022, Shipped: May 17 2022)
  • Run execution details(display stdout/stderr logs) (Start: Apr 25 2022, Shipped: May 17 2022)
  • Long sequences(up to 5M of steps) support (Start: Apr 25 2022, Shipped: Jun 22 2022)
  • Figures Explorer (Start: Mar 1 2022, Shipped: Aug 21 2022)
  • Notify on stuck runs (Start: Jul 22 2022, Shipped: Aug 21 2022)
  • Integration with KerasTuner (Start: Aug 10 2022, Shipped: Aug 21 2022)
  • Integration with WandB (Start: Aug 15 2022, Shipped: Aug 21 2022)
  • Stable remote tracking server (Start: Jun 15 2022, Shipped: Aug 21 2022)
  • Integration with fast.ai (Start: Aug 22 2022, Shipped: Oct 6 2022)
  • Integration with MXNet (Start: Sep 20 2022, Shipped: Oct 6 2022)
  • Project overview page (Start: Sep 1 2022, Shipped: Oct 6 2022)
  • Remote tracking server scaling (Start: Sep 11 2022, Shipped: Nov 26 2022)
  • Integration with PaddlePaddle (Start: Oct 2 2022, Shipped: Nov 26 2022)
  • Integration with Optuna (Start: Oct 2 2022, Shipped: Nov 26 2022)
  • Audios Explorer (Start: Oct 30 2022, Shipped: Nov 26 2022)
  • Experiment page (Start: Nov 9 2022, Shipped: Nov 26 2022)
  • HuggingFace datasets (Start: Dec 29 2022, Feb 3 2023)

👥 Community

Aim README badge Copy Chapter

Add Aim badge to your README, if you've enjoyed using Aim in your work:

[](https://github.com/aimhubio/aim)

Cite Aim in your papers Copy Chapter

In case you've found Aim helpful in your research journey, we'd be thrilled if you could acknowledge Aim's contribution:

@software{Arakelyan_Aim_2020,
  author = {Arakelyan, Gor and Soghomonyan, Gevorg and {The Aim team}},
  doi = {10.5281/zenodo.6536395},
  license = {Apache-2.0},
  month = {6},
  title = {{Aim}},
  url = {https://github.com/aimhubio/aim},
  version = {3.9.3},
  year = {2020}
}

Contributing to Aim Copy Chapter

Considering contibuting to Aim?
To get started, please take a moment to read the CONTRIBUTING.md guide.

Join Aim contributors by submitting your first pull request. Happy coding! 😊

Made with contrib.rocks.

More questions? Copy Chapter

1. Read the docs
2. Open a feature request or report a bug
3. Join Discord community server

Overview (`tests/README.md`) Copy Section

Testing

Goals Copy Chapter

Be able to test the correctness of the

• aim engine
• aim sdk
• aim ql
• extensions

Folder Structure Copy Section

tests
  engine
    test_*.py
  sdk
    test_*.py
  ql
    test_*.py
  ext
    test_*.py

Run Copy Chapter

Run tests via command python -m unittest discover -s tests in the root folder.

Overview (`aim/web/ui/tasks/cli/README.md`) Copy Section

aim-ui/cli

Supported commands Copy Section

• create-component

  $ node cli/index.js create-component --name=<yourComponentname> --path=<yourComponentPath> --lint

NOTES:

Created two scripts inside ui project

$ npm run crc-kit -- <ComponentName>

Will create a component named ComponentName in ./src/components/kit folder, and will run lint to fix formatting

$ npm run crc -- <ComponentName>

Will create a component named ComponentName in ./src/components/ folder, and will run lint to fix formatting

Overview (`aim/web/ui/src/services/live-update/README.md`) Copy Section

Usage Copy Chapter

// to describe webpack as this is a worker file
    import LUWorker from 'comlink-loader!./services/live-update/Worker';
    import { getDataFromTransferable } from 'services/live-update'
    import * as Comlink from 'comlink';
    
    function initizeWorker(subscriber) {
        let worker =  new LUWorker();
        
        // set config [name, uri, delay, enableLog]
        worker.setConfig('Runs', 'runs/search/run', 1000, true);
        worker.subscribeToApiCallResult(
            // if subscriber has a context, please bind to the context
            Comlink.proxy(subscriber),
        );
        
        return worker;
    }
    
    const subscriber = (data) => {
        const obj = getDataFromTransferable(data);
        console.log("DATA --- ", obj);
        
        // send to render
    }
    
    const w = initizeWorker(subscriber);
    
    w.start({ q: 'test', limit: 50 })
    // some condition
    w.stop().then().catch()// will pause to call
    // some condition
    // w.start({ q: 'test1', limit: 60 }).then().catch() 
    
   // to terminate
   w.close();

Source Overview (`docs/source/overview.md`) Copy Section

Overview

Jump to:

• Quick start
• Aim UI
• Using Aim
• Understanding Aim
• Examples
• API reference

If you have questions:

• Open a feature request or report a bug
• Join our slack

Source Examples Overview (`docs/source/examples/overview.md`) Copy Section

Aim usage examples Copy Chapter

This section covers real-life examples of how Aim used to get insights
from tracked metadata in various domains.

Source Ui Overview (`docs/source/ui/overview.md`) Copy Section

Overview Copy Chapter

Aim UI is one of the key two interfaces to interact with the ML training runs tracked by Aim.
It's super-powerful for comparing large number of experiments.

The Aim UI is mainly built around the Explorers that allow to query and compare metrics, images and params, scatterplots etc.

Explorers are built around the way to query runs and compare them. Each explorer has a way to navigate to a single run where all the run-related information is visualized and can be used to do deep-dive into runs.

Besides these, Aim UI also allows to tag the runs, delete/archive them and save Explorers state to share with the team.

Pythonic Search Copy Section

Once you track your experiments with Aim, they are available to be searched.
There are three objects that could be searched:

• Runs
• Metrics
• Images

This basically means that you can write a python if-statement over everything you have tracked with a few options of the output.

Explorers encompass these outputs and give you unique supwer-powers to manipulate and compare the results of the search.

Explorers Copy Section

Explorers are powerful tools to query ML training runs, select specific type of data tracked (metrics, params etc), apply modifications to them and compare them.

The runs are compared by grouping and dividing them via the tracked hyperparams.
Every param (even the system params) are available on Aim - to be used for runs comparison on all Explorers.

Runs Management Copy Section

Runs Management includes the Runs Explorer.
You can search through runs on the Runs Explorer and have the holistic view of your runs, their diff, the last values of the metrics, all the params etc.

This also contains the Single Run Page that will help you observer everything you have tracked for your run - all in one place. This includes params, metrics, images, distributions etc.

Other Aim UI features Copy Section

Besides this, you can also save the Explorer states for reproducible experiment analysis.
Aim also enables ways to tag the runs.

Source Understanding Overview (`docs/source/understanding/overview.md`) Copy Section

Overview Copy Chapter

Aim is built around several concepts allowing to make sure that it meets the following criteria:

• Run data isolation. Each training run process isolated in terms of data and do not require additional services to run.
• Scalability. Aim web app is able to handle 1000s of training runs. Starting from v3.4.0 Aim provides a
  Remote Tracking server allowing to run multiple parallel experiments in a distributed multi-host environment.
• Flexibility. Aim UI and query language allow users to select, group and filter the tracked data any way they want.

Aim Components Copy Section

In order to understand how Aim works, lets take a quick look on a different components it has.

• Aim Storage. At its core Aim uses a custom-built storage, based on rocksdb. More details in 'Where is data collected?'. Data tracked by
  different training runs collected and indexed in an aim repository (.aim). The storage itself is generic; it
  allows accessing the data as collection of dictionaries and arrays.
• Aim SDK. On top of the storage Aim SDK provides functionality to track/select/query data. Additionally, SDK is a layer used by Web APIs
  and CLI.
• Aim UI. Web app allowing to browse run metadata, metrics, images and other tracked data.
• Aim CLI. A collection of command line utilities for running Aim web server, managing aim repositories, runs, etc.
• Remote Tracking server. A fastapi/uvicorn based service accepting incoming traffic and storing data on a centralized server.

The next sections will describe various concepts Aim introduces and provide more detailed look on
individual components introduced above.

Requirements (`docs/requirements.txt`) Copy Section

mistune==0.8.4
sphinx
sphinx_rtd_theme
m2r2==0.3.3.post2
sphinx-copybutton
Cython==3.0.10

Source Index (`docs/source/index.rst`) Copy Section

.. mdinclude:: overview.md

.. raw:: html

.. toctree::
overview.md

.. toctree::
:maxdepth: 1
:name: start
:caption: Quick Start

quick_start/setup.md
quick_start/integrations.md
quick_start/convert_data.md
quick_start/supported_types.md
quick_start/next_steps.md

.. toctree::
:maxdepth: 1
:name: ui
:caption: Aim UI

ui/overview.md
ui/pages/home_page.md
ui/pages/run_management.md
ui/pages/explorers.md
ui/pages/table.md
ui/pages/bookmarks.md
ui/pages/tags.md
ui/pages/reports.md

.. toctree::
:maxdepth: 2
:name: guides
:caption: Using Aim

using/manage_runs.md
using/configure_runs.md
using/query_runs.md
using/search.md
using/remote_tracking.md
using/training_monitoring.md
using/logging.md
using/artifacts.md
using/notifications.md
using/callbacks.md
using/k8s_deployment.md
using/jupyter_notebook_ui.md
using/sagemaker_notebook_ui.md
using/integration_guides.md

.. toctree::
:maxdepth: 2
:name: concepts
:caption: Understanding Aim

understanding/overview.md
understanding/data_storage.md
understanding/storage_indexing.md
understanding/concepts.md
understanding/running_aim_with_profiling.md
understanding/metric_step_hashing.md

.. toctree::
:maxdepth: 1
:name: examples
:caption: Examples

examples/images_explorer_gan.md

.. toctree::
:maxdepth: 2
:name: refs
:caption: References

refs/cli
refs/sdk
refs/ql
refs/storage

.. toctree::
:maxdepth: 1
:name: community
:caption: Community

generated/CHANGELOG.md

.. raw:: html

Indices and tables

• :ref:genindex
• :ref:modindex

Source Examples Images Explorer Gan (`docs/source/examples/images_explorer_gan.md`) Copy Section

Track and compare GANs with Aim Copy Chapter

Overview Copy Section

Generative Adversarial Networks, or GANs, are deep-learning-based generative models.

Generative modeling is an unsupervised learning task in machine learning that involves automatically discovering and learning the patterns of input data in such a way that the model can be used to generate new examples that plausibly could have been drawn from the original dataset.

In this guide we will show you how to integrate Aim with your GAN and GAN with EMA to compare the generated images from both experiments to compare their performances.

Experiment Copy Section

We will train and compare a regular GAN vs GAN with EMA. EMA is a technique for parameter averaging in GAN training, which computes an exponentially discounted sum of weights.

We will use lightweight-gan model implemented by lucidrains and MetFaces Dataset as a training dataset.

To be able to analyze the results we will fix random 64 points and track them during the training both for a regular GAN and GAN w/ EMA.

Track images with Aim Copy Section

1. Initialize a new run in the trainer class to collect and store sequences of images:

class Trainer():
    def __init__(
        self,
        name = 'default',
        results_dir = 'results',
        models_dir = 'models',
				...
		):
	...		
	self.run = aim.Run()           # Initialize aim.Run
	self.run['hparams'] = hparams  # Log hyperparams
	...

Code on GitHub

2. Track images generated by a regular GAN:

# Regular GAN

# Get generated images
generated_images = self.generate_(self.GAN.G, latents)

aim_images = []
for idx, image in enumerate(generated_images):
    ndarr = image.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).to('cpu', torch.uint8).numpy()
    im = PIL.Image.fromarray(ndarr)
    aim_images.append(aim.Image(im, caption=f'#{idx}'))

# Store with Aim (name="generated" and context.ema=0)
self.run.track(value=aim_images, name='generated', step=self.steps, context={'ema': False})

Code on GitHub

3. Track images generated by a GAN with enabled EMA:

# GAN with moving averages

# Get generated images
generated_images = self.generate_(self.GAN.GE, latents)

aim_images = []
for idx, image in enumerate(generated_images):
    ndarr = image.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).to('cpu', torch.uint8).numpy()
    im = PIL.Image.fromarray(ndarr)
    aim_images.append(aim.Image(im, caption=f'EMA #{idx}'))

# Store with Aim (name="generated" and context.ema=1)
self.run.track(value=aim_images, name='generated', step=self.steps, context={'ema': True})

Code on GitHub

Explore the results with Aim UI Copy Section

1. Visualize images generated by a regular GAN:

2. Visualize images generated by GAN with EMA:

As you may notice GAN with EMA converges in an exponential fashion and has better results at the end.

3. Let's compare the final step of the two methods side by side:

Conclusion Copy Section

As you can see GAN with EMA performed much better compared to the regular one.

With Aim you can easily compare diff groups of tracked images from diff runs.

Group them by the run hash, other parameters available to slice and dice and observe the difference between the runs.

Source Quick Start Convert Data (`docs/source/quick_start/convert_data.md`) Copy Section

Migrate from other tools Copy Chapter

The Aim explorers add true superpowers to the AI engineer's arsenal. However not all training runs may have been tracked
by Aim. So it is important to be able to port existing training run logs. There might be 1000s of training runs tracked
with other tools. Aim has built-in converters to easily migrate logs from other tools. These migrations cover the most
common usage scenarios. In case of custom and complex scenarios you can use Aim SDK to implement your own conversion
script.

As of Aim v3.6.0 the following converters are supported:

• TensorBoard logs converter
• MLFlow logs converter
• Weights & Biases logs converter

We are working to constantly improve existing converters and implement new ones.

Show TensorBoard logs in Aim Copy Section

Aim gives you a possibility to convert TensorBoard
logs into native format and show them directly inside the Aim UI.

Before showing the logs in Aim, the files have to pass the conversion process.

Please note that only the following plugins are currently supported

• scalar
• image

To convert TensorBoard logs, aim convert command must be run on your log directory.

aim convert tensorboard --logdir ~/my_logdir

To make conversion process smooth please ensure that logs directory structure follows conventions below. Consider the
following directory hierarchy:

~/my_logdir/
    ├> run_1/
    │    ├> <evnet_file_1>
    │    └> <evnet_file_2>
    ├> group_1/
    │    ├> <evnet_file_3> (THIS LOG WILL BE IGNORED)
    │    ├> run_2/
    │    │    ├> train/
    │    │    │    ├> <evnet_file_4>
    │    │    │    └> <evnet_file_5>
    │    │    ├> validate/
    │    │    │    ├> <evnet_file_6>
    │    │    │    └> <evnet_file_7>
    │    │    ├> <evnet_file_8> (IGNORED IF "--flat" IS ACTIVE)
    │    │    └> <evnet_file_9> (IGNORED IF "--flat" IS ACTIVE)
    │    └> run_3/
    │        ├> <evnet_file_10>
    │        └> <evnet_file_11>
    ├> <evnet_file_12> (THIS LOG WILL BE IGNORED)
    └> <evnet_file_13> (THIS LOG WILL BE IGNORED)

Note that directory naming is not mandated and its up to you how to name them.

The conversion logic categorizes your hierarchy into one of group, run and context
categories where.

• group: Is a directory which has one or more run directories inside it,
• run: Is a directory which has either log files or context directory inside it,
• context: Is a directory inside of run directory which has at least one log file inside it.

Conversion process will scan and determine run directories for your hierarchy and will create a distinct run for each
of them.

From the hierarchy example above you can see that the following log files will be ignored since the converter treats
them as unorganized log files.

• ~/my_logdir/group_1/evnet_file_3
• ~/my_logdir/evnet_file_12
• ~/my_logdir/evnet_file_13

All other logs will either have Context or No Context. Context of the log is the name of the parent directory if
the parent directory hasn't been categorized into neither as run nor group category.

For example:

• Log files right underneath run_1, run_2 and run_3 will have no context
• Log files in run_2/train and run_2/validate will have train and validate as context accordingly.

In case the converter finds unorganized log files in your hierarchy a warning message will be issued.

To make the converter process these logs, consider re-structuring your directories so that it matches the sample
structure. (i.e. create a new directory and moving your unorganized logs there)

You can make converter treat every directory as a distinct run by supplying --flat option. In this case the following
directories will be categorized as a run directory.

• ~/my_logdir/run_1/
• ~/my_logdir/group_1/run_2/train/
• ~/my_logdir/group_1/run_2/validate/
• ~/my_logdir/group_1/run_3/

The log files in all other directories will be ignored.

Show MLflow logs in Aim Copy Section

Aim gives you a possibility to convert MLflow runs into native format and show them directly on
Aim UI.

Before showing your MLlfow runs on Aim, they need to pass conversion process where your metrics, tags, parameters, run
description/notes and some artifacts will be transferred into Aim storage.

Please note that as for now, only the artifacts having the following file extensions will be transferred into Aim
storage!

• Images: ( 'jpg', 'bmp', 'jpeg', 'png', 'gif', 'svg' )

• Texts: ( 'txt', 'log', 'py', 'js', 'yaml', 'yml', 'json', 'csv', 'tsv', 'md', 'rst', 'jsonnet' )

• Sound/Audios: ( 'flac', 'mp3', 'wav' )

To convert MLflow runs, aim convert mlflow command must be run on your log directory:

$ aim init
$ aim convert mlflow --tracking_uri 'file:///Users/aim_user/mlruns'

You can also set the MLFLOW_TRACKING_URI environment variable to have MLflow find a URI from there. In both cases, the
URI can either be an HTTP/HTTPS URI for a remote server, a database connection string, or a local path to log data to a
directory.

The conversion process will iterate over all your Experiments and create a distinct run for each run inside the
experiment. If you want to process only a single experiment, you can provide the experiment id or name to the conversion
command:

$ aim convert mlflow --tracking_uri 'file:///Users/aim_user/mlruns' --experiment 0

While converting the artifacts, the converter will try to determine file content type only based on its extension. A
warning message will be issued if artifact cannot be categorized, these artifacts will not be transferred to aim!
Please check the command output logs if you fail to see your artifact in Aim's web.

Show Weights and Biases logs in Aim Copy Section

Aim gives you a possibility to convert Weights & Biases runs into native format and explore them via Aim UI.

To be able to explore Weights & Biases runs with Aim, please run the WandB to Aim converter. All the metrics, tags, config, experiment description/notes and some artifacts will be converted and stored in .aim repo.

Execute aim convert wandb command to start converting Weights & Biases experiments:

$ aim init
$ aim convert wandb --entity 'my_team' --project 'test_project'

The convertor will iterate over all the experiments and create a distinct Aim run for each experiment. If you want to process only a single experiment, please specify the experiment id or name when running the command:

$ aim convert wandb --entity my_team --project test_project --run-id '<HASH_OF_RUN>'

While converting the artifacts, the converter will try to determine file content type based on its extension. A
warning message will be issued if artifact will not be categorized, these artifacts will not be transferred to aim.
Please note that current implementation tracks only the primitive datatypes.

If there are any issues with the conversion process, please submit an issue.

Once conversion process is complete - you can enjoy the power of Aim 🚀

Source Quick Start Integrations (`docs/source/quick_start/integrations.md`) Copy Section

Integrate Aim into an existing project Copy Chapter

Any python script Copy Section

from aim import Run

run = Run()

# Save inputs, hparams or any other `key: value` pairs
run['hparams'] = {
    'learning_rate': 0.001,
    'batch_size': 32,
}

# ...
for step in range(10):
    # Log metrics to visualize performance
    run.track(step, name='metric_name')
# ...

Aim easily integrates with your favourite ML frameworks.

Aim loggers give access to the aim.Run object instance via the experiment property. The aim.Run instance will help you to easily track additional metrics or set any other key: value pairs (params) relevant to your project.

In this way you can easily extend the default integrations. More info about this is abailable on Integration guides section.

Integration with Pytorch Ignite Copy Section

It only takes 2 steps to simply and easily inject Aim into pytorch ignite:

# import aim sdk designed for pytorch ignite
from aim.pytorch_ignite import AimLogger

Pytorch Ignite provides trainer objects to simplify the training process of pytorch model. We can attach the trainer object as AimLogger's output handler to use the logger function defined by aim to simplify the process of tracking experiments. This process is divided into 2 steps:

Step 1. Create AimLogger object

aim_logger = AimLogger(
    experiment='aim_on_pt_ignite',
    train_metric_prefix='train_',
    val_metric_prefix='val_',
    test_metric_prefix='test_',
)

Step 2. Attach output handler to the aim_logger object

aim_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="train",
    output_transform=lambda loss: {'loss': loss}
)

Adapter source can be found here.
Example using Pytorch Ignite can be found here.

Integration with Pytorch Lightning Copy Section

We only require 2 steps to simply and easily inject Aim into pytorch lightining:

# import aim sdk designed for pl
from aim.pytorch_lightning import AimLogger

Pytorch lighting provides trainer objects to simplify the training process of pytorch model. One of the parameters is called logger. We can use the logger function defined by aim to simplify the process of tracking experiments. This process is divided into 2 steps:

Step 1. Create AimLogger object

# track experimental data by using Aim
aim_logger = AimLogger(repo='path/to/aim/repo', experiment='aim_on_pt_lightning')

Step 2. Pass the aim_logger object as the logger argument

# track experimental data by using Aim
trainer = Trainer(gpus=1, progress_bar_refresh_rate=20, max_epochs=5, logger=aim_logger)

Managing Aim Contexts

Aim contexts can be automatically derived from your metric names by defining prefixes and postfixes with the context_prefixes and context_postfixes kwargs. By default, AimLogger recognizes train_, val_, and test_ prefixes and assign metrics with those postfixes to the subset context.

For instance, calling self.log("val_loss", loss_value) in a pl.LightningModule instance will log loss_value to Aim in a metric called "loss" with the "val" context.

Changing this default and adding new contexts is possible with the following usage:

Aimlogger(...,
    context_prefixes = dict(subset={'train': 'train_',    # the default behavior,
                                    'val':   'val_',      # shown here as example
                                    'test':  'test_'}),  
    contest_postfixes = dict(custom_context={'weighted':'_weighted', # a new, user-defined 
                                             'macro':   '_macro'},   # metric-averaging context
)

To remove all default context mapping, do AimLogger(..., context_prefixes={}).

If you need more than two contexts per metric, or have other context needs, you can always directly access Aim's tracking capabilities from a pl.LightningModule or pl.Trainer instance with:
instance.logger.experiment.track(value, name=..., context=...)

For the Pytorch Lightning AimLogger, arguments train_metric_prefix, val_metric_prefix, test_metric_prefix are deprecated but functional. However, using them in conjuncture with context_prefixes will raise a ValueError.

Adapter source can be found here.
Example using Pytorch Lightning can be found here.

Integration with Hugging Face Copy Section

You only need 2 simple steps to employ Aim to collect data ❤️

Step 1: Import the sdk designed by Aim for Huggingface.

from aim.hugging_face import AimCallback

Step 2: Hugging Face has a trainer api to help us simplify the training process. This api provides a callback function to return the information that the user needs. Therefore, aim has specially designed SDK to simplify the process of the user writing callback functions, we only need to initialize AimCallback object as follows:

# Initialize aim_callback
aim_callback = AimCallback(experiment='huggingface_experiment')
# Initialize trainer
trainer = Trainer(
    model=model,    
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
    callbacks=[aim_callback]
)

Adapter source can be found here.
Example using Hugging Face can be found here.

Integration with Keras & tf.Keras Copy Section

It only takes 2 steps to easily integrate aim in keras to record experimental information.

# call keras as the high api of tensorflow 
from aim.tensorflow import AimCallback
# call keras library directly
from aim.keras import AimCallback

In keras, we call the fit() method of the model object to train the data. The callbacks are provided here. AimCallback inherits the usage specification of callbacks. We just need to add it to the callbacks list.

model.fit(x_train, y_train, epochs=5, callbacks=[
          # in case of tf.keras, we use aim.tensorflow.AimCallback 
          AimCallback(experiment='aim_on_keras')                                      
])

Adapter source can be found here.
Example using Keras can be found here.
Example using tf.Keras can be found here.

Integration with Keras Tuner Copy Section

It only takes 2 steps to easily integrate aim in keras to record experimental information.

from aim.keras_tuner import AimCallback

In kerastuner, we call the search() method of the tuner object to perform a search for best hyperparameter configuations. The callbacks are provided here. AimCallback inherits the usage specification of callbacks. We just need to add it to the callbacks list.

tuner.search(
    train_ds,
    validation_data=test_ds,
    callbacks=[AimCallback(tuner=tuner, repo='./aim_logs', experiment='keras_tuner_test')],
)

Adapter source can be found here.
Example using Keras Tuner can be found here.

Integration with XGboost Copy Section

Enjoy using aim to track xgboost experimental data which requires two simple steps:

Step 1: Explicitly import the AimCallback for tracking training data.

# call sdk aim.xgboost 
from aim.xgboost import AimCallback

Step 2: XGboost provides the xgboost.train method for model training, in which the callbacks parameter can call back data information from the outside. Here we pass in aimcallbacl designed for tracking data information

xgboost.train(param, dtrain, num_round, watchlist,
                            callbacks=[AimCallback(experiment='xgboost_test')])

Adapter source can be found here.
Example using XGboost can be found here.

Integration with CatBoost Copy Section

It only takes two steps to integrate Aim with CatBoost.

Step 1: Import AimLogger to track the training metadata.

# call sdk aim.catboost 
from aim.catboost import AimLogger

Step 2: Pass the logger to the trainer.

Trainings in CatBoost are initiated with fit method.
The method can be supplied with log_cout parameter to redirect output logs into a custom handler.
Pass AimLogger to automatically track metrics and hyper-parameters with Aim.
Depending on the training log output, an additional argument logging_level could be passed to make Catboost yield more logs to track test & best values.

model.fit(train_data, train_labels, log_cout=AimLogger(loss_function='Logloss'), logging_level='Info')

AimLogger also accepts log_cout parameter to preserve the default functionality of Catboost's log handling.
You can pass your own handler, else it defaults to sys.stdout.

See AimLogger source here.
Check out a simple example with Aim and CatBoost here.

Integration with LightGBM Copy Section

Aim comes with a builtin callback designed to automatically track LightGBM trainings.
It takes two steps to integrate Aim into your training script.

Step 1: Explicitly import the AimCallback for tracking training metadata.

from aim.lightgbm import AimCallback

Step 2: Pass the callback to callbacks list upon initiating your training.

gbm = lgb.train(params,
                lgb_train,
                num_boost_round=20,
                valid_sets=lgb_eval,
                callbacks=[AimCallback(experiment='lgb_test')])

While your training is running you can start aim up in another terminal session and observe the information in real
time.

See AimCallback source here.
Check out a simple regression task example here.

Integration with fastai Copy Section

Aim comes with a builtin callback designed to automatically track fastai trainings.
It takes two steps to integrate Aim into your training script.

Step 1: Explicitly import the AimCallback for tracking training metadata.

from aim.fastai import AimCallback

Step 2: Pass the callback to cbs list upon initiating your training.

learn = cnn_learner(dls, resnet18, pretrained=True,
                    loss_func=CrossEntropyLossFlat(),
                    metrics=accuracy, model_dir="/tmp/model/",
                    cbs=AimCallback(repo='.', experiment='fastai_example'))

See AimCallback source here.
Check out a simple regression task example here.

Integration with MXNet Copy Section

To track MXNet experiments use Aim callback designed for MXNet fit method.
It takes two steps to integrate Aim into your training script.

Step 1: Import the AimLoggingHandler for tracking training metadata.

from aim.mxnet import AimLoggingHandler

Step 2: Pass a callback instance to event_handlers list upon initiating your training.

aim_log_handler = AimLoggingHandler(repo='.', experiment_name='mxnet_example',
                                    log_interval=1, metrics=[train_acc, train_loss, val_acc])

est.fit(train_data=train_data_loader, val_data=val_data_loader,
        epochs=num_epochs, event_handlers=[aim_log_handler])

See AimCallback source here.
Check out a simple regression task example here.

Integration with Optuna Copy Section

Aim provides a callback designed to automatically track optuna trainings.
The as_multirun is a boolean argument. If as_multirun is set True then the callback will create a run for each trial. Otherwise it will track all of the results in a single run.
One can also use the decorator function track_in_aim to log inside the objective function.

Step 1: Explicitly import the AimCallback for tracking training metadata.

from aim.optuna import AimCallback

Step 2: Pass the callback to cbs list upon initiating your training.

aim_callback = AimCallback(experiment_name="optuna_single_run")
study.optimize(objective, n_trials=10, callbacks=[aim_callback])

See AimCallback source here.
Check out a simple objective optimization example here.

Integration with PaddlePaddle Copy Section

Aim provides a built in callback to easily track PaddlePaddle trainings.
It takes two steps to integrate Aim into your training script.

Step 1: Explicitly import the AimCallback for tracking training metadata.

from aim.paddle import AimCallback

Step 2: Pass the callback to callbacks list upon initiating your training.

callback = AimCallback(repo='.', experiment='paddle_test')
model.fit(train_dataset, eval_dataset, batch_size=64, callbacks=callback)

See AimCallback source here.
Check out a simple objective optimization example here.

Integration with Stable-Baselines3 Copy Section

Aim provides a callback to easily track one of the reliable Reinforcement Learning implementations Stable-Baselines3 trainings.
It takes two steps to integrate Aim into your training script.

Step 1: Explicitly import the AimCallback for tracking training metadata.

from aim.sb3 import AimCallback

Step 2: Pass the callback to callback upon initiating your training.

model.learn(total_timesteps=10_000, callback=AimCallback(repo='.', experiment_name='sb3_test'))

See AimCallback source here.
Check out a simple objective optimization example here.

Integration with Acme Copy Section

Aim provides a built in callback to easily track Acme trainings.
It takes few simple steps to integrate Aim into your training script.

Step 1: Explicitly import the AimCallback and AimWriter for tracking training metadata.

from aim.sdk.acme import AimCallback, AimWriter

Step 2: Initialize an Aim Run via AimCallback, and create a log factory using the Run.

aim_run = AimCallback(repo=".", experiment_name="acme_test")
def logger_factory(
    name: str,
    steps_key: Optional[str] = None,
    task_id: Optional[int] = None,
) -> loggers.Logger:
    return AimWriter(aim_run, name, steps_key, task_id)

Step 3: Pass the logger factory to logger_factory upon initiating your training.

experiment_config = experiments.ExperimentConfig(
    builder=d4pg_builder,
    environment_factory=make_environment,
    network_factory=network_factory,
    logger_factory=logger_factory,
    seed=0,
    max_num_actor_steps=5000)

See AimCallback source here.
Check out a simple objective optimization example here.

Integration with Prophet Copy Section

Aim provides an AimLogger object designed to track Prophet hyperparameters and metrics.
It takes three steps to integrate Aim into your Prophet script.

Step 1: Explicitly import the AimLogger.

from aim.prophet import AimLogger

Step 2: After initializing a Prophet model, instantiate the AimLogger with your Prophet model.

model = Prophet()
logger = AimLogger(prophet_model=model, repo=".", experiment="prophet_test")

Step 3 (optional): pass any metrics you want after fitting the Prophet model.

metrics = {"backtest_mse": backtest_mse, "backtest_mape": backtest_mape}
logger.track_metrics(metrics)

Note that the metrics are assumed to be validation metrics by default. Alternatively, you can pass a context argument to the track_metrics method.

metrics = {"train_mse": backtest_mse, "train_mape": backtest_mape}
logger.track_metrics(metrics, context={"subset": "train"})

See AimLogger source here.
Check out a simple example here.

What's next? Copy Section

During the training process, you can start another terminal in the same directory, start aim up and you can observe
the information in real time.

Source Quick Start Next Steps (`docs/source/quick_start/next_steps.md`) Copy Section

Next steps Copy Chapter

In the Quick Start section you learned the basics of Aim functionality and how to use the Aim SDK.

In the following sections we will cover different aspects of Aim in more detail.
Aim is designed and built to explore and make sense of large volumns of training run logs.

If you are interested in exploring more about Aim, please navigate to Using Aim section.

You can browse real-life Aim use-cases in Examples section.

If you want to learn more about how Aim works under the hood and get familiar with Aim
terminology, navigate to Understanding Aim section.

Detailed API descriptions as well as CLI commands available at References.

Source Quick Start Setup (`docs/source/quick_start/setup.md`) Copy Section

Get started with Aim Copy Chapter

This section shows a simple end-to-end aim setup. It starts from the installation, shows how to run Aim UI and explore the
results.
Use this as a starting point to get familiar with the basics of Aim while getting up and running.

Installing Aim Copy Section

Aim is a python package available for Linux and MacOs for Python versions 3.6+. Install Aim using pip3:

pip3 install aim

Verify aim was properly installed

aim version

You should see the line listing newly installed version of Aim. For instance:

Aim v3.5.1

The installed package includes Python SDK needed for tracking training runs, UI for browsing the results and CLI
for managing UI and results.

Nightly releases Copy Section

Aim also provides daily dev packages with the features developed between main minor releases.

pip3 install --pre aim

Please note, that if the dependencies of aim are not already installed, this command will try to install the development versions of those packages as well.

Previous daily dev packages can be installed using the following command:

pip3 install aim==3.x.0.devyyyymmdd

Release history

Initializing Aim repository Copy Section

Aim repository is the space where all your training runs are logged.

To initialize aim repo in the current working directory, run:

aim init

You should see something like this on your Command line:

Initialized a new Aim repository at /home/user/aim

Your workspace is now ready for tracking training runs with Aim.

Tracking data with Aim SDK Copy Section

To start tracking, first create aim.Run object:

from aim import Run
run = Run()

Run class provides a dictionary-like interface for storing training hyperparameters and other dictionary-like metadata:

hparams_dict = {
    'learning_rate': 0.001,
    'batch_size': 32,
}
run['hparams'] = hparams_dict

These params can be used later on the UI to query runs, metrics, images.
To track metrics with aim use the Run.track method:

run.track(3.0, name='loss')

The complete list of supported inputs is available in section "Track media and objects"

Here's a full example demonstrating the steps above:

# aim_test.py
from aim import Run

run = Run()

# set training hyperparameters
run['hparams'] = {
    'learning_rate': 0.001,
    'batch_size': 32,
}

# log metric
for i in range(10):
    run.track(i, name='numbers')

Run the script above

python3 aim_test.py

Congrats! Your first run with Aim is ready!
Now it is time to explore results with Aim UI.

Browsing results with Aim UI Copy Section

Once the script above finishes you can open Aim UI and see the results:

aim up

You should see the following output meaning Aim UI is up and running:

Running Aim UI on repo `<Repo#-5930451821203570655 path=/.aim read_only=None>`
Open http://127.0.0.1:43800
Press Ctrl+C to exit

Open your browser and navigate to http://127.0.0.1:43800
You should be able to see the home page of Aim UI:

Click on Metrics Explorer icon

In the Search bar select a "numbers" metric and click "Search".
You should be able to see line chart with tracked metric:

Running Aim UI inside notebooks Copy Section

Run the following commands in the notebook to run the Aim UI:

1. Load Aim extension for notebooks:

%load_ext aim

2. Run %aim up to open Aim UI in the notebook:

%aim up

See integration guide with Jupyter Notebook for more details.

Running Aim UI and tracking server inside Docker container Copy Section

Aim Docker images are available for running Aim UI and Aim Remote tracking server. Default command
for Aim Docker image is aim up. To spin Docker container with Aim UI with port mapping:

docker run --publish 43800:43800 aimstack/aim

To run container with Aim Remote tracking server:

docker run --publish 53800:53800 aimstack/aim server

Source Quick Start Supported Types (`docs/source/quick_start/supported_types.md`) Copy Section

Track media and objects Copy Chapter

Aim supports variety of data sources. Basic logging of Run params covers Python builtin types (such as int, float
, bool, bytes and str) as well as composition of those into dictionaries, lists, tuples at any depth.

In addition to the builtin types, Aim provides native support for
OmegaConf configs, thus simplifying integration for projects
running with Hydra.

Starting from v3.6.0 Aim provides integration with activeloop/hub
datasets. Hub is the open-source dataset format for AI.

Tracking of data includes metrics, images, audio, text and chart figures. Here's the complete list of Aim objects
provided by the package:

• Metrics
• Distribution
• Image
• Audio
• Text
• Figure

Tracking multiple values Copy Section

Starting from v3.14.0 Aim provides ability to track multiple values at once. To track multiple values of different
sequences of the same context, just pass a dictionary of <name: value> pairs to the track method:

aim_run.track({'accuracy': 98.2, 'loss': 0.001}, context={'subset': 'train'}, step=10, epoch=1)

Note: The name argument should be set to None in this case.

More examples

The following code snippet

from aim import Run

aim_run = Run()

metrics = {'accuracy': 0.72, 'f1': 0.99}
for metric, val in metrics.items()
  aim_run.track(val, name=metric, context={'subset': 'train'})

can be simplified to:

from aim import Run

aim_run = Run()

aim_run.track({'accuracy': 0.72, 'f1': 0.99}, context={'subset': 'train'})

Distribution tracking with Aim Copy Section

You can store distribution objects in Aim repository using our aim.Distribution object.

from aim import Distribution

aim.Distribution accepts the following parameters

• distribution: array-like object used to construct aim.Distribution.
• bin_count: Optional distribution bin count. 64 by default, max 512.

Your data is converted to numpy.histogram upon initialization of the object.

Simple example of initializing and tracking distribution

import random
from aim import Run, Distribution

run = Run()
d = Distribution(
    distribution=[random.randrange(0, 10000) for _ in range(1000)],
    bin_count=250
)
run.track(d, name='dist', step=0)

Image tracking with Aim Copy Section

Aim lets your track an image using aim.Image object

To get started, first import the Image object into your code.

from aim import Image

Our Image object uses Pillow under the hood. Image object supports the
following inputs as data source.

• Path to an image file
• PIL (Pillow object)
• torch.Tensor (PyTorch tensor object)
• tf.Tensor (TensorFLow tensor object)
• np.array (Numpy array object)
• matplotlib.figure.Figure (matplotlib figure object)

Here's an example of tracking image from file path

path = "~/test_image.png"
aim_image = Image(path)

Image object also has the following arguments:

caption  (:obj: `str`,  optional): Optional image caption. '' by default. 
format   (:obj: `str`,  optional): Parameter for PIL's .save() method. 'png' by default.
quality  (:obj: `int`,  optional): Parameter for PIL's .save() method. 85 by default.
optimize (:obj: `bool`, optional): Parameter for PIL's .save() method. False by default.

For more information on the format, quality and optimize parameters, refer
to Pillow documentation.

Using these parameters you can manipulate image quality and/or convert the image format from .png to jpeg or to any
other format (which is supported by Pillow)

from aim import Run, Image

# Initialize a new run
run = Run()

for step in range(1000):
    # Log image
    path = f"~/test_image_{step}.png"
    aim_image = Image(
        path,
        format='jpeg',
        optimize=True,
        quality=50
    )

    run.track(aim_image, name='images', step=step)

Audio tracking with Aim Copy Section

Aim lets your track an audio data using aim.Audio object

To get started, first import the Audio object into your code.

from aim import Audio

You can use Audio object to track MP3, WAV and FLAC audio data. Audio object supports the following data as input.

• File path
• Raw bytes
• io.BytesIO stream
• Numpy array (only for WAV audio format)

This object comes with the following optional arguments.

format  (:obj:`str`): Format of the audio source. Choices are ('flac', 'mp3', 'wav')
rate    (:obj:`int`): Only for WAV. Rate of the audio file, defaults to 22500
caption (:obj:`str`): Optional audio caption. An empty string by default.

Complete example of tracking WAV audio data.

import os.path
from aim import Run, Audio

# Initialize a new run
run = Run()

for step in range(1000):
    # Log image
    path = f"~/test_audio_{step}.mp3"
    aim_audio = Audio(
        path,
        format='mp3',
        caption=os.path.basename(path)
    )

    run.track(aim_audio, name='audios', step=step)

Text tracking with Aim Copy Section

Aim lets your track text/string during your training process.

To get started, first import the Text object into your code.

from aim import Text

In order to use the Text object, you just need to ensure that your input data type is a string.

Here's an example of Text usage:

import random
import string
from aim import Run, Text

# Initialize a new run
run = Run()

for step in range(100):
    # Generate a random string for this example
    random_str = ''.join(random.choices(
        string.ascii_uppercase +
        string.digits, k=20)
    )
    aim_text = Text(random_str)
    run.track(aim_text, name='text', step=step)

Figure tracking with Aim Copy Section

Aim provides a Figure object which can be used to track plotly
and matplotlib figures.

To get started, first import the Figure object into your code.

from aim import Figure

You should pass
either Plotly Figure
or matplotlib Figure as input source to Aim's Figure object.

Here's an example of tracking a plotly figure

import plotly.express as px
from aim import Run, Figure

# Initialize a new run
run = Run()

# First we create Plotly figure
fig = px.bar(x=["a", "b", "c"], y=[1, 3, 2])

# Now we convert it to Aim Figure
aim_figure = Figure(fig)

run.track(aim_figure, name="plotly_figures", step=0)

It is also easy to track matplotlib figure. Please note that the conversion process is done by Plotly under the hood.

from aim import Run, Figure
import matplotlib.pyplot as plt

# Initialize a new run
run = Run()

# define matplotlib figure
fig = plt.figure()
plt.plot([1, 2, 3])
plt.close(fig)

# Now we convert it to Aim Figure using (Plotly's functions)
aim_figure = Figure(fig)

run.track(aim_figure, name="matplotlib_figures", step=0)

Tracking matplotlib figures with Aim Copy Section

In order to track matplotlib figures with Aim, either pass the matplotlib figure to Aim's Image or Figure
object.

Converting matplotlib to Aim Image

from aim import Run, Image
import matplotlib.pyplot as plt

run = Run()

# define matplotlib figure
fig = plt.figure()
plt.plot([1, 2, 3])
plt.close(fig)

# pass it to aim Image
aim_img = Image(fig)
run.track(aim_img, step=0, name="matplotlib_images")

Converting matplotlib to Aim Figure

Please note that the conversion process is done by Plotly under the hood.

from aim import Run, Figure
import matplotlib.pyplot as plt

run = Run()

# define matplotlib figure
fig = plt.figure()
plt.plot([1, 2, 3])
plt.close(fig)

aim_figure = Figure(fig)
run.track(aim_figure, step=0, name="matplotlib_figures")

Logging activeloop/hub dataset info with Aim Copy Section

Aim provides wrapper object for hub.dataset. It allows to store the dataset info as a Run
parameter and retrieve it later just as any other Run param. Here is an example of using Aim to log dataset info:

import hub

from aim.sdk.objects.plugins.hub_dataset import HubDataset
from aim.sdk import Run

# create dataset object
ds = hub.dataset('hub://activeloop/cifar100-test')

# log dataset metadata
run = Run(system_tracking_interval=None)
run['hub_ds'] = HubDataset(ds)

Log DVC metadata with Aim Copy Section

If you are using DVC to version your datasets or track checkpoints / other large chunks of data, you
can use Aim to record the info about the tracked files and datasets on Aim. This will allow to easily connect your
datasets info to the tracked experiments. Here is how the code looks like

from aim.sdk import Run
from aim.sdk.objects.plugins.dvc_metadata import DvcData

run = Run(system_tracking_interval=None)

path_to_dvc_repo = '.'
run['dvc_info'] = DvcData(path_to_dvc_repo)

If we consider the following sample repo provided by DVC team:

Run the following command to list repository contents, including files and directories tracked by DVC and by Git.

$ git clone https://github.com/iterative/example-get-started
$ cd example-get-started
$ dvc list .
.dvcignore
.github
.gitignore
README.md
data
dvc.lock
dvc.yaml
model.pkl
params.yaml
prc.json
roc.json
scores.json
src

If we apply our previous code snippet on the same repo - we can observe the same information added to Run parameters.

{
    'dvc_info.dataset.source': 'dvc',
    'dvc_info.dataset.tracked_files': [
        '.dvcignore', '.github', '.gitignore',
        'README.md', 'data', 'dvc.lock',
        'dvc.yaml', 'model.pkl', 'params.yaml',
        'prc.json', 'roc.json', 'scores.json', 'src'
    ]
}

Logging huggingface/datasets dataset info with Aim Copy Section

Aim provides wrapper object for datasets. It allows to store the dataset info as a Run
parameter and retrieve it later just as any other Run param. Here is an example of using Aim to log dataset info:

from datasets import load_dataset

from aim import Run
from aim.hf_dataset import HFDataset

# create dataset object
dataset = load_dataset('rotten_tomatoes')

# store dataset metadata
run = Run()
run['datasets_info'] = HFDataset(dataset)

Source Refs Cli (`docs/source/refs/cli.md`) Copy Section

Aim CLI Copy Chapter

Aim CLI offers a simple interface to easily organize and record your experiments. Paired with the Python Library, Aim is
a powerful utility to record, search and compare AI experiments. Here are the set of commands supported:

init Copy Section

This step is optional.
Initialize the aim repo to record the experiments.

$ aim init

Creates .aim directory to save the recorded experiments to. Running aim init in an existing repository will prompt
the user for re-initialization.

Beware: Re-initialization of the repo clears .aim folder from previously saved data and initializes new repo.
Note: This command is not necessary to be able to get started with Aim as aim is automatically initializes with
the first aim function call.

version Copy Section

Display the Aim version installed.

$ aim version

up Copy Section

Start the Aim web UI locally.

$ aim up [ARGS]

server Copy Section

Run a tracking server to collect tracked data from remote clients.

$ aim server [ARGS]

runs Copy Section

Upgrade Aim repository runs data.

$ aim runs [ARGS] SUBCOMMAND

runs subcommands

Global expression (*) support is available for run hashes. If hash contains *, it must be enclosed within
quotes ('') as bash resolves the expression before passing it to aim runs command.

$ aim runs ls

$ aim runs rm [HASH] ...

$ aim runs cp [ARGS] [HASH] ...

$ aim runs mv [ARGS] [HASH] ...

$ aim runs upload [ARGS] ...

convert Copy Section

Tool-set for converting 3rd party data into Aim readable format.

$ aim convert [ARGS] SUBCOMMAND

convert subcommands

Sub-command: tensorboard

Sub-command: mlflow

storage Copy Section

Perform various maintenance operations on Aim repository.

$ aim storage [ARGS] SUBCOMMAND

storage subcommands

Sub-command: update 3.11+

$ aim storage upgrade 3.11+ [HASH] ...

Sub-command: restore

$ aim storage restore [HASH] ...

Sub-command: reindex

Sub-command: prune

$ aim storage prune

Aim status watcher CLI Copy Chapter

Aim status CLI offers an interface to start training runs status monitoring and configure
how the notifications should be received. The entry point to this CLI is aim-watcher:

$ aim-watcher [ARGS] SUBCOMMAND

start Copy Section

Start watcher service for the given aim Repo.

aim-watcher --repo . start

notifiers Copy Section

Manipulate notifiers configuration.

notifiers subcommands

$ aim-watcher notifiers add

$ aim-watcher notifiers list

$ aim-watcher notifiers remove [NOTIFIER_ID]

$ aim-watcher notifiers disable [NOTIFIER_ID]

$ aim-watcher notifiers enable [NOTIFIER_ID]

Source Refs Sdk (`docs/source/refs/sdk.rst`) Copy Section

Aim SDK

aim.sdk.repo module Copy Chapter

.. automodule:: aim.sdk.repo
:members: Repo

aim.sdk.run module Copy Chapter

.. automodule:: aim.sdk.run
.. currentmodule:: aim.sdk.run

.. autoclass:: Run
:inherited-members:
:members:
:special-members: setitem, getitem, delitem

aim.sdk.objects.image Copy Chapter

.. automodule:: aim.sdk.objects.image
.. currentmodule:: aim.sdk.objects.image

.. autoclass:: Image
:members:

aim.sdk.objects.distribution Copy Chapter

.. automodule:: aim.sdk.objects.distribution
.. currentmodule:: aim.sdk.objects.distribution

.. autoclass:: Distribution
:members:

aim.sdk.objects.audio Copy Chapter

.. automodule:: aim.sdk.objects.audio
.. currentmodule:: aim.sdk.objects.audio

.. autoclass:: Audio
:members:

aim.sdk.objects.text Copy Chapter

.. automodule:: aim.sdk.objects.text
.. currentmodule:: aim.sdk.objects.text

.. autoclass:: Text
:members:

aim.sdk.objects.figure Copy Chapter

.. automodule:: aim.sdk.objects.figure
.. currentmodule:: aim.sdk.objects.figure

.. autoclass:: Figure
:members:

aim.sdk.sequence module Copy Chapter

.. automodule:: aim.sdk.sequence
.. currentmodule:: aim.sdk.sequence

.. autoclass:: Sequence
:members:

aim.sdk.sequences.metric module Copy Chapter

.. automodule:: aim.sdk.metric
.. currentmodule:: aim.sdk.metric

.. autoclass:: Metric
:exclude-members: allowed_dtypes
:members:

aim.sdk.sequences.image_sequence module Copy Chapter

.. automodule:: aim.sdk.sequences.image_sequence
.. currentmodule:: aim.sdk.sequences.image_sequence

.. autoclass:: Images
:exclude-members: allowed_dtypes
:members:

aim.sdk.sequences.distribution_sequence module Copy Chapter

.. automodule:: aim.sdk.sequences.distribution_sequence
.. currentmodule:: aim.sdk.sequences.distribution_sequence

.. autoclass:: Distributions
:exclude-members: allowed_dtypes
:members:

aim.sdk.sequences.audio_sequence module Copy Chapter

.. automodule:: aim.sdk.sequences.audio_sequence
.. currentmodule:: aim.sdk.sequences.audio_sequence

.. autoclass:: Audios
:exclude-members: allowed_dtypes
:members:

aim.sdk.sequences.text_sequence module Copy Chapter

.. automodule:: aim.sdk.sequences.text_sequence
.. currentmodule:: aim.sdk.sequences.text_sequence

.. autoclass:: Texts
:exclude-members: allowed_dtypes
:members:

aim.sdk.sequences.figure_sequence module Copy Chapter

.. automodule:: aim.sdk.sequences.figure_sequence
.. currentmodule:: aim.sdk.sequences.figure_sequence

.. autoclass:: Figures
:exclude-members: allowed_dtypes
:members:

aim.sdk.sequence_collection module Copy Chapter

.. automodule:: aim.sdk.sequence_collection
.. currentmodule:: aim.sdk.sequence_collection

.. autoclass:: SequenceCollection
:members:

.. autoclass:: SingleRunSequenceCollection
:show-inheritance:
:members: None

.. autoclass:: QuerySequenceCollection
:show-inheritance:
:members: None

.. autoclass:: QueryRunSequenceCollection
:show-inheritance:
:members: None

Source Refs Storage (`docs/source/refs/storage.rst`) Copy Section

Aim Storage

aim.storage.arrayview module Copy Chapter

.. automodule:: aim.storage.arrayview
:members: ArrayView

Source Ui Guides (`docs/source/ui/guides.md`) Copy Section

Guides Copy Chapter

Searching Copy Section

Grouping Copy Section

Control Panel Copy Section

Table Actions Copy Section

Source Understanding QL Concepts (`docs/source/understanding/QL_concepts.md`) Copy Section

Aim Query Language Copy Chapter

TODO: add content about query execution, laziness, etc.

Source Understanding Concepts (`docs/source/understanding/concepts.md`) Copy Section

Concepts Copy Chapter

All the functionality in Aim is build around several key concepts. This chapter will give a brief overview of these core
concepts. For more details please check the Reference section or Glossary.

Aim Run Copy Section

Run is an abstraction representing the tracked data for a single experiment. Its in memory model is SDK class aim.Run.
It is a core class used in your training script for tracing metrics and objects, as well as storing training hyperparams
and other data. Run object are queryable and UI provides a rich functionality for exploring runs and browsing single run
details.

Aim Repo Copy Section

While you do multiple training experiments, multiple runs data stored in a single directory
called Aim repository (repo for short). You can think of aim repo as an application centralized database.
SDK provides an in memory model for repo aim.Repo class. It is responsible for repository resources management and
might be used to query and/or iterate over the stored data.

Run Params Copy Section

Each run has a set of parameters associating with it. This might include the training script hyperparameters,
dataset information, etc. The Run object provides dictionary-like interface to set and access run params. Run parameters
are also available in the context of queries. You can set the whole configuration at once with the syntax like this:

run['hparams'] = conf

At this moment Run supports setting configuration from Python dictionaries and OmegaConf configs.
Support of popular configuration formats constantly added. You can check the full list in
Supported Data types section.

Run Sequence Copy Section

The sequence is a set of homogeneous ordered objects. In aim sequence must be bound to the Run object. When the value
is tracked in aim, it is appended to an existing or newly created Sequence object. The entire sequences can be queried
using aim QL and each sequence can be sliced further down. Sequence object is agnostic to the element type it holds. The way how
the sequence represented in UI, and the set of additional operations it might have depends on the element type. For example
Metric is a sequence of scalars. It can be represented as a value chart in UI, and SDK provides methods to convert it to
numpy.ndarray.

Sequence Context Copy Section

The sequence context provides a mechanism to query/group multiple sequences beyond simple string comparison on sequence name.
Sequences with the same name but with different context can perfectly coexist in the scope of one Run. In other words,
sequence defined by its Run, name and context. The example usage of this is tracking the same metric 'loss' for different stages
of training (train, validation, test). The resulting Run will have 3 metrics:

1. 'loss' {'subset': 'train'}
2. 'loss' {'subset': 'val'}
3. 'loss' {'subset': 'test'}
   Here is a small code example demonstrating how to specify context for a metric sequence:

from aim import Run

aim_run = Run()
for i in range(100):
    if i % 2 == 0:
        aim_run.track(i, name=r'numbers', context={'odds': True})
    else:
        aim_run.track(i, name=r'numbers', context={'odds': False})

Source Understanding Data Storage (`docs/source/understanding/data_storage.md`) Copy Section

Data storage - where Aim data is collected Copy Chapter

This section provides a deep-dive into Aim storage structure. It is important
to know the internal storage organization in order to understand how it affects
queries performance.

Storage structure Copy Section

The core foundation for Aim storage is rocksdb. It is a fast, embedded key-value
store maintained by facebook. The aim repository is a collection of individual
rocksdb databases with abstraction layers added to manage the collection as one database. The abstraction of
a single KV store called Container. Below is the directory structure for a typical Aim project:

.aim
    run_metadata.sqlite
    meta/
        index/
        chunks/
            aacf48e769534c32a9cc5a3c/
            80483ab611a24bf5bd8fc288/
            16f83a2c2f50477f8446f322/
        progress/
            aacf48e769534c32a9cc5a3c
            80483ab611a24bf5bd8fc288
        locks/
            aacf48e769534c32a9cc5a3c
            80483ab611a24bf5bd8fc288
    seqs/
        chunks/
            aacf48e769534c32a9cc5a3c/
            80483ab611a24bf5bd8fc288/
            16f83a2c2f50477f8446f322/
        progress/
            aacf48e769534c32a9cc5a3c
            80483ab611a24bf5bd8fc288
        locks/
            aacf48e769534c32a9cc5a3c
            80483ab611a24bf5bd8fc288

There are two main parts of the storage:

• run_metadata.sqlite: SQLite database for storing Run structured data, such as, creation time,
  name, Experiment it is attached to, Tags etc.
• meta/ and seqs/ directories: a collection of rocksdb storages. Used to write Runs tracked
  data, such as params, metrics and objects.

In the tree above the hash-strings (i.e. aacf48e769534c32a9cc5a3c) represent a single Run.
When the new Run is started, aim will create two Containers:

• Meta container for logged params as well as metadata about collected sequences, contexts, etc.
• Sequence container for the value series.

The reason the actual sequence data separated from metadata is the necessity for fast queries, regardless
sequence size.

Additionally, per each container two files will be created to properly manage the container state

• lock file, indicating that the container opened in write-mode.
• progress file, indicating that container potentially has un-indexed data.

This setup allows implementing concurrent training jobs setup without requiring additional synchronization
routines, and without the risk of losing or corrupting the data. For example, two jobs might run on different hosts where aim repo mounted on a shared NFS location.

What is the index container? Copy Section

Each run writes data into its own isolated containers. The aim queries require reading
the Run metadata from meta container. However, with 1000s of runs opening each meta container database will slow-down
the queries. Here the indexing of metadata becomes crucial.

Run object maintains lock for both meta and sequence containers during the training script execution. Run will
continue to write its data into its own containers. Once the execution finishes, meta container data indexed,
the container locks released, and the progress file removed. The sequence container data is not indexed, since the
individual points of a sequence are not queryable, and the sequence info is available in meta container.

How data written to/read from the storage? Copy Section

Run object provides interface for logging the Run parameters dict-like data and tracking series of scalars
and objects. Aim has a custom encoding layer which translates this hierarchical data into the sets of key-value
pairs to be written into rocksdb. The same encoding layer is responsible for re-constructing the tracked data/objects.
During query execution, aim SDK will walk through all runs in index container + the meta containers for chunks which
have progress file (remember that progress file indicates potentially un-indexed data for the Run). If the run/sequence
match the query expression, the appropriate run/sequence will be yielded. Notice that till this point no data was accessed
from the sequence container. The sequence data itself is read upon request.

Source Understanding Glossary (`docs/source/understanding/glossary.md`) Copy Section

Glossary Copy Chapter

TODO

Source Understanding Metric Step Hashing (`docs/source/understanding/metric_step_hashing.md`) Copy Section

Storing Metric sequence data for fast access Copy Chapter

Background Copy Section

Aim allows tracking metric sequences without any limitation of the number of steps tracked.
With a few thousands of steps per metric it is OK to load the entire sequence into memory
and slice it to get the desired number of points. However, when many (tens of millions) steps are tracked,
this approach cannot be applied.
Starting from version 3.11 Aim metric sequence data format has changed to improve the
data collection/sampling performance for long sequences.

What has changed? Copy Section

Each value tracked has its unique key in the Run's rocksdb database. Previously
the tracking step has been used as the part of this key. This means that in order to get
the sample of K values we have to either do K random access operations or iterate over
entire sequence and skip values. Both have negative implication on data read performance.
With Aim 3.11 the key uses the stable random hashing algorithm instead of tracking step.
Since the keys are sorted, reading first K keys is a good approximation for the entire
metric sequence.

What to do if the Metric has been logged already? Copy Section

In order to speed-up the metrics read for existing aim Repos the data format upgrade
should be executed. In cases when Run has been opened for write, the upgrade is done
automatically, since it is not possible to mix two formats.
For the rest of the cases new CLI command has been added to apply data format change
for selection of Runs or for the entire repo:

# update selected list of runs
aim storage upgrade 3.11+ <RUN_HASH_1> <RUN_HASH_2> ...

# update all runs in the repo
aim storage --repo <REPO_PATH> upgrade 3.11+ '*'

The Run data backup with old format is available in case of any issues. In order to rollback format
change and get back to the old format run the following command:

aim storage restore <RUN_HASH>

Source Understanding Remote Tracking (`docs/source/understanding/remote_tracking.md`) Copy Section

Remote Tracking Server Copy Chapter

Description

Data flow Copy Section

Code example Copy Section

Benefits Copy Section

Source Understanding Running Aim With Profiling (`docs/source/understanding/running_aim_with_profiling.md`) Copy Section

Running Aim with profiling Copy Chapter

Aim comes with profiling feature which is logging all api requests to the backend into a directory inside your
repository. We use pyinstrument as underlying profiler. To toggle
profiling, run aim up with --profiler flag.

$ aim up --profiler

This will instruct backend to create a new directory inside your repository (.aim/profler). On every api call,
profiler will create an .html file containing whole run trace of that api. Basically you can navigate into that
directory and open the file in your browser to see where's the performance bottleneck.

Why would you need to enable the profiling. Copy Section

Well you don't, unless we explicitly ask you to in case you have any performance issues while using Aim.

Please note that no data is sent to us when profiling is toggled. Everything is stored locally and managed by the end
user.

Source Understanding Storage Indexing (`docs/source/understanding/storage_indexing.md`) Copy Section

Storage indexing - how Aim data is indexed Copy Chapter

Background Copy Section

When tracking experiment metadata with Aim, each run creates its own isolated space in
aim repository. This allows to run multiple concurrent experiments without setting-up additional
services responsible for data writes synchronization. Once run is complete, all the data it
tracked is being indexed. We call this step run finalization. When the training script terminated
with SIGTERM signal, Aim will handle this and make sure that run properly finalized and data is
indexed. However, there are cases when training terminated abnormally and data remains unindexed.

How things worked before? Copy Section

Due to the chunks of data being unindexed, chunks of data would remain in the runs' separate
storage but not in index storage. This means that queries had to open multiple files to read the
repo data. Once failed runs started to accumulate, queries will slow down. In order to mitigate this
aim reindex command has been introduced.
The command will scan the aim repo and index all stalled runs.

Automatic indexing Copy Section

Though aim reindex command will address the performance issues it is not the most convenient
way to do. The questions such as "When should I run aim reindex?" or "How frequent should I run aim reindex?" depend on the actual
aim repository and use-case. Thus, we need to automate the indexing of aim repository.
Each time aim up command is ran, Aim will spawn a background thread along with the web server.
The thread will check for the unindexed runs and reindex them one at the time. This will keep
queries performance high without locking the index storage for too long.

Conclusion Copy Section

With the new automatic indexing logic in place, users don't have to manually run aim reindex
command. It is still in place for cases when all the runs data should be indexed at once. The
combination of automatic (implicit) and manual (explicit) reindexing makes sure aim repo has
good performance in a long-term usage screnarios and provides good overall user experience.

Source Using Artifacts (`docs/source/using/artifacts.md`) Copy Section

Logging artifacts with Aim Copy Chapter

One of the key aspects of experiment tracking is the ability to link certain
files/artifacts to the training run metadata. Examples of such artifacts are the
model checkpoints, training run configuration files, etc.

Starting from the version 3.19, Aim provides the logging API for artifacts, and the
UI for showing artifact's metadata.

Artifacts logging SDK Copy Section

There are only two steps for logging artifacts with Aim:

1. Set the artifacts storage URI for the Aim Run:

import aim

run = aim.Run()
# Use S3 as artifacts storage
run.set_artifacts_uri('s3://aim/artifacts/')

# Use file-system as artifacts storage
run.set_artifacts_uri('file:///home/user/aim/artifacts/')

Aim will create directory with the name of run.hash and store all artifacts there.
Note that setting artifacts storage URI is required only once per run.
2. Log artifact object:

# Log run configuration files as artifacts
run.log_artifact('config.yaml', name='run-config')

Once logged, artifact metadata will appear in the Aim UI Run details page:

Storage backends Copy Section

Training artifacts may represent large files, and, depending on the use case, might require
different storage backends. For example, in case of small runs using the local file system,
or network shared FS might be a good option. In case of the large models, cloud-based object
stores, such as AWS S3, may be a better choice.

When the artifacts URI is set, Aim will detect storage backend based on the URI scheme.
Currently supported backends for artifacts storage are.

• S3
• File System

S3 Artifacts Storage Backend

Aim uses boto3 Python package for accessing S3 resources. By default boto3 targets AWS S3 resources. Connection and credential validation is handled by boto3. A typical way of supplying credentials for instance is by setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. More details on how configuration is done
for boto3 is available here.

If you require direct control of how the boto3 client handles your s3 connection, you may use aim.storage.artifacts.s3_storage.S3ArtifactStorage_clientconfig(...). S3ArtifactStorage_clientconfig accepts any keyword-arguments that the boto3.client accepts, overwriting other means of boto3 configuration. This allows for the setting of credentials and connection details such as endpoint_url and botocore Config parameters.

import aim
from aim.storage.artifacts.s3_storage import S3ArtifactStorage_clientconfig

S3ArtifactStorage_clientconfig(aws_access_key_id=..., aws_secret_access_key=...,
                               endpoint_url=..., config={'retries': {...}, },)
run = aim.Run(...)
run.set_artifacts_uri('s3://...')
run.log_artifact(..., name=...)

File-system Artifacts Storage Backend

Aim provides ability to use mounted FS as an artifact storage. Any kind of storage that provides a mounted FS
interface can thus be used as Aim artifact storage. However, performance of the solution should be considered
when choosing this approach.

Source Using Callbacks (`docs/source/using/callbacks.md`) Copy Section

Define custom callbacks Copy Chapter

Many things can go wrong during ML training (incorrect driver versions, plateaued metrics, etc)
that could result in wasted GPUs and time.
The Aim callbacks API helps to define custom callbacks to be executed at any point during
ML training - a programmable way to guard ML training from wasting resources.

Callbacks can actually encompass any programmable functionality, such as
logging messages and sending notifications, or
killing the training process when the given condition is met.

Callbacks Copy Section

Terms:

• callback - python function that implements a custom logic to be executed at a certain point during the training.
• callbacks class - python class to group callback functions. Can be used to share state between different callbacks (example below).
• event - represents an event to be bound to the training.

The callbacks API:

• TrainingFlow - defines the training flow/events.
• events.on.* - decorators to define when the callback function is executed.

The list of currently available training events:

• events.on.training_started - called after the training start.
• events.on.training_successfully_finished - called after the training is successfully finished, meaning no unexpected exceptions are raised, even a manual keyboard interruption (ctrl+C). Please note that programmatic early stopping is considered as a successful finish.
• events.on.training_metrics_collected - called after the training metrics are calculated and ready to be logged. Typically called at each N batches.
• events.on.validation_metrics_collected - called after the validation metrics are calculated and ready to be logged. Mostly called only once, after the validation loop is finished.
• events.on.init - automatically called after the callbacks class initialization and before all the other events. Must not be called manually. Typical use-case can be initializing a shared state for callback functions (example below).

Example Copy Section

The below example demonstrates how to implement custom callbacks to check and notify, when:

• wrong driver versions are installed.
• gnorm metrics explode.
• model starts to overfit.

Defining the callbacks

/* Detailed source-code truncated for AI context efficiency. */

Registering the callbacks

from aim import TrainingFlow, Run

aim_run = Run()

training_flow = TrainingFlow(run=aim_run, callbacks=[MyCallbacks()])
# or
training_flow = TrainingFlow(run=aim_run)
training_flow.register(MyCallbacks())

Source Using Configure Runs (`docs/source/using/configure_runs.md`) Copy Section

Configure runs Copy Chapter

You can configure Aim run instance to change the default repository location, retrieve a particular run instance, or
store custom parameters for your run. You can configure your Run upon instantiation.

Here you can see all available options you can use to configure your Run instance.

• run_hash: Run's hash. Can be used to restore/retrieve a particular run form the repo. This is handy if you want to
  update something or add new metrics in your already existing run. If skipped, the hash will be generated
  automatically. Example covered in Reusing Run in your Repo section.
  Starting from Aim version 3.13, run_hash values restricted to the auto-generated values only. This
  is done to prevent multiple undesired cases, such as, creating a new Run instead of re-opening
  the existing one due to the typo, putting special symbols in Run.hash, etc. If the wrong run_hash is
  specified, MissingRunError will be raised.

• repo: Aim repository path or Repo object to which Run object is bound. If skipped, default Repo is used. More
  covered in Defining custom location for your Repo section.

• read_only: Run creation mode. Default is False, meaning Run object can be used to track metrics.

• experiment: Sets Run's experiment property. 'default' if not specified. Can be used later to query runs/sequences.

• system_tracking_interval: Sets the tracking interval in seconds for system usage metrics (CPU, Memory, etc.). Set
  to None to disable system metrics tracking.

• log_system_params: Enable/Disable logging of system params such as installed packages, git info, environment
  variables, etc. Checkout Training Run Reproducibility section for more info.

Reusing Run in your Repo Copy Section

Sometimes you might want to add/update parameters and tracked values in your existing Run instance. This can be achieved
trough using run_hash parameter. Aim generates a unique id (uuid) for your Run instance only if run_hash parameter
is not supplied. Example of re-using generated run.

from aim import Run

# You can also use ID shown in Aim Web after 
# creating a run without supplying run_hash parameter
uid = '508c5b29-02c7-4875-a157-f099ea193bfa'
run = Run(run_hash=uid)
for i in range(100):
    run.track(i, step=i, name='test')

In the example above, no matter how often this piece of code will be executed, only a single run entry will be created
or updated.

Defining custom location for your Repo Copy Section

When you use aim, the library automatically creates .aim directory which we call "Repo" in your current directory. Aim
stores data about your runs inside the Repo. Sometimes creating a Repo in your current directory is not convenient due
to various reasons (like lack of permissions), and in such scenarios you can define your own path where a new .aim
directory will be created or just reused.

from aim import Run

run = Run(repo='~/repo')
...

The code above will create a new dir entry ~/repo/.aim

Organizing Runs in Experiments Copy Section

Aim allows you to group runs under experiments, which can be useful for comparing runs intended to tackle a particular
task.

from aim import Run

run = Run(experiment="fraud-detection")

Adding tags and params to Run Copy Section

Aim lets you add tags and parameters on your Run object. Example of this would be if you want to tag your run with a
specific version or keyword. Or you might need to log parameters, so you have some insight on what params have been used
to process that particular Run.

Example of adding and removing tags

from aim import Run

run = Run()
run.add_tag('v1.0')
run.add_tag('some-awesome-tag')

Or you can modify tags on your existing run, but first you have to restore Run object by using it's hash value

from aim import Run

uid = '508c5b29-02c7-4875-a157-f099ea193bfa'
run = Run(run_hash=uid)
run.remove_tag('some-awesome-tag')
run.add_tag('another-awesome-tag')

Almost same approach goes for parameters

from aim import Run

run = Run()
var = 'some value'

run['var'] = var

Also, you can restore the Run object using its hash key and overwrite value of the previously stored parameter.

from aim import Run

uid = '508c5b29-02c7-4875-a157-f099ea193bfa'
run = Run(run_hash=uid)
var = 'another value'

run['var'] = var

Training Run Reproducibility Copy Section

When running multiple training jobs it is crucial to understand the factors affecting the trained models performance.
While the hyperparameters diff is an obvious place to look at, the training script environment itself can change the
collected metadata in unexpected ways. It is important to be able to reproduce your runs’ environment to presumably get
the same results. Sometimes even minor version change in your script dependencies or a small tweak in the training code
itself can affect training results. Thus, it's important to collect and log information such as package versions,
environment variables, input arguments, etc. with each run.

Doing this manually requires a lot of code to be added to your training script. This is where Aim's logging of system
parameters can come in handy!

What data is logged automatically?

Aim lets you enable system params logging for your Run which in result will log the following parameters

• Environment Variables
• Executables
• CLI arguments
• Installed packages and their versions
• Git information such as current branch, commit hash, author, etc. (if applicable)

How to enable system params automatic logging?

To enable logging of the parameters listed above, your Run instance must be supplied with
log_system_params=True option, by default it is disabled!

run = Run(log_system_params=True)

In addition, these logged parameters can be used in the search box to filter runs based on the supplied parameters.
Everything is searchable at Aim!

Here is an example of what you can do with it:

run.__system_params.git_info.branch == 'feature/testing'

Capturing Terminal logs Copy Section

When it comes to automating training of multiple runs with job schedulers or workload managers on a cluster,
it becomes hard to track the terminal logs of the runs. This includes the stdout, stderr from the training nodes too.

Inspecting these terminal logs is important, as it may convey crucial information about the training, for example:

• warnings, alerts and errors about the training process,
• debugging information,
• model training statistics, model architecture or summary.

Aim makes it is easier to analyze the terminal logs in one place by streaming them to Aim UI near-real-time.

Terminal logs are captured by default. In order to disable it, set the capture_terminal_logs argument to False, when initializing the Run:

aim_run = Run(capture_terminal_logs=False)

Check out how to view captured logs on Aim UI here.

Source Using Integration Guides (`docs/source/using/integration_guides.md`) Copy Section

Integration guides Copy Chapter

Aim integrates seamlessly with your favorite ML frameworks - Pytorch Ignite, Pytorch Lightning, Hugging Face and others.
Basic integration guides can be found at Quick Start section.

In this section we're going to deep-dive into the ways we can extend the basic loggers, manipulate them to track a lot more. The basic loggers can track specific metrics and hyper-params only.

There are two ways Aim callbacks/adapters/loggers can be extended:

• by deriving and overriding the main methods that are responsible for logging.
• by using public property called experiment which gives access to underlying aim.Run object to easily track new metrics, params and other metadata that would benefit your project.

Pytorch Ignite Copy Section

Both callback extension mechanisms are available with Pytorch Ignite.
In the example below you'll see how to use the experiment property to track confusion matrix as an image using aim.Image after the training is completed.

Here is an example colab notebook.

from aim import Image
from aim.pytorch_ignite import AimLogger

import matplotlib.pyplot as plt
import seaborn as sns

# Create a logger
aim_logger = AimLogger()
...
@trainer.on(Events.COMPLETED)
def log_confusion_matrix(trainer):
    metrics = val_evaluator.state.metrics
    cm = metrics['cm']
    cm = cm.numpy()
    cm = cm.astype(int)
    classes = ['T-shirt/top','Trouser','Pullover','Dress','Coat','Sandal','Shirt','Sneaker','Bag','Ankle Boot']
    fig, ax = plt.subplots(figsize=(10,10))  
    ax= plt.subplot()
    sns.heatmap(cm, annot=True, ax = ax,fmt="d")
    # labels, title and ticks
    ax.set_xlabel('Predicted labels')
    ax.set_ylabel('True labels') 
    ax.set_title('Confusion Matrix') 
    ax.xaxis.set_ticklabels(classes,rotation=90)
    ax.yaxis.set_ticklabels(classes,rotation=0)
    aim_logger.experiment.track(Image(fig), name='cm_training_end')

With Pytorch Ignite there's also a 3rd approach to extend the integration.
For example Pytorch Ignite's Tensorboard logger provides a possibility to track model's gradients and weights as histograms.
Same can be achieved with Aim

from typing import Optional, Union

import torch.nn as nn
from ignite.contrib.handlers.base_logger import BaseWeightsHistHandler
from ignite.engine import Engine, Events

from aim.pytorch_ignite import AimLogger
from aim import Distribution


class AimGradsHistHandler(BaseWeightsHistHandler):
    def __init__(self, model: nn.Module, tag: Optional[str] = None):
        super(GradsHistHandler, self).__init__(model, tag=tag)

    def __call__(self, engine: Engine, logger: AimLogger, event_name: Union[str, Events]) -> None:
        global_step = engine.state.get_event_attrib_value(event_name)
        context = {'subset': self.tag} if self.tag else {}
        for name, p in self.model.named_parameters():
            if p.grad is None:
                continue
            name = name.replace(".", "/")
            logger.experiment.track(
                Distribution(p.grad.detach().cpu().numpy()),
                name=name,
                step=global_step, 
                context=context
            )

# Create a logger
aim_logger = AimLogger()

# Attach the logger to the trainer to log model's weights norm after each iteration
aim_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=AimGradsHistHandler(model)
)

Pytorch Lightning Copy Section

In the example provided in the Aim GitHub repo using PL + Aim there's already a reference how to customize an integration.

def test_step(self, batch, batch_idx):
        ...
        # Track metrics manually
        self.logger.experiment.track(1, name='manually_tracked_metric')

So you can track lots of metadata at each iteration of test step: images, texts, whatever is needed by you and supported by Aim.

Hugging Face Copy Section

Here is how to extend the basic Hugging Face logger.
Below is an example of a CustomCallback that's derived from the AimCallback. The main HF method here is the on_log() that's overriden.

This allows us to track any str object that is passed to on_log() method as aim.Text.

from aim.hugging_face import AimCallback
from aim import Text


class CustomCallback(AimCallback):
    def on_log(self, args, state, control,
               model=None, logs=None, **kwargs):
        super().on_log(args, state, control, model, logs, **kwargs)

        context = {
            'subset': self._current_shift,
        }
        for log_name, log_value in logs.items():
            if isinstance(log_value, str):
                self.experiment.track(Text(log_value), name=log_name, context=context)

TF/keras Copy Section

Here is how to track confusion matrices with Aim while extending the default callback provided for tf.keras.
We have taken and adapted this example. to Aim. Here is how it looks:

from aim.tensorflow import AimCallback

class CustomImageTrackingCallback(AimCallback):
    def __init__(self, data):
        super().__init__()
        self.data = data

    def on_epoch_end(self, epoch, logs=None):
        super().on_epoch_end(epoch, logs)
        from aim import Image
        # Use the model to predict the values from the validation dataset.
        test_pred_raw = self.model.predict(test_images)
        test_pred = np.argmax(test_pred_raw, axis=1)

        # Calculate the confusion matrix.
        cm = sklearn.metrics.confusion_matrix(test_labels, test_pred)
        # Log the confusion matrix as an image summary.
        figure = plot_confusion_matrix(cm, class_names=class_names)
        cm_image = Image(figure)

        # Log the confusion matrix as an Aim image.
        self.experiment.track(cm_image,"Confusion Matrix", step=epoch)

aim_callback = CustomImageTrackingCallback()

model.fit(
    train_images,
    train_labels,
    epochs=5,
    verbose=0, # Suppress chatty output
    callbacks=[aim_callback],
    validation_data=(test_images, test_labels),
)

XGBoost Copy Section

Here is how to override the AimCallback for XGBoost.

from aim import Text
from aim.xgboost import AimCallback


class CustomCallback(AimCallback):

    def after_iteration(self, model, epoch, evals_log):
        for data, metric in evals_log.items():
            for metric_name, log in metric.items():
                self.experiment.track(Text(log), name=metric_name)
        
        return super().after_iteration(model, epoch, evals_log)

Catboost Copy Section

Catboost's .fit() has log_cout parameter which can be used to redirect log output into a custom object
which has write attribute. Our logger is an object which implements write method to parse log string according to
its content. Thus, most of the log output will be ignored by our parser logic, but you still can code up your own logic
on top of ours to fill the gap for your needs.

from aim.catboost import AimLogger


class CustomLogger(AimLogger):

    def write(self, log):
        # Process the log string through our parser
        super().write(log)

        # Do your own parsing
        log = log.strip().split()
        if log[1] == 'bin:':
            value_bin = log[1][4:]
            value_score = self._to_number(log[3])
            self.experiment.track(value_score, name='score')

LightGBM Copy Section

Here is how to override the AimCallback for LightGBM.

from aim.lightgbm import AimCallback


class CustomCallback(AimCallback):

    def before_tracking(self, env):
        for item in env.evaluation_result_list:
            # manipulate item here
            pass

    def after_tracking(self, env):
        # do any other action if necessary after tracking value
        pass

Source Using Jupyter Notebook Ui (`docs/source/using/jupyter_notebook_ui.md`) Copy Section

Run Aim UI on Jupyter Notebook Copy Chapter

Start your notebook with the following code to install Aim:

!pip install aim

Next, initialize a new run and save some hyperparameters:

from aim import Run

run = Run()

run['hparams'] = {
    'learning_rate': 0.001,
    'batch_size': 32,
}

.. note::
Do not forget to call run.finalize() once the training is over.

After tracking runs with Run,
run the following commands in the notebook to run the Aim UI:

1. Load Aim extension for notebooks:

%load_ext aim

2. Run %aim up to open Aim UI in the notebook:

%aim up

Source Using K8s Deployment (`docs/source/using/k8s_deployment.md`) Copy Section

Host Aim on Kubernetes (K8S) Copy Chapter

Since Aim can run as a local server through FastAPI, it can be deployed to a K8S cluster! Hosting Aim on K8S comes with
several advantages:

• multiple users of your organization can access Aim in a single spot, which removes the need for ML practitioners to
  run Aim themselves
• Aim runs can be centralized on a remote volume, which provides additional support and encouragement for remote model
  training and monitoring
• a deployment to K8S abstracts away the Aim CLI, which empowers users to focus on the value provided by Aim (
  visualizations/applications vs. CLI up and repo understanding)

The following sections illustrate how to deploy and serve Aim on K8S. The sections assume:

• access to a cloud provider, such as GCP, AWS, or Azure
• a repository that can host Dockerfiles, such as Google Artifact Registry or Dockerhub
• ability/permissions to provision a ReadWriteMany volume, or bind an existing one to a K8S deployment

Dockerfile Copy Section

The following Dockerfile image should suffice for getting Aim running in a container:

# python3.7 should be sufficient to run Aim
FROM python:3.7

# install the `aim` package on the latest version
RUN pip install --upgrade aim

# make a directory where the Aim repo will be initialized, `/aim`
RUN mkdir /aim

ENTRYPOINT ["/bin/sh", "-c"]

# have to run `aim init` in the directory that stores aim data for 
# otherwise `aim up` will prompt for confirmation to create the directory itself.
# We run aim listening on 0.0.0.0 to expose all ports. Also, we run 
# using `--dev` to print verbose logs. Port 43800 is the default port of 
# `aim up` but explicit is better than implicit.
CMD ["echo \"N\" | aim init --repo /aim && aim up --host 0.0.0.0 --port 43800 --workers 2 --repo /aim"]

Assuming you store the above in your current directory, the container can be built
using docker build . -t my-aim-container:1 and pushed to your repository
with docker push my-docker-repository.dev/deployments/aim:1.

Volume Copy Section

The core advantage of using a K8S volume to store Aim runs is that other K8S deployments can mount the same volume and
store their runs on it! This way, the core Aim K8S deployment can read the new runs and display them to users who want
to visualize their results. For example, one can have a deployment that performs model training and records Aim runs on
the same volume that is mounted to the Aim deployment! This model is illustrated by the following diagram:

Generally, volumes that support have the ReadWriteMany property are manually provisioned, such as Filestore instances
on Google Cloud or, generally, GlusterFS volumes. Once a disk is provisioned, it can be bound to a persistent volume via
an IP. Assuming you can provision a disk like this on your cloud provider and obtain an IP, we can create a volume
representation, along with a claim for it. The persistent volume (aim-pv.yaml) can be formulated as:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: aim-runs
spec:
  capacity:
    storage: 1Ti # or whatever size disk you provisioned
  accessModes:
    - ReadWriteMany
  nfs:
    path: /aim
    server: 123.12.123.12  # add your own IP here

The persistent volume claim (aim-pvc.yaml) is:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: aim-runs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "" # if you have a custom storage class, use it! Otherwise, it's `default`
  volumeName: aim-runs
  resources:
    requests:
      storage: 1Ti

These can be provisioned via:

> kubectl apply -f aim-pv.yaml
> kubectl apply -f aim-pvc.yaml

Once the volume is provisioned, we can mount it to our deployments!

Deployment Copy Section

The main Aim deployment will have a single container that runs Aim. This deployment will mount the volume that was
provisioned previously, and the main Aim repository will be initialized at the path the volume is mounted to. For
example, if the volume is mounted to /aim, then the deployment will initialize and read Aim runs from that path. The
K8S deployment is:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-aim-deployment
  name: my-aim-deployment
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-aim-deployment
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: my-aim-deployment
    spec:
      containers:
        image: my-docker-repository.dev/deployments/aim:1
        name: my-aim-deployment
        ports:
          - containerPort: 43800
            protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
        volumeMounts:
          - mountPath: /aim
            name: aim-runs
      volumes:
        - name: aim-runs
          persistentVolumeClaim:
            claimName: aim-runs-claim

This K8S deployment:

• defines a pod with a single replica that runs the Aim server defined by the Dockerfile
• mounts the persistent volume aim-run through the aim-run-claim persistent volume claim
• the Dockerfile initializes the /aim directory as the Aim repo. Note that the Dockerfile already passes N to the
  confirmation prompt in case the repo is already initialized (this will be the case after the initial deployment
  creation, since the repo has to be initialized only once, but it's nice to avoid manual work)
• starts up the Aim server on port 43800, which reads all the runs stored at /aim

Service Copy Section

Now that a deployment is deployed, the Aim server can be exposed through a K8S service! Depending on your cluster setup,
you have several options for exposing the deployment. One option is to run:

> kubectl expose deployment my-aim-deployment --type=LoadBalancer --name=my-aim-service

Another alternative is to create the service definition yourself, and apply it. The definition (aim-svc.yaml) can be:

apiVersion: v1
kind: Service
metadata:
  name: my-aim-service
spec:
  selector:
    app: my-aim-deployment
  ports:
    - protocol: TCP
      port: 80
      targetPort: 43800

The service definition can be applied via:

> kubectl apply -f aim-svc.yaml

Conclusion Copy Section

That's it! Now you have the following structure serving your users' Aim runs:

Assuming your users can submit a model training run to some pod/deployment that runs model training and has the
right aim code to record a run at path /aim, your Aim deployment will be able to display the run the next time it
performs a live update!

Source Using Logging (`docs/source/using/logging.md`) Copy Section

Log messages during training process Copy Chapter

During the training many key events may happen. Use Aim logging API to log any text message.

Additionally, set up the notification service to automatically send notifications,
when a new message is logged.

The logging API Copy Section

Aim SDK provides an interface to log any string message in the “.aim” repo.

• Run.log_debug(msg) - aim.Run method to log a message with level DEBUG.
• Run.log_info(msg) - aim.Run method to log a message with level INFO.
• Run.log_warning(msg) - aim.Run method to log a message with level WARNING.
• Run.log_error(msg) - aim.Run method to log a message with level ERROR.

Notifying on logged messages Copy Section

Aim notification service constantly checks for new logged messages and sends notifications
to the enabled channels. It is also possible to
configure the level of notifications to be sent.

See how to configure notifiers here.

Source Using Manage Runs (`docs/source/using/manage_runs.md`) Copy Section

Manage runs Copy Chapter

Create runs Copy Section

Run is the main object that tracks and stores ML training metadata(e.g. metrics or hyperparams).

When initializing a Run object, Aim creates a .aim repository at the specified path.
The tracked data is stored in the .aim repo.
If the repo path is not specified, the run data is stored in the current working directory.

Use the following Run arguments to:

• repo: define where to store the data
• experiment: define experiment name to group related runs together
• system_tracking_interval: Enable system resource usage tracking (CPU, GPU, memory, etc..). By default enabled. Set to None to disable
• capture_terminal_logs: Enable process output capturing. By default enabled. Set to False to disable.

from aim import Run

my_run = Run(
    repo='/repo/path/to/store/runs',
    experiment='experiment_name'
)

Run class full spec.

Additionally, Aim SDK also gives a flexibility to:

• Use multiple Runs in one training script to store multiple runs at once. Usually handy when doing hyperparameter search.
• Use integrations to automate tracking

Continue runs Copy Section

Each Run object has a hash associated with it which could be looked up at aim runs ls (check out the Aim CLI here).
Specify the run hash when initializing a Run object to continue tracking.

from aim import Run

run = Run(run_hash='run_hash')

Delete runs Copy Section

There are cases when Run data is not needed. Examples of such cases are,
failed training runs or simple disk space cleanup. Aim provides SDK and CLI interfaces
to delete Runs.

To remove Runs via the SDK:

from aim import Repo

repo = Repo.from_path('aim_repo_path')
repo.delete_run('run_hash')
repo.delete_runs(['run_hash_1', 'run_hash_2'])

Repo class full spec.

To remove Runs using command line:

aim runs rm run_hash_1 run_hash_2 run_hash_3

Cleanup (prune) run params and metric names/contexts

Due to the Aim storage structure, after runs are deleted their params and metric names/contexts
are still available for autocomplete in Aim UI explorers.

In order to clean up those dangling params and metric properties using command line:

aim storage prune

Upload runs Copy Section

Aim provides a way to create backup snapshots of a .aim repository in AWS s3.
The snapshot will be created in an s3 bucket. The bucket name is passed as an argument.
A new s3 bucket will be created if it does not exist already.

Please note that boto3 package and sufficient AWS permissions are required for this command.

The details of how to configure boto3 credentials can be found here.

To create a snapshot using command line:

aim runs upload bucket_name_1

More details on aim runs in CLI reference.

Source Using Notifications (`docs/source/using/notifications.md`) Copy Section

Set up the notification service Copy Chapter

Being notified about key events of the training process may save hours of compute resources and researcher's time.
Aim notification service sends notifications to the enabled channels (slack, workplace), when a new message is logged
or a stuck run is detected.

Quick start Copy Section

These are the required steps in order to set up the notifications service.

1. Configure notifiers. The notifier configuration is described in detail in Configuring Notifiers section.

2. Start notifier service:

   shell Copy

   aim-watcher --repo <AIM_REPO_PATH> start

Aim notifications service is comprised of the following components:

• Aim watcher service. A service responsible for sending notifications.
• Notifier configuration CLI. Command line utility for configuring where and how you want to
  receive notifications about training run status.

Below are the components described in more detail.

Configuring notifiers Copy Section

Use aim-watcher CLI to interactively choose and setup the notifiers.
Currently, following types of notifiers are supported:

• Slack: to receive notifications as a message on a slack channel.
• Workplace: to receive notifications as a post in a workplace group.
• Logger: to echo notification text to watcher standard output.

The base command is:

aim-watcher notifiers add

Once notifier type is selected, the watcher will ask to fill-in the required inputs for that
particular notifier.

aim-watcher allows configuring multiple notifiers in case you'd like to receive them in
multiple channels at the same time. Once notifier configuration added, aim-watcher should be re-run, so
new configuration can take effect.

Next sections will describe the setup flow for each of them.

Configuring notifier for slack

In order to execute the steps described below you must have admin permissions for Slack workspace.

Here are the steps to configure Aim notifier for Slack.

1. Go to Slack API web page and push "Create an app" button.

2. Select "From scratch" and follow the steps.

3. Make sure Incoming Webhooks are active. This is the Slack API Aim will use to send
   notifications.

4. Create a new Webhook URL and copy to clipboard.

5. Run aim-watcher notifiers add slack and once prompted for Webhook url, paste the one copied
   in step 4.

6. Optional Change the training run failure message template.

7. Confirm setup.

Configuring notifier for workplace

In order to execute steps described below, please make sure you have access to Workplace Admin panel

Here are the steps to configure Aim notifier for Workplace.

1. Go to "Your workplace" Admin Panel

2. Select "Integrations" tab and press "Create custom integration"

3. Follow the setup steps.

4. In the "Permissions" tab make sure "Manage group content" checkbox checked.

5. Select the group(s) where you want to receive notifications.

6. In the "Details" tab press "Create access token" and copy the generated access token.

7. Run aim-watcher notifiers add workplace and provide group ID and copied access token.

8. Optional Change the training run failure message template.

9. Confirm setup.

Enabling/disabling notifier configuration

Aim watcher CLI allows to temporarily disable or completely remove notifiers. Each notifier
configuration has unique auto-generated ID which can be used to manipulate the notifier.
In order to list available notifiers for the aim Repo:

aim-watcher --repo . notifiers list

> NOTIFIER ID                              TYPE       STATUS    
> 88bbacb3-159d-4ee2-8f33-180addd162f1     slack      enabled   
> 75fe529d-c060-4816-9cbb-246770cb8592     slack      disabled  
> 248821e4-2bf3-4aa9-857e-194bf3ab2f8a     logger     enabled

You can disable notifier by given ID:

aim-watcher --repo . notifiers disable 88bbacb3-159d-4ee2-8f33-180addd162f1

and enable it back:

aim-watcher --repo . notifiers enable 88bbacb3-159d-4ee2-8f33-180addd162f1

In order to remove notifier configuration completely:

aim-watcher --repo . notifiers remove 88bbacb3-159d-4ee2-8f33-180addd162f1

Complete reference for aim-watcher CLI is available here.

Configuring notification levels

The notification levels to notify on logged messages can be configured.
Available options are DEBUG, INFO, WARNING, ERROR.
Run aim-watcher notifiers set-log-level [LVL] command to set the level.

.. note::
WARNING is set as a default level.

Source Using Query Runs (`docs/source/using/query_runs.md`) Copy Section

Query runs and objects Copy Section

Use Repo object to query and access saved Runs.

Initialize a Repo instance:

from aim import Repo

my_repo = Repo('/path/to/aim/repo')

Repo class full spec.

Query logged metrics and parameters:

query = "metric.name == 'loss'" # Example query

# Get collection of metrics
for run_metrics_collection in my_repo.query_metrics(query).iter_runs():
    for metric in run_metrics_collection:
        # Get run params
        params = metric.run[...]
        # Get metric values
        steps, metric_values = metric.values.sparse_numpy()

Besides querying Runs and metrics, you can also query logged Image objects:

query = "images.name == 'mnist_dataset' and images.context.subset == 'val"

# Get collection of Image sequences
for image_seq in my_repo.query_images(query).iter():
    # Get first tracked batch of each sequence
    image_batch = image_seq.values.first_value()
    # Get Image metadata
    image_meta = map(Image.json, image_batch)
    # Convert to PILImage
    pil_images = map(Image.to_pil_image, image_batch)

Image class full spec.

See more advanced usage examples here.

Source Using Remote Tracking (`docs/source/using/remote_tracking.md`) Copy Section

Track experiments with aim remote server Copy Chapter

Overview Copy Section

Aim remote tracking server allows running experiments in a multi-host environment and collect tracked data in a centralized location.
It provides SDK for client-server communications and utilizes http/ws protocols as its core transport layer.

In this guide we will show you how to set up Aim remote tracking server and how to integrate it in client-side code.

Prerequisites Copy Section

Remote tracking server assumes multi-host environments used to run multiple training experiments.
The machine running the server have to accept incoming http traffic on a dedicated port (default is 53800).

Server-side setup Copy Section

1. Make sure aim 3.4.0 or upper installed:

$ pip install "aim>=3.4.0"

2. Initialize aim repository (optional):

$ aim init

3. Run aim server with dedicated aim repository:

$ aim server --repo <REPO_PATH>

You will see the following output:

> Server is mounted on 0.0.0.0:53800
> Press Ctrl+C to exit

The server is up and ready to accept tracked data.

4. Run aim UI

$ aim up --repo <REPO_PATH>

Client-side setup Copy Section

With the current architecture there is almost no change in aim SDK usage. The only difference from tracking locally is
that you have to provide the remote tracking URL instead of local aim repo path. The following code shows how to create
Run with remote tracking URL and how to use it.

from aim import Run

aim_run = Run(repo='aim://172.3.66.145:53800')  # replace example IP with your tracking server IP/hostname

# Log run parameters
aim_run['params'] = {
    'learning_rate': 0.001,
    'batch_size': 32,
}
...

You are now ready to use aim_run object to track your experiment results. Below is the full example using
pytorch + aim remote tracking on MNIST dataset.

/* Detailed source-code truncated for AI context efficiency. */

SSL support Copy Section

Aim Remote Tracking server can be set up to use SSL certificates to run on secure mode.
To enable secure mode for server provide --ssl-keyfile and --ssl-certfile arguments to aim server command,
where --ssl-keyfile is the path to the private key file for the certificate and --ssl-certfile is the path to the signed certificate
(check out the Aim CLI here).

If you're using self-signed certificates the client has to be configured accordingly, otherwise the client will automatically detect if the server supports secure connection or not.
Please set AIM_CLIENT_SSL_CERTIFICATES_FILE environment variable to the file, where PEM encoded root certificates are located, e.g.

export __AIM_CLIENT_SSL_CERTIFICATES_FILE__=/path/of/the/certs/file

Note:
For the sake of convenience of providing only one file to the client, the private key will be taken from certfile as well.
The following example will do the trick:

# generate the cert and key files
openssl genrsa -out server.key 2048
openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650 -subj '/CN={DOMAIN_NAME}'
# append the private key to the certs in a new file
cat server.crt server.key > server.includesprivatekey.pem
# set the env variable for aim client
export __AIM_CLIENT_SSL_CERTIFICATES_FILE__=./server.includesprivatekey.pem

Conclusion Copy Section

As you can see, aim remote tracking server allows running experiments on multiple hosts with simple setup and
minimal changes to your training code.

Source Using Sagemaker Notebook Ui (`docs/source/using/sagemaker_notebook_ui.md`) Copy Section

Run Aim UI on SageMaker Notebook instance Copy Chapter

In this guide you will learn how to run Aim UI on your Sagemaker Jupyter instance

Aim can be installed and used on Jupyter notebooks. With SageMaker notebook instance, there are difficulties with establishing POST requests which prevents some Aim functionalities to work.
So we have built a script to update the proxying package on SageMaker notebook instance.
In order to work with Aim UI, please follow the steps below:

• Create a lifecycle configuration on your SageMaker Service.
• Copy and paste the script in your configuration's Start Notebook phase.

For more information how to create a lifecycle configuration on AWS SageMaker Service, please go through the AWS SageMaker Documentation.

After creating a lifecycle configuration, attach it to your SageMaker instance and restart the instance.

It is possible to run this script manually from the terminal of your jupyter instance as well.

.. Note::
Why this script is used for?

The script above will uptade the `jupyter-server-proxy` package on your notebook instance.
To access a port on your instance (i.e. open the Aim UI) API requests and static files need to go through the proxy, which is created by the package (see the script).
It will replace the existing proxy and make sure Aim and other such requests go through. 
Fore more information please go through the https://github.com/jupyterhub/jupyter-server-proxy/pull/328#issue-1145874348.

Once your notebook instance is successfully restarted you will be able to use Aim UI on the instance.

Next, install aim.

$ pip install aim

Initialize a new run and save some hyperparameters.

from aim import Run

run = Run()

run['hparams'] = {
    'learning_rate': 0.001,
    'batch_size': 32,
}

Using Terminal Copy Section

In order to run Aim UI from a terminal, execute the following command:

$ aim up --base-path=/proxy/absolute/<your-port>/aim-sage

After running this command you will be able to open <sagemker_instance>/proxy/absolute/<your-port>/aim-sage/ in your browser.
The default port is 43800.

It is possible to set __AIM_PROXY_URL__ env variable, and aim up command will print out the generated url for Aim UI.

To find your proxy url, just copy your SageMaker URL and remove /lab postfix.

Using Notebook Extension Copy Section

1. Load Aim extension for notebooks:

%load_ext aim

2. Run %aim up command to open Aim UI in the notebook:

%aim up --proxy-url=https://<instance>.notebook.<region>.sagemaker.aws

Will load the Aim UI in the notebook cell.

Will print out the proxy url under the loaded Aim UI. Use that URL to open Aim UI on your browser.

The default port is 43801 for notebook extension to prevent confusions.
The --port=<your-port> argument is supported as well.

.. note::
In notebook extension, the only mandatory argument is --proxy-url when using it on SageMaker.

Source Using Search (`docs/source/using/search.md`) Copy Section

Query language basics Copy Chapter

Introduction Copy Section

Aim enables a powerful query language(Aim QL) to filter through all the stored metadata.

AimQL filters the tracked metadata using python expression.
Think of it as a python if statement over everything you have tracked.
Hence, nearly any python compatible expression is available with some restrictions in place.

The data is saved as different types of entities (e.g. run, metric). The search queries are written against these entities.
When iterating over entities the python expression is evaluated in a Boolean context. When the value is "truthy", then the current entity is yielded. Otherwise the entity is skipped over.

.. note::
Currently, AimQL is only used for filtering data, and has no role in sorting or aggregating the data.

Searching Copy Section

Let's track several Runs via Aim SDK:

from aim import Run
# Initialize run_1
# Define its params and track loss metric within test and train contexts
run_1 = Run()
run_1['learning_rate'] = 0.001
run_1['batch_size'] = 32
for i in range(10):
    run_1.track(i, name='loss', context={ 'subset':'train' })
    run_1.track(i, name='loss', context={ 'subset':'test' })

# Initialize run_2
run_2 = Run()
run_2['learning_rate'] = 0.0007
run_2['batch_size'] = 64
for i in range(20):
   run_2.track(i, name='loss', context={ 'subset':'train' })
   run_2.track(i, name='loss', context={ 'subset':'test' })
   run_2.track(i/100, name='accuracy')

# Initialize run_3
run_3 = Run()
run_3['learning_rate'] = 0.005
run_3['batch_size'] = 16
for i in range(30):
   run_3.track(i, name='loss', context={ 'subset':'train' })
   run_3.track(i, name='loss', context={ 'subset':'test' })
   run_3.track(i/100, name='accuracy')

Aim SDK will collect and store the above metadata in .aim repo.

|

|
| run_2 <hash=a32c911> |

|

|
| run_3 <hash=a32c912> |

|

|

When searching runs, use the run keyword which represents the Run object. It has the following properties:

Run parameters could be accessed both via chained properties and attributes.

.. note::

The three following examples are functionally equal:

• run.hparams.learning_rate == 32 (recommended)
• run["hparams", "learning_rate"] == 32 (recommended)
• run["hparams"]["learning_rate"] == 32

.. warning::
AimQL has been designed to be highly performant.
Only the params that are used in the query will be loaded into memory.

If you use the ['hparams']['learning_rate'] syntax Aim will load the whole dictionary into memory. The search performance will be impacted.

We recommend to use either ['hparams', 'learning_rate'] or hparams.learning_rate syntax which are equivalent to each other in terms of the performance.

Query examples:

1. Get runs where learning_rate is greater than 0.0001 and batch_size is greater than 32.

run.learning_rate > 0.0001 and run.batch_size > 32

Result:

|

2. Get runs where learning_rate is either 0.0001 or 0.005.

run.learning_rate in [0.0001, 0.005]

Result:

|
| run_3 <hash=a32c912> |

|

3. Search runs based on metrics' last values.

metrics property takes 2 arguments: metric name and context (name is required, context is empty({}) by default).

a. Get runs where accuracy, {} last value is greater than 0.25

run.metrics['accuracy'].last > 0.25

Result:

|

|

b. Get runs where loss, {'subset': 'train'} last value is smaller than 25

run.metrics['accuracy', {'subset': 'train'}].last < 25

Result:

|

|
| run_2 <hash=a32c911> |

|

|

4. Search runs where duration is smaller than 10s (to filter out failed runs for example)

run.duration < 10

This will return all the runs on our test set, but will have a use case on real life experiments to filter the runs that finished under 10 seconds.
Then those can be archived/deleted as they've probably crashed or were stopped.

5. Search runs based on creation or end times

created_at and finalized_at properties are represented as python datetime objects. So any of these queries are a competent way to filter runs:

a. Get runs for a specific date

run.created_at.strftime("%d/%m/%Y") == "24/12/2021"

b. Get runs after a specific date

run.finalized_at > datetime(2022, 2, 2)

c. Get runs of a specific month

run.created_at.month == 2 and run.created_at.year == 2022

Searching metrics and images Copy Section

Searching metrics

When iterating over metrics, use the metric keyword which represents the tracked metric. While searching metrics, you can also refer to the related runs via the run keyword.

metric has the following default properties.

Query examples

1. Query metrics by name:

metric.name == "loss"

Result:

2. Query metrics by name and context

metric.name == "loss" and metric.context.subset == "train"

Result:

3. Query metrics by name and run parameters

metric.name == "loss" and run.learning_rate >= 0.001

Result:

4. Query metrics by name and last value

metric.name == "loss" and metric.last >= 15

Result:

Searching images

Images search works in the same way as metrics.
When iterating over images, use the images keyword which represents the tracked images sequence.
While searching images, you can also refer to the related runs via the run keyword.

images keyword has the following default properties.

Query examples:

• images.name == "generated" and run.learning_rate >= 0.001
• images.name == "generated" and images.context.ema == 0

Security restrictions Copy Section

AimQL expression is evaluated with RestrictedPython.

RestrictedPython is a tool that helps to define a subset of the Python language which allows to provide a program input into a trusted environment.

We have followed these restrictions to avoid security risks such as executing a non-safe function via AimQL.

Source Using Training Monitoring (`docs/source/using/training_monitoring.md`) Copy Section

Notify on failed/stuck runs Copy Chapter

Many things can cause the training process to fail or get stuck:

• hardware failure
• power outage
• programmatic error

At first, it may seem a relatively easy, however checking that everything is OK with the training run is proven
to be a non-trivial task.

In some cases the training process may not even have a chance to notify about the failure. The process
could also be stuck due to

• network issues
• filesystem I/O issues
• etc

With this in mind, the following definition of failed process is used:

The Run is considered as failed if it hasn't reported any progress in a predefined time-interval.

Once the progress is not reported for the given period of time, a notification will be sent to the enabled channels.

See how to set up the notification service and add notification channels here.

Run progress reporting SDK Copy Section

Aim SDK provides interface for reporting Run progress. The Run class has now two methods:
report_progress and report_successful_finish to report progress and successful finish respectively.
Here's a small code snippet showing how Run.report_progress() method can be integrated in the training loop.

from aim import Run

# prep dataset and model
...
aim_run = Run()

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)
    
        # Report progress assuming each iteration shoud take less than 3 sec.
        aim_run.report_progress(expect_next_in=3)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if i % 30 == 0:
            # Track data with aim
            aim_run.track(loss.item(), name='loss', epoch=epoch)

...
# Training is done, report success
aim_run.report_successful_finish()

report_progress() takes an ETA (in seconds) of the next anticipated progress report call.

Detailed description of interfaces is available in aim SDK reference.

.. note::
Additional grace period of 100s is enabled to compensate for possible hardware (e.g. filesystem) latency.

Source Ui Overview UI Basics (`docs/source/ui/overview/UI_basics.md`) Copy Section

Overview Copy Chapter

Aim enables powerful UI to explore logged ML runs and metadata.

Runs explorer Copy Section

Runs explorer will help you to holistically view all your runs, each metric last tracked values and tracked hyperparameters.

Features:

• Full Research context at hand
• Search runs by date, experiment, hash, tag or parameters
• Search by run/experiment

Metrics explorer Copy Section

Metrics explorer helps you to compare 100s of metrics within a few clicks.
It helps to save lots of time compared to other open-source experiment tracking tools.

Features:

• Easily query any metric
• Group by any parameter
• Divide into subplots
• Aggregate grouped metrics (by conf. interval, std. dev., std. err., min/max)
• Apply smoothing
• Change scale of the axes (linear or log)
• Align metrics by time, epoch or another metric

Images explorer Copy Section

Track intermediate images and search, compare them on the Images Explorer.

Features:

• Easily query any image
• Group by images by run parameters
• Group images by step

Params explorer Copy Section

Params explorer enables a parallel coordinates view for metrics and params. Very helpful when doing hyperparameter search.

Features:

• Easily query any metrics and params
• Group runs or divide into subplots
• Apply chart indicator to see correlations

Single run page Copy Section

Explore all the metadata associated with a run on the single run page.
It's accessible from all the tables and tooltips.

Features:

• See all the logged params of a run
• See all the tracked metrics(including system metrics)

Source Ui Pages Bookmarks (`docs/source/ui/pages/bookmarks.md`) Copy Section

Bookmarks Copy Chapter

Overview Copy Section

Use the Bookmarks to save the Aim Explorer state. This includes search query, aggregations and any other modifications applied to the explorer. The Bookmarks page is a list of cards to quickly access the explorer state with one click.

There are bookmark buttons available on all Explorer pages - on the top right . In order to create a bookmark, just press the bookmark button on your preferred Explorer page.

This will open a create bookmark form with fields for title and description.

The created bookmark will contain all the current configuration of the explorer.
 
 

The Bookmark Card Copy Section

Each of the bookmark cards contains the following:

• Explorer Icon
• Title
• View Bookmark button
• Delete Bookmark button
• Search query
• Selected metrics and params

Delete Bookmark Copy Section

Clicking the delete bookmark button will open a modal, where you can confirm or cancel the deletion.

Source Ui Pages Explorers (`docs/source/ui/pages/explorers.md`) Copy Section

Explorers Copy Chapter

Explorers will help you to compare 1000s of AI experiments with a few clicks.
Explorers are the main tools that Aim is built around.

In this section we will go through the Aim explorers, introduce their features and how to use them.

Metrics Explorer Copy Section

Overview

Use Metrics explorer to search and compare 1000s of ML training metrics.

The Metrics Explorer allows you to search, group and compare your metrics.
Due to this and number of other visual features on the Metrics Explorer, you will save considerable amounts of time when comparing experiments- compared to other open-source experiment tracking tools.

The Metric Explorer has the following main sections:

• Metrics Select: to select the metrics for exploration
• Search bar: to query the runs for exploration
• Charts explorer: the space where the metrics are rendered
• Metrics modifiers: all the grouping, chart division and other metrics modifier tools
• Context table: the full information about the selected metrics is available

.. note::
There is also an advanced mode of search is available too where you can use the full Aim QL (more on this further in this section).

There are two ways you can query metrics and runs

• Select metrics and query runs
• Advanced Search mode

An overview of what you can do with queried metrics - the modifiers:

• Group by tracked parameters
• Aggregate grouped metrics
• Configure axes properties (x-axis alignment, axes range)
• Change scale of the axes (linear or log)
• Apply smoothing
• Ignore outliers
• Metric highlight modes (metric on hover, run on hover)
• Set chart tooltip parameters
• Apply zoom in/out on charts
• Export chart as image

Select metrics and query runs

On the Metrics Explorer, there is + Metrics button.
Once pressed, a dropdown will appear with all your tracked metrics and their contexts flattened.
The dropdown is searchable - so you can get to your metric of interest within a stroke!

The Search bar is located below the + Metric button. It allows to do a pythonic query (that is eval-ed as python statement) over every param you have tracked.

Search runs with Aim QL

Advanced Search mode

Once you press the Enable advanced search mode button underneath the main Search button, it will enable the full Aim QL search editor - to query the metrics, the runs via full Aim QL

Here is an example:

((metric.name == 'bleu' and metric.context.subset == 'val') or (metric.name == 'loss' and metric.context.subset == 'train')) and 1e-5 < run.hparams.learning_rate < 1e-2

Group by any parameter

Grouping selected metrics by any tracked params will allow you to quickly distinguish the most impactful params, decisions you have made (the preprocessing steps, the hyperparams etc).

The parameters include not only the ones you have tracked but also the native Aim objects too such as

• metric.name
• metric.context.[context_key]
• run.hash

There are several ways you can group the selected metrics and runs - by color, by stroke and by chart.

Group by Color

Use this to divide the selected metrics into different clusters based on selected values of params.
Each cluster gets colored differently.

There are a number of options available when grouping

• group by values - divides into clusters as per the values of selected params)
• reverse grouping - divides into clusters by every param except for the chosen one.

The grouping colors are picked randomly, however it is possible to fix with the advanced coloring features.

Here are the features in the advanced mode:

• Fix the colors of the grouping
• Control the color palette to use during the grouping

Group by Stroke

Groups the metrics by a stroke style.
Has all the rest of the other features available on the color grouping except the advanced mode.

Group by Chart

The end result of using this feature: divides into subplots based on the value of the selected params.
Why this is a grouping mechanism? It groups the metrics belonging to the same group into separate charts.

Aggregate metrics

The metrics aggregation helps to quickly see the trends of each individual group of metrics. See more about metrics grouping.

There are two aspects of aggregation you can control:

• the trend-line
• the area that the group of metricsc take

The trend-line calculation methods:

• Mean
• Median
• Min
• Max

The area calcualtion methods:

• None (when you'd like to remove the area)
• Min/Max
• Mean +/- Standard Deviation
• Mean +/- Standard Error
• Confidence Interval (95%)

Pls see the screenshot:

Axes properties

Axes properties section is for aligning metrics by time, epoch or another metric and for setting axes range manually.

Alignment:

Following types of metrics alignment are available: Step, Epoch, Relative Time, Absolute Time and Custom Metric.
By default, metrics aligned by Step.

Step

By setting metrics alignment to Step, x-axis values will represent the steps of tracked metrics.

Epoch

By setting metrics alignment to Epoch, x-axis values will represent the epochs of tracked metrics.

Relative Time

By setting metrics alignment to Relative Time, x-axis values will represent by HH:mm:ss, duration of tracking process.

Absolute Time

By setting metrics alignment to Absolute Time, x-axis values will represent by date HH:mm:ss D MMM, YY, since the start date of the first run until the last run.

Custom Metric

By setting metrics alignment to Custom Metric, x-axis values will represent selected metric values, you can detect correlations between queried metrics and selected metric.

Set axes range:

To fix an axis range across all the charts, set the corresponding axis minimum and maximum bounds in the form.

Axes Scale

Axes Scale section gives ability to display axes scale's linear or logarithmic.

By default, axes scale's are Linear.

Linear Scale

X-axis scale: Linear, Y-axis scale: Log

X-axis scale: Log, Y-axis scale: Linear

Log Scale

Chart Smoothing

While smoothing the chart, the data points are modified so individual points higher than the adjacent points (presumably because of noise) are reduced, and points that are lower than the adjacent points are increased leading to a smoother signal.
You can select curve interpolation methods: Linear or Cubic.
By default, chart smoothing in Exponential moving average mode and curve interpolation method is Linear.

Exponential moving average

An exponential moving average, also known as an exponentially weighted moving average (EWMA),
is a first-order infinite impulse response filter that applies weighting factors which decrease exponentially.

Centered moving average

When you center the moving averages, the data points are placed at the center of the range rather than the end of it.
This is done to position the moving average values at their central positions in time.

Ignore outliers

An outlier is an observation that lies an abnormal distance from other values in a random sample from a population.
Examination of the data for unusual observations that are far removed from the mass of data.
These points are often referred to as outliers.

Excluding outliers can cause your results to become statistically significant.
By default, outliers are ignored.

Highlight Modes

Highlighting functionality is useful for filtering metrics and highlight only hovered metric.
Following types of highlighting mode are available: Highlight Off, Highlight Metric on Hover, and Highlight Run on Hover.
By default, highlighting mode is the Highlight Run on Hover.

Highlight Off

By setting Highlight mode Off, there is no highlighting functionality on hover.

Highlight Metric on Hover

By setting Highlight mode Metric on Hover, mouse point closest metric highlights and other metrics displays with opacity.

Highlight Run on Hover

By setting Highlight mode Run on Hover, mouse point closest metric highlights and highlighted metric corresponding run also highlights other metrics displays with opacity.

Set tooltip parameters

You can select tooltip parameters to show params and values in tooltip Params section.
You can select hide or show button to display or hide tooltip on hover.

Apply zoom on charts

Zoom In

Zoom Out

Export chart as image

Metric explorer also, gives ability to export your chart as image.
By clicking export button from control panel, will be opened chart preview modal.
You can change exportable chart image width, single chart height, set image name and format.

Following formats of chart export are available: SVG, JPEG, PNG.

Images Explorer Copy Section

Overview

Track intermediate images search easily by using select form functional compare them on the Images Explorer by using reach controls panel.

Features:

• Easily query any image
• Group images by run parameters, step and index
• Use controls from right control panel to configure workspace
  • Image properties control
  • Images sorting control
  • Images group stacking control
  • Images tooltip params control

Query any image

Use select form to easily query any image. There are two option to query images using dropdown, by using Aim QL language and advance mode for Aim QL.

How to use select form?

• Click on Images button
• Select options you are want to use in query
• Click on the Search button

How to reset select form value?

How to use advanced query mode?

• Click on pencel icon in the right side of select form to show input
• Type advance Aim QL query
• Click on the Search button

Group image by run parameters

Use select grouping dropdown which is located in the right top corner of the image explore page.

How to use image grouping?

• Click on grouping button
• Select fields by which you want to groupe images

Grouping will be apply after each field selection also you can select grouping mode (Group or Reverse)

Image explorer right controls panel

Any change in controls will help to explore images better on the workspace

Images size manipulation control

• Click on image property button

• Select value from dropdown to align image. (by default dropdown value is Height). Use slider to configure value for scale relative to window size by default scale value is 15%.

  • By height

  • By width

  • Original size

• Use image rendering variation by default value of this control is Pixelated

Images sorting control

• Click on image sorting button
• Select fields for sorting images. Selection ordering is meaningful and data will be sorting by selection order. Bellow is visible Ordered By list where contains all selected fields from dropdown. You can remove any already selected field by clicking on x icon or change sorting direction by clicking radio button Asc or Desc. Default selected direction is Asc.
• For reset all existing sorting fields you can simply click on Reset Sorting button

Set tooltip parameters

You can select tooltip parameters to show params and values in tooltip Params section.
You can select hide or show button to display or hide tooltip on hover.

Params Explorer Copy Section

Overview

Params explorer helps you to represent high dimensional data as a multi-dimensional visualization.
Features:

• Easily query any metrics and params
• Group runs by color, stroke, or chart
• Make the crossings easier with curve interpolation
• Learn patterns and colorations easier colored by the last dimension with a color indicator

Query any metrics and params

Select params and metrics from dropdown

Search runs with Aim QL

Grouping

Group by color, stroke, or chart with selected parameters

Curve interpolation

By clicking on the Curve interpolation button in the Controls panel, it's possible to make lines from straight to curve to show
correlations between non-adjacent axes.

Color indicator

By clicking on the Color indicator button in the Controls panel, it's possible to turn on lines gradient coloring by the last dimension.

Scatters Explorer Copy Section

Scatter explorer gives ability to visualize correlations between metric last value data with hyper-parameter.

It represents graph where corresponding values from a set of data are placed as points on a coordinate plane. A relationship between the points is sometimes shown to be positive, negative, strong, or weak.

Abilities provided by Scatter explorer

• Easily align metric last value data with hyper parameter
• Group runs by color and chart
• Represent the points with trend line
• Export chart

Align metric last value data with hyper-parameter

Select params and metrics from X and Y axes dropdowns to align metric last value data with hyper-parameter.

• X axis

• Y axis

Also, you can search runs with Aim QL

Grouping

Easily group data by color and chart with selected parameters.

• By Color

• By Chart

Trend line

A trend line is a straight line that best represents the points on a scatter plot. The trend line may go through some points but need not go through them all.

From trend line options popover you can change regression from Linear (by default) to LOESS(locally weighted smoothing), which creates a smooth line through a scatter plot to help you to see relationship between variables and foresee trends. Also, you can change the bandwidth with slider

Export chart

Scatter explorer also, gives ability to export your chart as image.

By clicking export button from control panel, will be opened chart preview modal.
You can change exportable chart image width, single chart height, set image name and format.

Following image formats are available export: SVG, JPEG, PNG.

Source Ui Pages Home Page (`docs/source/ui/pages/home_page.md`) Copy Section

Home Page Copy Chapter

Aim Home Page is a high-level overview of your training activities and how to find your way around the Aim UI.

There are three main sections:

• Explore Aim

• Overview

• Latest Changes

Explore Aim Copy Section

In the Explore Aim section, you will find a list of recommended resources to help you get started with Aim or navigate through an existing Aim project. Here are its main sections

• Quick navigation to some of the most used features queries
• Recent searches
• Experiments table
• Tags table
• Bookmarks

Quick Navigation

The quick navigation section provides a quick way to navigate through your runs. It includes:

• Active Runs
• Archived Runs
• Last week's runs

Recent searches

In this block, you will find your last three successful queries with the ability to navigate through them.

Experiments table

This block displays your experiment data in the table format. Also, it provides the ability to compare the selected ones on the explorer pages.

Tags table

The tags table provides a quick way to navigate to your tags and compare them on the other explorer pages.

Bookmarks

This section displays your bookmarks and provides the ability to navigate through them.

Overview Copy Section

The overview section is a high-level overview of your training activities. It includes:

• Statistics of your training activities
• Active Runs table
• Contributions heatmap
• Activity feed

Statistics

Here you can find the total number of runs and tracked sequences. Also, each card is clickable and you can navigate to the Runs and Sequences pages.

Active Runs table

The active runs table provides the list of your active runs. Also, the ability to compare your runs in other explorer pages.

Contributions heatmap

The heatmap shows the intensity of experiments you have made for the day - the darker the color the more experiments.

Each cell represents the set of training runs for that day. The cells are clickable!

Once clicked, you will navigate to the Runs explorer page and automatically query the runs made on that day.

Activity feed

This section represents the activity feed of your contributions. It includes the following information:

• The date of the activity
• The type of activity
• The name of the activity

Latest Changes Copy Section

In this section, you can find the latest changes in Aim. It includes:

• The latest changes to the Aim version
• The Changelog of the Aim versions
• The helpful doc links for exploring the Aim

Source Ui Pages Reports (`docs/source/ui/pages/reports.md`) Copy Section

Reports Copy Chapter

Overview Copy Section

Use Aim reports to organize runs, embed visualizations, describe your findings, and share updates with collaborators.

The following image shows an example of a report created from metrics that were logged to aim over the course of a training.

The report page Copy Section

In single report page you can find 2 modes: view and edit.

Create the report in edit mode and view the final result in the view mode. The example above is on the view mode.
You can use
```aim sections to retrieve data from Aim storage and visualize them. The example shown below is an overview of the edit mode.

The details on objects and methods for data fetching and visualization are in the next sections.

Data fetching methods Copy Section

To retrieve data from Aim storage the following methods of repo object.

Note: the repo object is by default available in the report context.

All the methods accept a query parameter which defaults to True (fetches all the available sequences).

The results of this
Example:

```aim
metrics = repo.fetch_metrics('metric.name == "loss"')
linechart = LineChart(metrics)
```

Visualization objects and methods Copy Section

The following classes and methods are used to visualize all the data types fetched from Aim storage.
There are small examples provided with each one showing a basic usage.

• LineChart

Signature:
LineChart(data, x, y)

Parameters:

Example:

```aim
metrics = repo.fetch_metrics()
linechart = LineChart(metrics, x='timestamps')
```

• ImagesList

Signature:
ImagesList(data)

Parameters:

Example:

```aim
images = repo.fetch_images()
ImagesList(images)
```

• AudiosList

Signature:
AudiosList(data)

Parameters:

Example:

```aim
audios = repo.fetch_audios()
AudiosList(audios)
```

• FiguresList

Signature:
FiguresList(data)

Parameters:

Example:

```aim
figures = repo.fetch_figures()
FiguresList(figures)
```

• TextsList

Signature:
TextsList(data)

Parameters:

Example:

```aim
texts = repo.fetch_texts()
TextsList(texts)
```

All the aforementioned objects have group method available to them:

def group(prop: str, value: Union[str, list]):
    ...

prop- name of the property to be grouped by. Available options are: color, stroke_style, row, column (the first 2 are available for LineChart only)

value- single or multiple values of sequence fields to be grouped by. Available fields are all those fields that are also available in grouping options of the explorer pages.

The group method can be applied multiple times sequentially.

Example:

```aim
metrics = repo.fetch_metrics()
linechart = LineChart(metrics)
linechart.group('color', 'run.hash')
linechart.group('row', ['metric.name', 'metric.context.subset'])
```

Search Report Copy Section

The search bar is available so you can easily find the report you are looking for.

Delete Report Copy Section

The Delete button is available with the options button in the upper right corner of the report card.
It will open a modal, where you can confirm or cancel the deletion.

Source Ui Pages Run Management (`docs/source/ui/pages/run_management.md`) Copy Section

Runs Management

Runs Explorer Copy Chapter

Overview Copy Section

To navigate into Runs Explorer, click on the Runs navigation item from the left sidebar.

Runs explorer helps you to

• Search runs with pythonic query
• Observe runs in real time
• Delete or archive runs
• Export Runs report

Search Runs

Use Search bar to query runs with Aim QL.

Follow runs in real time

Switch Live Update to turn on the real time mode.

Delete or archive runs

Step 1: Select runs on the runs table:

Click on the Archive button. Fonfirmation popup appears. Click Archive again and the runs are archived!

In order to batch delete the selected runs, just use the Delete button.
In this case ase well, press Delete again on the confirmation popup and the runs will be hard deleted.
Warning: this operation is irreversible and the runs are deleted from the disk.

Export Runs report

Generate Runs CSV report by clicking on the Export button on the Runs table.

Single run page Copy Chapter

Each training run has a dedicated page on Aim. Use the single run page to observe all the tracked metadata associated with that run.

Here are the tabs available on the single run page.
Each tab visualizes respective tracked metadata or empty if not tracked.

• Overview
• Params
• Metrics
• System
• Distributions
• Images
• Audios
• Texts
• Figures
• Terminal Logs
• Settings

Overview Copy Section

Overview tab shows overall info about the run.

• These cards can contain information about Parameters, Metrics, System Metrics, CLI Arguments, Environment Variables, Packages and Git information. With this data, you can easily reproduce your run.

• Sidebar contains information about Run Date, Run Duration, Run Hash, attached Tags and gives the ability to navigate through tabs.

Also, you can apply advanced searching/filtering to those card tables.

Params Copy Section

Params tab contains a JSON-like visualization of all the tracked params data related to a single run of interest.

Metrics Copy Section

Metrics tab contains the visualizations of all the metrics tracked for the given run.

When exploring run results, some metrics are looked at more often. Pin those metrics to the top of the page to find them easier and faster. Undo pinning by clicking on "unpin" button.

Note: you can track arbitrary number of runs with lots of steps with Aim!

System Copy Section

Aim automatically tracks system metrics, so you can use them in order to detect potential resource mismanagements or anomalies.

System tab contains all the tracked system metrics for a single run.

Distributions Copy Section

You can track the gradient, the weights and the biases distributions of all the layers for lots of steps with Aim.

The Distributions tab will allow you to observe them for a single run. You can also

• navigate between the layers
• search for distribution on specific steps

The single run distributions tab is quite powerful!

Images Copy Section

The Images tab contains all the tracked images of a single run.
You can track runs with different contexts and at different steps of training.

On the left-hand side are the names of different image-sets you have tracked along with their context unpacked.

Usually the images are tracked at diff steps and with batches. This control will allow you to quickly slice and dice the specific subset of images to view.
Use these sliders to search

• which subset of steps you'd like to see (on the left-hand side)
• which indices you'd like to see (on the right-hand side)

If there is only 1 step or only 1 index, you will see the info message instead of the control

Audios Copy Section

You can track audios with Aim. Use the Audios tab to view and play the tracked audios of a single run.

Texts Copy Section

Use the Texts tab to view and search all the texts tracked for a single run.

On the left-hand side you will see the name and context of the tracked texts.
You can use the search-bar on top to search the text with regexp or just match word or case.

Use the bottom controllers to control the steps and the indices of the tracked texts too.

Figures Copy Section

Aim allows tracking Plotly and matplotlib figures.
On the Figures tab you can view all the track figures over different contexts and steps.

Note: Aim will render figures with passed or default dimensions. There will be scrolls if the size exceeds the plotly container space of the Figures tab.

Terminal Logs Copy Section

Aim streams the process output logs (terminal logs) to the Aim UI automatically near-real-time. The terminal logs are displayed under Logs tab on the single run page.The terminal logs can be disabled programmatically if needed. More on terminal logs here.

Settings Copy Section

Use the Settings tab to delete or archive the single run

Delete Run

Archive Run

Source Ui Pages Table (`docs/source/ui/pages/table.md`) Copy Section

Table Copy Chapter

Overview Copy Section

The Aim Table helps to explore and compare training runs - their results, logged hyper-parameters, system info, etc. You can also use the table to hide, delete, archive and compare selected training runs in a few clicks.

Delete the selected runs Copy Section

To delete a set of runs, select rows and click on the Delete button, which is located at the top of the table.

.. note::
In-progress runs can't be deleted. Please wait until the run is finished or terminate it to be able to delete the run.

After clicking on the Delete button a popup appears where finished and in-progress runs are divided into separate tables.

To undo the selection, hover over the selected row and click on the X button (appears on hover).

Archive/Unarchive the selected runs Copy Section

To archive/unarchive a set of runs, select rows and click on the Archive/Unarchive button, which is located at the top of the table.

After clicking on the Archive/Unarchive button a popup appears, where archived and unarchived runs are divided into separate tables.

To undo the selection, hover over the selected row and click on the X button (appears on hover).

Hide/Show the selected runs Copy Section

To hide/show a set of runs, select rows and click on the Hide/Show Lines button, which is located at the top of the table.

Source Ui Pages Tags (`docs/source/ui/pages/tags.md`) Copy Section

Tags page

Overview Copy Section

Tags functionality intended to mark a runs. A tag can be attached to the runs to distribute by segments and then find it quickly.

Create tag Copy Section

How to create tag?
There are two options for creating a tag.

First option

• Go to the tags page by clicking on the Tags from the left sidebar.

• Click on the create tag button to open the create tag form modal. In this modal there is the form that has two input fields first one for tag name the second one for tag comment and also there are exist color pallets for selecting tag color.

• Type name for a tag. Name filed is mandatory and can't be empty for tag creation form and has maximum 50 symbol limit validation.

• Type comment for a tag. Comment field is optional for tag creation form and has max 200 symbol limit validation.

• Select color for a tag from the color pallet.

• Click to the create button for saving a the tag then. After successful saving should appear toaster approving the create on the right top corner of the window. Optional there are default selected colors for tag.

Second option

• Go to any explorer page (metrics, params, images, scatters).

• Click to one of the sequence unit to open popover where is exist tag section with attach button.

• Click on attach button to open the select tag popover where you will see all your previously created tags.

• Click create tag button and you will be redirected to the tags page than the actual first option.

Attach tag Copy Section

How to attach tag to the run?

• Go to any explorer page (metrics, params, images, scatters).

• Click to one of the sequence unit to open popover where is exist tag section with attach button.

• Click on attach button to open the select tag popover where will be visible all tags.

• Select a tag you want to attach to the sequence unit. You can select more then on tag for each point.

Update attached tags Copy Section

How to update attached tags?

• Go to any explorer page (metrics, params, images, scatters).

• Select point which you want to attach tag and click on it to open popover where is the exist tag section. In tags section will be visible already attached tags.

• Click on attach button for adding new tag to open the select tag popover where will be visible all existing tags.

• Click on x icon in the right end of the each tag for removing the tag from the point.

Edit tag Copy Section

How to edit tag?

• Go to the tags page

• Click to edit icon in the right side in the tag row

• Then should appear the edit modal. In the edit modal there is the form that has two input fields first one for tag name the second one for tag comment and also there are exist color pallets for selecting tag color. In this modal is possible to make changes for tag.

• Make changes you need.

• Then you have three possible actions close modal, save changes and reset changes. After closing the modal you will lose all changes, after clicking the reset button modal form fields will be reset to initial values and by clicking the save button you will save all changes for the tag. After successful saving should appear toaster approving the update on the right top corner of the window.

Delete tag Copy Section

How to delete tag?

• Go to the tags page.

• Click on the trash icon in the right side in the tag row to open the delete modal.

• In the delete modal there is a tag name input field and a tag name label at the top of the tag name input. You need to type the tag name for approving you are want to delete that tag.

• Then you have two possible actions delete the tag or close the modal by canceling the delete operation. If you want to delete a tag please double-check the tag name and click to delete button. After tag deletion, there are no possibilities to recover it. Also if you are deleting the tag it will be removed from all relations too.

Used in overlay Copy Section

In the tags page you can select tag by clicking on circle icon then will opened overlay in the right side of window. Here is visible that runs which are use the tag. By clicking a run hash you will be redirected to single run page.

--- METRICS ---

• Files Extracted: 50
• Estimated Token Budget: ~60191 tokens
• Canonical Reference: https://codewiki.google/github.com/aimhubio/aim