elyra Documentation

Elyra extends JupyterLab with an AI centric approach.

1,991 stars Python Python 1 file · ~17,517 tokens #ai#airflow#anaconda#apache-airflow#binder
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy elyra on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name elyra --restart always elyra-ai/elyra:latest

File: README.md







Elyra

Elyra is a set of AI-centric extensions to JupyterLab Notebooks.

Elyra currently includes the following functionality:

The Elyra Getting Started Guide includes more details on these features. A version-specific summary of new features is located on the releases page.

Try Elyra

Using container images

You can also try Elyra by running one of the container images from Docker Hub or quay.io:

  • elyra/elyra:latest has the latest released version installed.
  • elyra/elyra:x.y.z has a specific version installed.

Note: You can also build a container image from the main branch ("dev build") to try out features that have not been released yet.

To run one of the container images, issue the following command, specifying a tag of your choice.

text
docker run -it -p 8888:8888 elyra/elyra:dev jupyter lab --debug

To make a local directory containing your Notebooks (e.g. ${HOME}/opensource/jupyter-notebooks/) available in your
docker container, you can use a mount command similar to the following:

text
docker run -it -p 8888:8888 -v ${HOME}/opensource/jupyter-notebooks/:/home/jovyan/work -w /home/jovyan/work elyra/elyra:dev jupyter lab --debug

These should produce output similar to that below, where you can then find the URL to be used to access Elyra in your local browser.

text
To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://4d17829ecd4c:8888/?token=d690bde267ec75d6f88c64a39825f8b05b919dd084451f82
     or http://127.0.0.1:8888/?token=d690bde267ec75d6f88c64a39825f8b05b919dd084451f82

Refer to the installation documentation for details.

Installation

For detailed information refer to the installation documentation.

Prerequisites :

Install current release (for JupyterLab 4.x)

The current release version is displayed at the top of this page.

  • Install from PyPI

    bash
    pip3 install --upgrade "elyra[all]"

  • Install from conda-forge

    bash
    conda install -c conda-forge "elyra[all]"

Install older release

Installation instructions and JupyterLab support vary by release. Note that a JupyterLab build is required. Installation instructions are located in the release-specific documentation, which can be accessed by selecting a specific version.

Elyra 4.x (JupyterLab 4.2.5+)

  • Install from PyPI

    bash
    pip3 install --upgrade "elyra[all]"

  • Install from conda-forge

    bash
    conda install -c conda-forge "elyra[all]"

Elyra 3.7 < 4.0 (JupyterLab 3.x)

  • Install from PyPI

    bash
    pip3 install --upgrade "elyra[all]<4.0.0"

  • Install from conda-forge

    bash
    conda install -c conda-forge "elyra[all]<4.0.0"

Elyra 3.1 < 3.7 (JupyterLab 3.x)

  • Install from PyPI

    bash
    pip3 install --upgrade "elyra[all]>=3.1.0" && jupyter lab build

  • Install from conda-forge

    bash
    conda install -c conda-forge "elyra[all]>=3.1.0" && jupyter lab build

Elyra 2.0 < 3.1 (JupyterLab 3.x)

  • Install from PyPI

    bash
    pip3 install --upgrade "elyra>=2.0.1" && jupyter lab build

  • Install from conda-forge

    bash
    conda install -c conda-forge "elyra>=2.0.1" && jupyter lab build

Verify Installation

Run the following commands to verify the installation. Note that in the example output below the [version] placeholder is displayed instead of an actual version identifier, which might change with every release.

bash
jupyter server extension list

Should output:

text
Config dir: /.../.jupyter

Config dir: /.../etc/jupyter
    elyra enabled
    - Validating elyra...
      elyra  OK
    jupyter_lsp enabled
    - Validating jupyter_lsp...
      jupyter_lsp [version] OK
    jupyter_resource_usage enabled
    - Validating jupyter_resource_usage...
      jupyter_resource_usage [version] OK
    jupyter_server_mathjax enabled
    - Validating jupyter_server_mathjax...
      jupyter_server_mathjax  OK
    jupyterlab enabled
    - Validating jupyterlab...
      jupyterlab [version] OK
    jupyterlab_git enabled
    - Validating jupyterlab_git...
      jupyterlab_git [version] OK
    nbclassic enabled
    - Validating nbclassic...
      nbclassic  OK
    nbdime enabled
    - Validating nbdime...
      nbdime [version] OK

Config dir: /.../etc/jupyter

NOTE: If you don't see the Elyra server extension enabled, you may need to explicitly enable
it with jupyter server extension enable elyra

bash
jupyter labextension list

Should output:

text
JupyterLab [version]
/.../share/jupyter/labextensions
        nbdime-jupyterlab [version] enabled OK
        @jupyter-server/resource-usage [version] enabled OK (python, jupyter-resource-usage)
        @krassowski/jupyterlab-lsp [version] enabled OK (python, jupyterlab_lsp)
        @elyra/code-snippet-extension [version] enabled OK
        @elyra/metadata-extension [version] enabled OK
        @elyra/pipeline-editor-extension [version] enabled OK
        @elyra/python-editor-extension [version] enabled OK
        @elyra/scala-editor-extension [version] enabled OK
        @elyra/r-editor-extension [version] enabled OK
        @elyra/theme-extension [version] enabled OK
        @jupyterlab/git [version] enabled OK (python, jupyterlab-git)

Other labextensions (built into JupyterLab)
   app dir: /.../share/jupyter/lab

Starting Elyra

After verifying Elyra has been installed, start Elyra with:

bash
jupyter lab

Getting Help

We welcome your questions, ideas, and feedback. Check the Getting Help section in the Getting Started guide to learn more about the channels you can use to get in touch with us.

Contributing to Elyra

If you are interested in helping make Elyra better, we encourage you to take a look at our
Contributing page,
Development Workflow
documentation, and invite you to attend our weekly dev community meetings.

Meetup with Us!

Our daily and weekly community meeting schedule can be found here.

File: cypress/README.md

Testing

Frontend Test Development

The frontend of Elyra utilizes a testing framework called Cypress.
Cypress includes a useful UI for developing and debugging tests, which can be used by
running npm test-debug from the root directory of this project. To run tests that output
a detailed log instead of opening the Cypress UI, run npm test. Tests can also be run
from the makefile by running make test. The spec code for tests is located in
the cypress/tests/ directory.

File: docs/source/developer_guide/contributing.md

Contributing to Elyra

Elyra is an open-source project and welcomes contributions from the community be it as code, documentation, scenarios, etc.

Before you start, look at the project:

Also, for details on configuring your development environment, see:

Elyra UI

Elyra runs as extensions to the Jupyter ecosystem, thus it's UI is currently implemented as
JupyterLab widget extensions.

Contributing to Elyra UI Extensions

Existing UI Extensions

The existing extensions are currently available as individual npm packages inside the
packages folder.

text
elyra
 -- packages
    -- code-snippet
    -- metadata
    -- metadata-common
    -- pipeline-editor
    -- python-editor
    -- r-editor
    -- script-editor
    -- services
    -- theme
    -- ui-components

File Naming and File Creation conventions

Elyra extensions use a separate file for each widget, and keep helper functions and classes in the file with that widget.
A file containing a widget is named after the widget (i.e. PipelineEditorWidget.tsx contains the class PipelineEditorWidget).

Files containing util functions are named generically after the utils functionality
(i.e. if a util function returned a custom dialog, the file that contained that function would be called dialog.tsx).

The index file contains only the definition of the extension class and any other exports that the extension creates.

Import sections conventions

Elyra extensions create separate sections for imports from different categories
(i.e. all imports from @jupyterlab would be in a separate section from imports from @lumino).
Sections are separated by a blank line. Each section is alphabetized by the name of packages.

UI Tests

Elyra uses two types of frontend tests: integration tests (which use cypress)
and unit tests (which use jest).

UI Integration tests

Before running integration tests, docker needs to be installed and running on your machine.
There are two ways to run the integration tests:

  • To only see the output logs from all of the integration tests,
    run make test-integration from the root directory.
  • To run or debug specific tests,
    run make test-integration-debug. This will open an interactive UI tool for writing and debugging individual test files.

Elyra's integration tests automatically start JupyterLab and visit / interact with pages through cypress API calls.
The tests use the cypress API to check for the existence of various buttons and visual elements.
Refer to the cypress API for more details.

New integration tests can be added to cypress/tests.

UI Unit tests

To run all of the unit tests, use make test-ui-unit from the root directory. To run the unit tests for a specific Elyra package, simply run jest or npm run test from that package's directory (under packages/). To turn on the watch mode just run jest --watch or npm run test --watch.

Elyra's unit tests test the various classes and objects used by Elyra extensions. Refer to the jest API for more details.

To add unit tests for a package that doesn't have tests set up, some configuration files are required. In the directory for the package being tested, add a file titled jest.config.js that contains the following:

text
module.exports = require('../../testutils/jest.config');

Then, in the package.json, add the following under 'scripts':

text
"test": "jest",
"build:test": "tsc --build tsconfig.test.json",

And the following under 'dev_dependencies':

text
"@jupyterlab/testutils": "4.4.2",
"@types/jest": "^29.2.0",
"jest": "^29.2.0",
"jest-raw-loader": "^1.0.1",
"ts-jest": "^29.3.2",
"ts-loader": "^6.2.1",
"typescript": "~5.5.4",

Create a file tsconfig.test.json that contains:

text
{
  "extends": "../../tsconfigbase.test",
  "include": ["src/*", "test/*"],
  "references": []
}

Finally, create a folder called test in the src directory of the package being tested, and add tests using the file extension .spec.ts.

Elyra Backend

Elyra runs as extensions to the Jupyter ecosystem, thus it's backend is currently implemented as
JupyterServer server extensions
and exposed as REST APIs to frontend clients.

Elyra backend services

The existing services are currently available as individual python modules inside the
elyra folder.

text
elyra
 -- packages
    -- airflow
    -- api
    -- cli
    -- contents
    -- kfp
    -- metadata
    -- pipeline
    -- templates
    -- util

Backend tests

To run all server tests, use make test-server from the root directory. There are also two ways to run only specific backend tests:

  • To run all tests in a specific directory or file, run pytest [resource] where resource is the relative path to a directory or file inside the server tests folder.
  • To run specific tests by function name, run pytest -k [test_function1 test_function2 ...]

Elyra documentation

The Elyra documentation's source is stored in the Elyra repository and hosted at Elyra's Read the Docs. The documentation is written in Markdown and built using Sphinx.

Contributing to the Elyra documentation

To contribute content to the Elyra documentation follow these steps:

  1. Fork the repository.

  2. Clone your fork.

    text
    git clone https://github.com/git-id-or-org/elyra

    The documentation assets are located in the /docs directory.

    • To add a new document create a new markdown file in the appropriate section subdirectory (e.g. /docs/source/getting_started) and add an entry to that section in /docs/source/index.rst.
    • To update an existing document edit the corresponding markdown file.
    • Place new or updated images in the /docs/source/images directory. For example, images for the pipeline components topic in the user guide are stored in docs/source/images/user_guide/pipeline-components/. PNG is the recommended format.

  3. Build the documentation assets locally.

    In the repository's root directory (not the /docs directory) run

    text
    make docs

  4. Review the build output and verify that your updates introduced no warnings or errors.

  5. Review the updated documentation assets.

    1. Navigate to the /docs/_build/html/ directory.
    2. Open index.html in a web browser.

  6. Commit your updates to a new branch and open a pull request.

File: docs/source/developer_guide/development-workflow.md

Development Workflow

This section describes the steps necessary to build Elyra in a development environment.

Requirements

Setting up your development environment

  • Install Miniconda
    Download and install a Python 3 version of Miniconda according to your Operating System

  • Create a new Python environment using a version that is supported by Elyra.

    text
    conda create -n <env-name> python

    The Python version of your environment will match the miniconda version you installed. You can override the default by explicitly setting python=3.10, for example.

  • Activate the new environment

    text
    conda activate <env-name>

  • Verify your miniconda environment

    text
    python --version # should yield a version that is supported by Elyra
which python     # displays current `python` path
pip3 --version   # should be a recent version to avoid build issues
which pip3       # displays current `pip` path

    Python path must be under miniconda envs folder.
    Confirm pip3 location matches where miniconda is installed.

  • Install a version of Node.js that is supported by Elyra.

    text
    conda install -y -c conda-forge/label/main nodejs

  • Verify node is installed correctly

    text
    node --version

  • Install Yarn

    text
    conda install -y -c conda-forge/label/main yarn

  • Verify yarn is installed correctly

    text
    yarn --version

  • Install GNU Make

    Refer to the following link for installation instructions:
    GNU Make

    To verify the installation, run make.
    If you have yet to set up the repository, you should see a message like the following:

    text
    make: *** No targets specified and no makefile found.  Stop.

    Once the repository is set up, running make from that location should display the available tasks that are listed in the Build & Installation section below.

Setting up your Elyra Github repository

  • Fork the Elyra Github repository (if you haven't already)

  • Make a local copy of Elyra fork

    text
    git clone https://github.com/<your-github-id>/elyra.git
cd elyra

  • Set upstream as described in the GitHub documentation

Building

Elyra is divided in two parts, a collection of Jupyter Notebook backend extensions,
and their respective JupyterLab UI extensions. Our JupyterLab extensions are located in our packages
directory.

Build & Installation

Elyra uses make to automate some of the development workflow tasks.

Issuing a make command with no task specified will provide a list of the currently supported tasks.

bash
$ make

build-ui-dev                   Build UI packages for development
build-ui-prod                  Build UI packages for production
clean                          Make a clean source tree and uninstall extensions
container-images               Build all container images
docs                           Build docs
elyra-image-env                Creates a conda env consisting of the dependencies used in images
install-all-dev                Build and install for development, including examples
install-all-prod               Build and install for production, including examples
install-dev                    Build and install for development
install-examples               Install example pipeline components
install-prod                   Build and install for production
install-server                 Build and install backend
lint                           Run linters
package-ui-dev                 Package UI for development
package-ui-prod                Package UI for production
publish-container-images       Publish all container images
release                        Build wheel file for release
test                           Run all tests (backend, frontend and cypress integration tests)
watch                          Watch packages. For use alongside jupyter lab --watch

You can build and install all Elyra packages with:

bash
make clean install-prod

You can check that the notebook server extension was successfully installed with:

bash
jupyter serverextension list

You can check that the JupyterLab extension was successfully installed with:

bash
jupyter labextension list

NOTE:
When switching between Elyra major versions, it is recommended to clean your JupyterLab environment before a build.
The clean-jupyterlab removes your JupyterLab packages and completely deletes your Jupyter workspace.
Make sure to backup any important data in your environment before running the script.
To clean your environment and install the latest JupyterLab:
etc/scripts/clean-jupyterlab.sh
To specify a JupyterLab version x.y.z to be installed:
etc/scripts/clean-jupyterlab.sh --version x.y.z

Parallel Development with @elyra/pipeline-editor

You can install Elyra using a local build of @elyra/pipeline-editor with:

bash
make clean install-dev

Back-end Development

After making code changes to the back-end, you can re-build Elyra's Python package with:

bash
make install-server

This command builds and installs the updated Python package independently, skipping any UI component build.

Restart JupyterLab to pick up the new code changes.

Front-end Incremental Development

Elyra supports incremental development using --watch. This allows you to make code changes to
front-end packages and see them without running make install again.

After installation run the following to watch for code changes and rebuild automatically:

bash
make watch

Then in a separate terminal, using the same Python environment, start JupyterLab in watch mode:

bash
jupyter lab --watch

When in watch mode JupyterLab will watch for changes in the build of each package and rebuild.
To see your changes just refresh JupyterLab in your browser.

NOTE: JupyterLab watch mode will not pick up changes in package dependencies like services.
So when making changes to services you will need to stop and restart jupyter lab --watch and
not just refresh your browser.

Building the Elyra Container Image

Elyra's container image can be built in two ways (production and development):

Development:

bash
make elyra-image

By default, the command above will build a container image (development) with the changes that exist in your local branch.

Production:
From main branch:

bash
make elyra-image TAG=3.7.0

or after checking out a git tag e.g. git checkout tags/v3.7.0

bash
make elyra-image

In order to build from a particular release (production), you can pass a TAG parameter to the make command
or you can checkout the respective tagged release and omit the TAG parameter.

Official container images are published on Docker Hub
and quay.io.

Developing Elyra against the Jupyterlab source repo

Sometimes it is useful to develop Elyra against a local build of Jupyterlab. To use a local build of Jupyterlab use the
following steps in the same python environment.

  1. Uninstall any pip installations of Jupyterlab. You can use etc/scripts/clean-jupyterlab.sh --version dev as
    mentioned above with --version dev to not reinstall Jupyterlab at the end of the script.

  2. Build your local repo of Jupyterlab, step-by-step instructions can be found in the
    Jupyterlab documentation.
    Uninstalling in the previous step will also wipe any previous installations of a local build.

  3. cd to the builder/ directory in your Jupyterlab repo and run yarn link. The Elyra Makefile will use this yarn
    link in step 6.

  4. In your Elyra repo, uncomment the following line in tsconfig.base.json to tell Typescript to use the local
    Jupyterlab packages when building:

    "paths": { "@jupyterlab/*": ["../jupyterlab/packages/*"] },

  5. Comment out jupyterlab and jupyterlab-lsp in the install_requires section of setup.py in your Elyra repo.
    This will prevent Jupyterlab from being pip installed during the Elyra build.
    Note: jupyterlab-lsp also pip installs Jupyterlab when installed

  6. Run make install-dev to install Elyra using the linked @jupyterlab/builder from step 3.

  7. You can now start Jupyterlab by running jupyter lab --dev-mode --extensions-in-dev-mode, this will automatically
    watch for changes in the Jupyterlab repo. To also watch for changes in Elyra run make watch in a separate terminal
    in the same Python environment.

When you want to switch back to developing Elyra against a Jupyterlab release, you just have to undo the comments in
steps 4 and 5 and rebuild with make clean install

Analyzing automated test failures

The Elyra GitHub repository is configured to run automated tests whenever a pull request is opened. These tests include static code quality analysis and UI, server, and integration tests.

The test results can be accessed from the pull request or the actions tab. If the test log does not include enough details to diagnose failures, download and review test artifacts that might have been generated.

Accessing test artifacts

  1. Open the Elyra repository actions panel (https://github.com/elyra-ai/elyra/actions).

  2. Locate the failing workflow.

  3. Open the workflow.

  4. Click the 'home' (summary) button.

  5. Locate the 'Artifacts' section. If present, it should contain a download link.

  6. Download the archive, extract it, and review the artifacts.

File: docs/source/developer_guide/issue-management.md

Issues Workflow

This section describes how Elyra issues are organized.

Labels used for tagging Issues and Pull Requests

Keyword Tags
Area
Impact
Help Wanted
Mentoring Program
Kind
Feedback
External
Component
Platform
Browser
Inactive
Status
Priority

File: docs/source/developer_guide/logging.md

Logging

This section will go over how to properly log messages in Elyra

Pipelines Extension Logging

Python

The Kubeflow Pipelines extension uses a custom IPython handler to handle interactions with
the Notebook server instance. By extending the IPython handler we are able to leverage Tornado logging
since the Jupyter Notebook server uses Tornado as its web framework.

Pipelines -> IPythonHandler -> Tornado.log -> lib/logging

When to add log messages

Use INFO level for routines(functions), like when handling requests or state changes and DEBUG for troubleshooting and diagnostics.

text
def foo (arg):
    self.debug.info(f"doing something with this {arg}")
    do something with arg
    self.log.info(f"Result of arg is ... {result}")

Use WARNING level when it is important, but not an error, for example, when a user attempts to login with an incorrect password

text
def authenticate(username, password):
    if username and password doesn't work:
        self.log.warn(f"Failed login attempt from {username} : {password}")

Use ERROR level when something is actually wrong, like when an exception is thrown, IO operation failure or connectivity
issue (e.g. Minio client cant connect to endpoint)

text
try:
        if not minio_client.bucket_exists(bucket_name):
            minio_client.make_bucket(bucket_name)
    except ResponseError:
        self.log.error("Minio error", exc_info=True)

Adding log messages

Add log messages by inserting statements as follows:
self.log.debug("insert debug message", *args, *kwargs)
Use the appropriate logging levels when inserting specific messages e.g. log.debug, log.warn etc.

The log output will be color coded, legend as follows:

Log Level Color
DEBUG Blue
INFO Green
WARNING Yellow
ERROR Red
CRITICAL Magenta

Setting the Log Level

The default Log Level is set to INFO. If you need to increase log verbosity, you can start Jupyterlab with
the --debug option for example:

text
jupyter lab --debug <other options>

If too verbose, you can filter out only the most important messages by directly setting the value
with --log-level= using one of the following {DEBUG|INFO|WARN|ERROR|CRITICAL}. For example:

text
jupyter lab --log-level=WARN <other options>

indicates that only log levels of WARN (or higher: ERROR and CRITICAL) will be logged.

TypeScript / JS

Use console.log(), console.warn() to output debug messages

File: docs/source/developer_guide/metadata.md

Metadata Services

While building enterprise extensions for the Elyra project we identified a requirement to integrate with external runtimes, data sources, and other
components that require additional metadata in order to connect
to these external components.

The Metadata Service provides a generic service that can be used to store metadata
that can be easily integrated with Elyra backend and/or frontend components.

Metadata Service Overview

The core design of the metadata service is that it be primarily driven by schema. That is, all metadata instances must adhere to a schema. Metadata schema are organized, both logically and physically, into schemaspaces where a given schemaspace contains a set of schemas and each instance of a schemaspace's schemas are co-located relative to the configured storage mechanism.

Externally, both in its persisted and serialized forms, a metadata instance is represented as a JSON structure. Within the metadata service, however, this JSON structure is represented as a Metadata class instance. The Metadata class can be extended via subclassing to enable custom behaviors that are triggered following load operations and/or before and after create, update and delete operations on a per schema level. That is, it is the respective schema that identifies the name of the class to use as the internal representation.

As previously mentioned, the metadata service has a pluggable storage mechanism. By default, metadata instances are stored within the file system. However, alternative storage mechanisms can be configured for, say, NoSQL DB or traditional, table-oriented, databases.

Schemaspaces and their schemas can be dynamically introduced via entrypoints. When the system starts, it will identify applicable entrypoints and load their corresponding objects to seed the list of configured schemaspaces and schemas.

Metadata Components

The Elyra metadata service is composed of several components, many of which were previously referenced, and each responsible for a different task.

The following briefly describes each of those components.

  • Schemaspace: This is the top-most component and represents a container of schemas. It has id (UUID), name (symbolic name), display_name (descriptive name), and description properties, along with a set of schemas. Instances of a schemaspace's schemas are logically grouped together in whatever storage model is in use. Schemaspaces are discovered and loaded via entrypoints.
  • SchemasProvider: A schemas provider is a class that is also discovered and loaded via entrypoints. Its purpose is to provide schemas. Since a schemaspace is always referenced within the schema, a given SchemasProvider instance could provide schemas corresponding to different schemaspaces although a schemas provider will typically correspond to a single schemaspace.
  • Metadata: This class is used internally to house the JSON instance data. It can be subclassed to enable applications to hook persistence and retrieval operations.
  • SchemaManager: This class is a singleton instance that loads and houses all schemas organized by their respective Schemaspace. All schemas provided by a SchemasProvider must adhere to the meta-schema and the SchemaManager is responsible for validating provided schemas.
  • MetadataManager: This class operates on Metadata instances. Prior to each persistence operation, and following each load, the MetadataManager is responsible for validating the instance against its corresponding schema, which it obtains from the SchemaManager. It is also responsible for driving the persistence and retrieval of instances against the configured MetadataStore.
  • MetadataStore: This class is an abstract base class that defines methods relative to the persistence and retrieval of metadata instances. It can be subclassed such that each subclass typically represents an alternative storage mechanism. The system provides a FileMetadataStore instance - which is configured as the default storage mechanism.
  • REST/CLI API: The metadata service entities can be manipulated via a REST API consisting of /elyra/metadata, /elyra/schema and /elyra/schemaspace endpoints. Similar capabilities can also be performed using the command-line tool (CLI) elyra-metadata.

Metadata Services structure using the default file system store

The default implementation is for the metadata services to store metadata files in the file system, grouped
by directories according to the metadata instance's schemaspace.

The root directory for metadata is relative to the 'Jupyter Data directory' (e.g. jupyter --data-dir)

text
/Users/jovyan/Library/Jupyter/metadata/

Each instance of metadata is then stored in a child directory representing the instance's schemaspace.

As an example, runtimes is the schemaspace for runtime metadata instances that reside in the following directory:

text
/Users/jovyan/Library/Jupyter/metadata/runtimes

The contents of this folder would then include multiple JSON files, each associated with a schema corresponding to the desired runtime platform.

For example, the following contains runtime metadata instances for two runtime platforms, airflow and kfp,
where each runtime has 1 and 2 runtime instances defined, each of which adheres to their respective schemas.

text
/Users/jovyan/Library/Jupyter/metadata/runtimes/airflow-cloud.json
/Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-fyre.json
/Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-qa.json

And each metadata file contains information similar to the following (note the reference to its schema_name):

json
{
  "display_name": "Kubeflow Pipeline - Fyre",
  "schema_name": "kfp",
  "metadata": {
    "api_endpoint": "http://weakish1.fyre.ibm.com:32488/pipeline",
    "api_username": "[email protected]",
    "api_password": "mypassword",
    "cos_endpoint": "http://weakish1.fyre.ibm.com:30427",
    "cos_username": "minio",
    "cos_password": "minio123",
    "cos_bucket": "test-bucket",
    "tags": [
      "kfp", "v1.1"
    ]
  }
}

Metadata Service REST API

A REST API is available for easy integration with frontend components:

Retrieve all metadata for a given schemaspace:

rest
GET /elyra/metadata/<schemaspace>

Retrieve a given metadata resource from a given schemaspace:

rest
GET /elyra/metadata/<schemaspace>/<resource>

Metadata Client Tool

Users can easily manipulate metadata via the Client Tool

bash
elyra-metadata list runtimes
text
Available metadata instances for runtimes (includes invalid):

Schema   Instance       Resource  
------   --------       -------- 
kfp      kfp-fyre       /Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-fyre.json
kfp      kfp-qa         /Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-qa.json
airflow  airflow-cloud  /Users/jovyan/Library/Jupyter/metadata/runtimes/airflow-cloud.json

Metadata APIs

A Python API is also available for accessing and manipulating metadata. This is accomplished using the MetadataManager along with a corresponding storage class. The default storage class is FileMetadataStore.

python
from elyra.metadata import MetadataManager, FileMetadataStore

metadata_manager = MetadataManager(schemaspace="runtimes",
                                   store=FileMetadataStore(schemaspace='runtimes'))

runtime_configuration = metadata_manager.get('kfp')

if not runtime_configuration:
    raise RuntimeError("Runtime metadata not available.")

api_endpoint = runtime_configuration.metadata['api_endpoint']
api_username = runtime_configuration.metadata['api_username']
api_password = runtime_configuration.metadata['api_password']
cos_endpoint = runtime_configuration.metadata['cos_endpoint']
cos_username = runtime_configuration.metadata['cos_username']
cos_password = runtime_configuration.metadata['cos_password']
bucket_name = runtime_configuration.metadata['cos_bucket']
tags = runtime_configuration.metadata['tags']

Bring Your Own Schemas

The metadata service enables the ability to introduce your own schemaspaces and bring custom schemas to existing schemaspaces via entrypoints. It is necessary to introduce Schemaspaces when developing a new extension or adding functionality in which it makes sense to group instances of the functionality together. For example, Code Snippets consist of their own Schemaspace and, currently, are based on a single schema.
Applications extending existing functionality do not need to introduce a new Schemaspace, since one will already exist, but must be able to introduce their custom schema. Introducing your own runtime would be an example where a Schemaspace is not required. In this case, only a SchemasProvider would be necessary.

Schemaspace

A Schemaspace instance is used to represent a set of schemas. Custom schemaspace classes must derive from the Schemaspace base class and are responsible for providing basic information like id (UUID), name (used as a symbolic name for lookups), display_name (a human-readible, descriptive name) and description. They also contain a dictionary of JSON schemas, indexed by schema name. However, the dictionary is empty at construction and filled during system initialization via the SchemasProvider mechanism. By convention, schemaspace names are plural, while schema names are singular.

Registration

Schemaspaces are discovered via the entrypoints functionality. When the SchemaManager starts, it fetches all entrypoint instances named by the group 'metadata.schemaspaces', and uses the entrypoint data to load each found instance.
Entrypoints are registered in the package's setup configuration.

Here is an example of the schemaspace entrypoints defined in Elyra:

text
entry_points={
  'metadata.schemaspaces': [
    'runtimes = elyra.metadata.schemaspaces:Runtimes',
    'runtimes-images = elyra.metadata.schemaspaces:RuntimeImages',
    'code-snippets = elyra.metadata.schemaspaces:CodeSnippets',
    'component-registries = elyra.metadata.schemaspaces:ComponentRegistries'
  ]
}

As you can see, each entry has a name that equates to a module and object/class.

SchemasProvider

As mentioned above, Schemaspace instances do not know their set of schemas at the time of their construction. Instead, the SchemaManager loads a different entrypoints group following the loading of schemaspaces named 'metadata.schemas_providers'. These entrypoints identify a SchemasProvider object and this object is responsible for returning schemas when called upon by the SchemaManager.
SchemasProvider instances are concrete instances of the abstract base class SchemasProvider, which requires an implementation of the get_schemas() method:

python
class SchemasProvider(ABC):
    """Abstract base class used to obtain schema definitions from registered schema providers."""

    @abstractmethod
    def get_schemas(self) -> List[Dict]:
        """Returns a list of schemas"""
        pass

When called, the SchemasProvider instance is responsible for producing a list of schemas. These schemas must conform to the metadata service's meta-schema and, as a result, will include the schemaspace name and schemaspace id to which they correspond, in addition to a name property representing that schema's name. The SchemaManager uses this meta-information to confirm the schemaspace has been loaded and register that schema with the schemaspace following its successful validation against the meta-schema.

Some SchemasProvider instances may have schema information which is dynamic, based on the active configuration of the system. In such cases, they are responsible for returning information pertinent to the current configuration. For example, a schema may use an enum that holds a set of values pertinent to the system's configuration. In this case, the SchemasProvider may alter its static schema to dynamically update its enumerated property corresponding to the current runtime environment. Note however, that care should be taken when dynamically altering schema since it could lead to instances that will no longer load because they (now) fail schema validation if the contents of the enum no longer satisfy previous instance values. Of course, this may very well be proper behavior, depending on the intention.

Validators

A validator is used
to validate instances of a given schema. Within the metadata service, instances
are validated whenever they are retrieved, and immediately prior to their persistence.

The SchemasProvider base class provides an instance of the Draft7Validator which should
be sufficient for most schemas. Implementations requiring their own validator, or that wish to
extend the Draft7Validator, must override the get_validator() method in their
subclass of SchemasProvider:

python
def get_validator(self, schema: dict) -> Any:
        """Returns an instance of a validator used to validate instances of the given schema.

        The default implementation uses the `Draft7Validator`.  SchemasProviders requiring
        the use of a different validator must override this method.
        """
        return Draft7Validator(schema, format_checker=draft7_format_checker)

Registration

Like Schemaspaces, SchemasProvider implementations are discovered via the entrypoints functionality. When the SchemaManager starts, and after it loads all Schemaspaces, it fetches all entrypoint instances named by the group 'metadata.schemas_providers', and uses the entrypoint data to load each found instance.

Here is an example of the schemas_providers entrypoints defined in Elyra:

text
entry_points={
  'metadata.schemas_providers': [
    'runtimes = elyra.metadata.schemasproviders:RuntimesSchemas',
    'runtimes-images = elyra.metadata.schemasproviders:RuntimeImagesSchemas',
    'code-snippets = elyra.metadata.schemasproviders:CodeSnippetsSchemas',
    'component-registries = elyra.metadata.schemasproviders:ComponentRegistriesSchemas'  ]
}

Implementing Custom Metadata Classes

Applications can also influence behavior by introducing their own sublcass of Metadata to wrap instance data. When this is done, those implementations will be called prior to and after any create, update and delete operations, in addition to following any load (get) operations. As a result, applications can influence what is persisted and what is retrieved.

It should be noted that any exceptions raised from the method overrides will affect the overall operation. This may be warranted in some cases in which the application finds some invariant condition violations, perhaps those that can't be expressed in the schema, in which the operation should be terminated. If the exceptions are raised in either post_save() or post_delete(), the MetadataManager will also apply a compensating transaction corresponding to the current operation. For example, if an instance is being created and the post_save() method raises an exception, the MetadataManager will apply a compensating transaction that deletes the newly-created instance.

Registration

Unlike Schemaspaces and SchemasProvider registrations, metadata class overrides are registered within their respective schema file via the metadata_class_name property. The value of this property denotes the dotted notation corresponding to the location of the implementation, the hosting package of which must be installed and locatable. (Note that someday, we may choose to change this value to be an entrypoint name of a well-documented entrypoint group. In such cases, the value would not be "dotted" and, therefore, backwards compatible with the current approach.)

Here's an example of what the component-registry schema provides:

python
"metadata_class_name": "elyra.pipeline.component_metadata.ComponentRegistryMetadata"

In this case, the ComponentRegistry module overrides the post_save() and post_delete() methods to update its component's cache.

Migration Strategies

It's inevitable that schemas will change over time. When such changes occur, the existing instances may be impacted such that they no longer conform to the schema or should belong to a different schema entirely. That is, the instances need to be migrated to adhere to the updated schema. This section covers two migration strategies, Implicit Migration and Explicit Migration.

Implicit Migration

Implicit migration occurs on the fly, requiring no user intervention. This form of migration typically occurs when the schema is altered to include a new required property or a schema within the schemaspace has been deprecated and existing instances should be associated with a different schema. With implicit migrations, the instances will typically remain in their current schemaspace.

The recommended approach in these scenarios is to leverage the Metadata subclass that can be associated with each instance. By introducing a subclass via the metadata_class_name meta-property, applications can then use the on_load() method to hook the loading of instances, apply the necessary alterations, persist the updates, and return.

For example, let's say that schema foo has been deprecated in favor of schema bar and that all instances of foo should be updated to instances of bar. One could perform the following steps to ensure instances are migrated from foo to bar:

  1. Introduce a metadata_class_name to schema foo(if one doesn't exist already).
  2. Implement (or alter) the on_load() method. This method needs to detect pre-migrated instances. In this case, we're changing the instance's schema, so any instance triggering this method is considered pre-migrated. However, if the instance will remain under the schema, this is probably a good time to introduce an application-level version indicator to determine if the instance is current or requires migration.
  3. Make the necessary changes, including the update of the schema_name field, and persist the changes using an appropriate MetadataManager instance (tied to the schemaspace), being sure to include the for_migration=True property in the update() method.
python
def on_load(self, **kwargs: Any) -> None:
        """Perform implicit migration and update schema from 'foo' to 'bar' """

        prev_schema = self.schema_name
        self.schema_name = "bar"

        getLogger('ServerApp').info(f"Migrating '{prev_schema}' instance '{self.name}' "
                                    f"to schema '{self.schema_name}'...")
        MetadataManager(schemaspace="my_schemaspace").update(self.name, self, for_migration=True)

With implicit migration, the application migrates the instance on load, so you should ensure the instances are "touched". If that is an issue, you could use the elyra-metadata CLI to list the instances - triggering their migration. For example: elyra-metadata list my_schemaspace

Explicit Migration

Explicit migration is typically necessary when the changes are more widespread. Examples of such migration will be when instances need to move from one schemaspace to another. In such cases, implicit migration is not practical.

An example of an explicit migration occurred in the Elyra 3.3 release when instances of the component-registry schema in schemaspace component-registries were migrated to different schemas (depending on a field's value) in a new schemaspace named component-catalogs. In this case, we chose to use an explicit migration because there was no application-level trigger to "touch" existing instances because the component-registries schemaspace was considered deprecated.

Explicit migration centers around the schemaspace from which instances are being migrated. This is because it is the only entity that knows it has instances. This migration is accomplished via the base Schemaspace migrate() method. This method is marked as final meaning that it cannot be overridden by Schemaspace subclasses, and it merely enumerates the schemas (and their providers) calling the migrate() method on the corresponding SchemasProvider subclass.

When called SchemasProvider.migrate() enumerates instances of the given schema, updates those instances, persisting changes as necessary, and returns the list of migrated instances to the caller (i.e., the migrate() method on Schemaspace). (The code for the migration of component-registry instances to the component-catalogs schemaspace can be found here.)

To drive explicit migration, the elyra-metadata CLI tool has been updated with a migrate command - which acts on the pre-migration schemaspace. This option essentially calls the migrate() method on the given Schemaspace instance - which then invokes the appropriate SchemasProvider to migrate its schema instances. See Migrating user-defined component registries to 3.3 for details.

File: docs/source/developer_guide/pipeline-component-connectors.md

How to develop a component catalog connector

Elyra uses connectors to retrieve pipeline components from local or remote component catalogs.

The examples repository includes a list of available catalog connectors and instructions on how to create a custom connector.

File: docs/source/developer_guide/pipelines.md

Pipelines

The following sections cover the details relative to working with pipelines in a development capacity.

Pipeline definition

Below is a json representation of the pipeline that is generated by the Pipeline Editor and referred to as the pipeline definition.

The pipeline is then submitted to the Elyra Scheduler which parses, properly packages and
submits the pipeline definition to the chosen target runtime (e.g. Kubeflow Pipelines).

For more details on the pipeline json definition see it's json schema

Pipeline definition json example

text
/* Detailed source-code truncated for AI context efficiency. */

Pipeline Processor Customization

Elyra implements an extensible pipeline processor engine, which enables the addition of new pipeline processors utilizing
a service discovery mechanism. The pipeline processor class hierarchy is depicted here:

This section outlines what is needed to introduce your own runtime for integration with Elyra. In essence, two criteria must be fulfilled to introduce a new runtime:

  1. A schema describing the necessary runtime metadata
  2. A pipeline processor implementation appropriately associated to the runtime

Custom Runtime Schema

The first requirement for introducing a new runtime for use in Elyra is to define the necessary metadata corresponding to the runtime. This is accomplished via a JSON schema file describing the necessary metadata used to integrate with the targeted runtime platform.

The schema will be made available to the application via a SchemasProvider (See Bring Your Own Schema in Metadata Services) which implements a single method get_schemas().

The schema name, represented by the value of the top-level name property is what the pipeline engine uses to locate the appropriate pipeline processor implementation. This value is also set into the pipeline definition's runtime property in Elyra's UI, thereby tying the pipeline to the appropriate processor.

The schema should minimally include property definitions for the Cloud Object Storage (cos) properties found in the built-in schema definitions for Kubeflow Pipelines and Apache Airflow.

Custom Runtime Pipeline Processor Implementation

The pipeline processor implementation that corresponds to the targeted runtime platform must be a subclass of elyra.pipeline.processor.RuntimePipelineProcessor - which, itself, derives from elyra.pipeline.processor.PipelineProcessor.

To facilitate discovery by the pipeline engine, this implementation must use a value for its type property identical to its schema name.

Processor Registration

The pipeline processor should be registered using entry_points with a name that matches both the schema name corresponding to the new runtime and the pipeline processor's type property value. Although this is not technically required, we would like to possibly utilize this during discovery in the future (and is true relative to the built-in processors).

Entrypoint definitions are typically found in the package's setup.py or setup.cfg files similar to the following example:

python
entry_points = {
        'elyra.pipeline.processors': [
            'my_runtime = acme.my_runtime:MyRuntimePipelineProcessor'
        ]
    },

In this example, and corresponding to the criteria above, pipeline processor acme.my_runtime.MyRuntimePipelineProcessor will:

  • derive from elyra.pipeline.processor.RuntimePipelineProcessor

  • use a value of my_runtime for its type property value

  • implement a SchemasProvider that will return the JSON schema for my_runtime via its get_schemas() method.

File: docs/source/developer_guide/refresh-elyra-on-odh.md

Elyra refresh on Open Data Hub

A custom Elyra container image is included with Open Data Hub. This document outlines how to upgrade that image to a newer version. This task is typicaly only performed by Elyra maintainers.

Update Elyra image stream

The s2i-lab-elyra repository contains the artifacts that make Elyra available as an image stream.

  • Install pipenv (version <= 2022.1.8) from PyPI.

    text
    $ pip install pipenv==2022.1.8

  • Fork the https://github.com/opendatahub-io/s2i-lab-elyra repository.

  • Git clone the fork.

    text
    $ git clone [email protected]:<your-fork>/s2i-lab-elyra.git

  • Create a new branch named update-to-vX.Y.Z (replacing X, Y, and Z with the latest major, minor, and patch version number for Elyra).

  • Update the following as needed:

    • .aicoe-ci.yaml if any version changes to the base container image are needed. Only updated periodically after a new version of the base image has been released and tested and vetted by the upstream team for ODH.
    • .s2i/environment if any additional environment variables need to be changed or set
    • .s2i/run if any additional bash setup commands need to be added/changed prior to starting Elyra

  • Update Pipfile:

    • Replace the Elyra version number
      text
      "elyra[all]" = "==X.Y.Z"

  • Regenerate the pipfile.lock file

    text
    $ pipenv lock --pre

  • Create a pull request providing the following information:

    • Title: Update elyra image to vX.Y.Z
    • Description: add links to the release notes for every version that isn't available yet on Open Data Hub. For example, if the last version on ODH is 3.9.1 and you are bumping Elyra to 3.10.1, add links to the release notes for 3.10.0 and 3.10.1. This enables reviewers and users to determine what is new and why an upgrade to this release should be considered.

    Example pull request: https://github.com/opendatahub-io/s2i-lab-elyra/pull/43

  • After the pull request was merged a new image release is created for you. Take note of the new release vY.Y.Y.

    The release version is different from the Elyra version!

Update manifest files

Once the image stream pull request was merged you can update Open Data Hub manifest files.

  • Fork the https://github.com/opendatahub-io/odh-manifests repository.

  • Git clone the fork.

    text
    $ git clone [email protected]:<your-fork>/odh-manifests.git

  • Create a new branch named update-to-vY.Y.Y replacing YYY with the image release number.

  • Update jupyterhub/notebook-images/overlays/additional/elyra-notebook-imagestream.yaml

    • Change the existing opendatahub.io/notebook-python-dependencies annotation from "version":"X.X.X" to "version":"X.Y.Z".
    • Change all existing occurrences of the old image release vX.X.X to the new image release vY.Y.Y.

  • Update jupyterhub/notebook-images/overlays/build/elyra-notebook-buildconfig.yaml

    • Change the existing image stream tag name from s2i-lab-elyra:vX.X.X to s2i-lab-elyra:vY.Y.Y.
    • Change the git ref from ref: vX.X.X to ref: vY.Y.Y.

  • Create a pull request providing the following information:

    • Title: Update Elyra notebook image to vY.Y.Y
    • Description: add links to the release notes for every version that isn't available yet on Open Data Hub. For example, if the last referenced version in the ODH manifest is 3.9.1 and you are bumping Elyra to 3.10.1, add links to the release notes for 3.10.0 and 3.10.1. This enables reviewers and users to determine what is new and why an upgrade to this release should be considered.

    Example pull request: https://github.com/opendatahub-io/odh-manifests/pull/579

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ai Browse ai catalog → #airflow Browse airflow catalog → #anaconda Browse anaconda catalog → #apache-airflow Browse apache-airflow catalog → #binder Browse binder catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →