## File: README.md

[](https://badge.fury.io/py/elyra)
[](https://anaconda.org/conda-forge/elyra)
[](https://pepy.tech/project/elyra)
[](https://elyra.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/elyra-ai/elyra/issues)
[](https://bestpractices.coreinfrastructure.org/projects/5761)
[](https://gitter.im/elyra-ai/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)

# Elyra

Elyra is a set of AI-centric extensions to JupyterLab Notebooks.

Elyra currently includes the following functionality:

- [Visual Pipeline Editor](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#ai-pipelines-visual-editor)
- [Ability to run a notebook, Python or R script as a batch job](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#ability-to-run-a-notebook-python-or-r-script-as-a-batch-job)
- [Reusable Code Snippets](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#reusable-code-snippets)
- [AI Assistant integration](https://github.com/jupyterlab/jupyter-ai) for AI-powered code assistance in notebook cells. See the [AI Assistant setup guide](docs/source/user_guide/ai-assistant-setup.md) for configuration details.
- [Hybrid runtime support](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#hybrid-runtime-support) based on [Jupyter Enterprise Gateway](https://github.com/jupyter/enterprise_gateway)
- [Python and R script editors with local/remote execution capabilities](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#python-and-r-scripts-execution-support)
- [Python script navigation using auto-generated Table of Contents](https://elyra.readthedocs.io/en/latest/getting_started/overview.html##python-and-r-scripts-execution-support)
- [Python script integrated debugger (Experimental)](https://elyra.readthedocs.io/en/latest/getting_started/overview.html##python-and-r-scripts-execution-support)
- [Notebook navigation using auto-generated outlines using Table of Contents](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#notebook-navigation-using-auto-generated-table-of-contents)
- [Language Server Protocol integration](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#language-server-protocol-integration)
- [Version control using Git integration](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#version-control-using-git-integration)



The [Elyra Getting Started Guide](https://elyra.readthedocs.io/en/latest/getting_started/overview.html) includes more details on these features. A version-specific summary of new features is located on the [releases page](https://github.com/elyra-ai/elyra/releases).

## Try Elyra

#### Using container images

You can also try Elyra by running one of the container images from [Docker Hub](https://hub.docker.com/r/elyra/elyra/tags) or [quay.io](https://quay.io/repository/elyra/elyra?tab=tags):
- `elyra/elyra:latest` has the latest released version installed.
- `elyra/elyra:x.y.z` has a specific version installed.

Note: You can also [build a container image from the `main` branch ("dev build")](https://github.com/elyra-ai/elyra/tree/main/etc/docker/elyra) to try out features that have not been released yet.

To run one of the container images, issue the following command, specifying a tag of your choice.

```
docker run -it -p 8888:8888 elyra/elyra:dev jupyter lab --debug
```

To make a local directory containing your Notebooks (e.g. ${HOME}/opensource/jupyter-notebooks/) available in your
docker container, you can use a mount command similar to the following:

```
docker run -it -p 8888:8888 -v ${HOME}/opensource/jupyter-notebooks/:/home/jovyan/work -w /home/jovyan/work elyra/elyra:dev jupyter lab --debug
```

These should produce output similar to that below, where you can then find the URL to be used to access Elyra in your local browser.

```
    To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://4d17829ecd4c:8888/?token=d690bde267ec75d6f88c64a39825f8b05b919dd084451f82
     or http://127.0.0.1:8888/?token=d690bde267ec75d6f88c64a39825f8b05b919dd084451f82
```

Refer to the [installation documentation](https://elyra.readthedocs.io/en/stable/getting_started/installation.html#docker) for details.

## Installation

For detailed information refer to the [installation documentation](https://elyra.readthedocs.io/en/stable/getting_started/installation.html).

### Prerequisites :
* [Node.js 22](https://nodejs.org/en/)
* [Python 3.10+](https://www.python.org/downloads/)
* [Miniconda](https://docs.conda.io/en/latest/miniconda.html) / [Micromamba](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html) (optional)

### Install current release (for JupyterLab 4.x)

The current release version is displayed at the top of this page.

  - Install from PyPI

    ```bash
    pip3 install --upgrade "elyra[all]"
    ```

  - Install from conda-forge

    ```bash
    conda install -c conda-forge "elyra[all]"
    ```

### Install older release

Installation instructions and JupyterLab support vary by release. Note that a JupyterLab build is required. Installation instructions are located in the [release-specific documentation](https://elyra.readthedocs.io/en/stable/), which can be accessed by selecting a specific version.


  Elyra 4.x (JupyterLab 4.2.5+)

  - Install from PyPI

    ```bash
    pip3 install --upgrade "elyra[all]"
    ```

  - Install from conda-forge

    ```bash
    conda install -c conda-forge "elyra[all]"
    ```



  Elyra 3.7 < 4.0 (JupyterLab 3.x)

  - Install from PyPI

    ```bash
    pip3 install --upgrade "elyra[all]<4.0.0"
    ```

  - Install from conda-forge

    ```bash
    conda install -c conda-forge "elyra[all]<4.0.0"
    ```



  Elyra 3.1 < 3.7 (JupyterLab 3.x)

  - Install from PyPI

    ```bash
    pip3 install --upgrade "elyra[all]>=3.1.0" && jupyter lab build
    ```

  - Install from conda-forge

    ```bash
    conda install -c conda-forge "elyra[all]>=3.1.0" && jupyter lab build
    ```



  Elyra 2.0 < 3.1 (JupyterLab 3.x)

  - Install from PyPI

    ```bash
    pip3 install --upgrade "elyra>=2.0.1" && jupyter lab build
    ```

  - Install from conda-forge

    ```bash
    conda install -c conda-forge "elyra>=2.0.1" && jupyter lab build
    ```


### Verify Installation

Run the following commands to verify the installation. Note that in the example output below the `[version]` placeholder is displayed instead of an actual version identifier, which might change with every release.

```bash
jupyter server extension list
```
Should output:
```
Config dir: /.../.jupyter

Config dir: /.../etc/jupyter
    elyra enabled
    - Validating elyra...
      elyra  OK
    jupyter_lsp enabled
    - Validating jupyter_lsp...
      jupyter_lsp [version] OK
    jupyter_resource_usage enabled
    - Validating jupyter_resource_usage...
      jupyter_resource_usage [version] OK
    jupyter_server_mathjax enabled
    - Validating jupyter_server_mathjax...
      jupyter_server_mathjax  OK
    jupyterlab enabled
    - Validating jupyterlab...
      jupyterlab [version] OK
    jupyterlab_git enabled
    - Validating jupyterlab_git...
      jupyterlab_git [version] OK
    nbclassic enabled
    - Validating nbclassic...
      nbclassic  OK
    nbdime enabled
    - Validating nbdime...
      nbdime [version] OK

Config dir: /.../etc/jupyter
```

NOTE: If you don't see the Elyra server extension enabled, you may need to explicitly enable
it with `jupyter server extension enable elyra`

```bash
jupyter labextension list
```
Should output:
```
JupyterLab [version]
/.../share/jupyter/labextensions
        nbdime-jupyterlab [version] enabled OK
        @jupyter-server/resource-usage [version] enabled OK (python, jupyter-resource-usage)
        @krassowski/jupyterlab-lsp [version] enabled OK (python, jupyterlab_lsp)
        @elyra/code-snippet-extension [version] enabled OK
        @elyra/metadata-extension [version] enabled OK
        @elyra/pipeline-editor-extension [version] enabled OK
        @elyra/python-editor-extension [version] enabled OK
        @elyra/scala-editor-extension [version] enabled OK
        @elyra/r-editor-extension [version] enabled OK
        @elyra/theme-extension [version] enabled OK
        @jupyterlab/git [version] enabled OK (python, jupyterlab-git)

Other labextensions (built into JupyterLab)
   app dir: /.../share/jupyter/lab
```

## Starting Elyra
After verifying Elyra has been installed, start Elyra with:
```bash
jupyter lab
```

## Getting Help

We welcome your questions, ideas, and feedback. Check the [`Getting Help` section in the `Getting Started guide`](https://elyra.readthedocs.io/en/latest/getting_started/getting-help.html) to learn more about the channels you can use to get in touch with us.

## Contributing to Elyra
If you are interested in helping make Elyra better, we encourage you to take a look at our
[Contributing](CONTRIBUTING.md) page,
[Development Workflow](https://elyra.readthedocs.io/en/latest/developer_guide/development-workflow.html)
documentation, and invite you to attend our weekly dev community meetings.

## Meetup with Us!
Our daily and weekly community meeting schedule can be found [here](https://github.com/elyra-ai/community#daily-dev-meetings).

---

## File: cypress/README.md

# Testing
## Frontend Test Development
The frontend of Elyra utilizes a testing framework called [Cypress](cypress.io).
Cypress includes a useful UI for developing and debugging tests, which can be used by
running `npm test-debug` from the root directory of this project. To run tests that output
a detailed log instead of opening the Cypress UI, run `npm test`. Tests can also be run
from the makefile by running `make test`. The spec code for tests is located in
the `cypress/tests/` directory.

---

## File: docs/source/developer_guide/contributing.md

# Contributing to Elyra

Elyra is an open-source project and welcomes contributions from the community be it as code, documentation, scenarios, etc.

Before you start, look at the project:
- [Code of Conduct](https://github.com/elyra-ai/community/blob/main/code-of-conduct.md)
- [Contribution Guidelines](https://github.com/elyra-ai/community/blob/main/contributing.md)

Also, for details on configuring your development environment, see:
- [Setting up your development environment](development-workflow.md)

## Elyra UI

Elyra runs as extensions to the Jupyter ecosystem, thus it's UI is currently implemented as
[JupyterLab widget extensions](https://jupyterlab.readthedocs.io/en/stable/user/extensions.html).

### Contributing to Elyra UI Extensions

#### Existing UI Extensions

The existing extensions are currently available as individual npm packages inside the
[packages](https://github.com/elyra-ai/elyra/tree/main/packages) folder.

```
elyra
 -- packages
    -- code-snippet
    -- metadata
    -- metadata-common
    -- pipeline-editor
    -- python-editor
    -- r-editor
    -- script-editor
    -- services
    -- theme
    -- ui-components
```

#### File Naming and File Creation conventions
Elyra extensions use a separate file for each widget, and keep helper functions and classes in the file with that widget.
A file containing a widget is named after the widget (i.e. `PipelineEditorWidget.tsx` contains the class `PipelineEditorWidget`).

Files containing util functions are named generically after the utils functionality
(i.e. if a util function returned a custom dialog, the file that contained that function would be called `dialog.tsx`).

The index file contains only the definition of the extension class and any other exports that the extension creates.

#### Import sections conventions
Elyra extensions create separate sections for imports from different categories
(i.e. all imports from `@jupyterlab` would be in a separate section from imports from `@lumino`).
Sections are separated by a blank line. Each section is alphabetized by the name of packages.


### UI Tests
Elyra uses two types of frontend tests: integration tests (which use [cypress](https://docs.cypress.io/))
and unit tests (which use [jest](https://jestjs.io/docs/en/getting-started)).

#### UI Integration tests
Before running integration tests, docker needs to be installed and running on your machine.
There are two ways to run the integration tests:
* To only see the output logs from all of the integration tests,
run `make test-integration` from the root directory.
* To run or debug specific tests,
run `make test-integration-debug`. This will open an interactive UI tool for writing and debugging individual test files.

Elyra's integration tests automatically start JupyterLab and visit / interact with pages through cypress API calls.
The tests use the cypress API to check for the existence of various buttons and visual elements.
Refer to the [cypress API](https://docs.cypress.io/api/api/table-of-contents.html) for more details.

New integration tests can be added to `cypress/tests`.

#### UI Unit tests
To run all of the unit tests, use `make test-ui-unit` from the root directory. To run the unit tests for a specific Elyra package, simply run `jest` or `npm run test` from that package's directory (under `packages/`). To turn on the watch mode just run `jest --watch` or `npm run test --watch`.

Elyra's unit tests test the various classes and objects used by Elyra extensions. Refer to the [jest API](https://jestjs.io/docs/en/getting-started) for more details.

To add unit tests for a package that doesn't have tests set up, some configuration files are required. In the directory for the package being tested, add a file titled `jest.config.js` that contains the following:
```
module.exports = require('../../testutils/jest.config');
```
Then, in the `package.json`, add the following under `'scripts'`:
```
"test": "jest",
"build:test": "tsc --build tsconfig.test.json",
```
And the following under `'dev_dependencies'`:
```
"@jupyterlab/testutils": "4.4.2",
"@types/jest": "^29.2.0",
"jest": "^29.2.0",
"jest-raw-loader": "^1.0.1",
"ts-jest": "^29.3.2",
"ts-loader": "^6.2.1",
"typescript": "~5.5.4",
```
Create a file `tsconfig.test.json` that contains:
```
{
  "extends": "../../tsconfigbase.test",
  "include": ["src/*", "test/*"],
  "references": []
}
```

Finally, create a folder called `test` in the `src` directory of the package being tested, and add tests using the file extension `.spec.ts`.

## Elyra Backend

Elyra runs as extensions to the Jupyter ecosystem, thus it's backend is currently implemented as
[JupyterServer server extensions](https://jupyter-server.readthedocs.io/en/latest/developers/extensions.html)
and exposed as REST APIs to frontend clients.


### Elyra backend services

The existing services are currently available as individual python modules inside the
[elyra](https://github.com/elyra-ai/elyra/tree/main/elyra) folder.

```
elyra
 -- packages
    -- airflow
    -- api
    -- cli
    -- contents
    -- kfp
    -- metadata
    -- pipeline
    -- templates
    -- util
```

### Backend tests
To run all server tests, use `make test-server` from the root directory. There are also two ways to run only specific backend tests:

* To run all tests in a specific directory or file, run `pytest [resource]` where `resource` is the relative path to a directory or file inside the [server tests](https://github.com/elyra-ai/elyra/tree/main/elyra/tests) folder.
* To run specific tests by function name, run `pytest -k [test_function1 test_function2 ...]`


## Elyra documentation

The Elyra documentation's source is stored in the [Elyra repository](https://github.com/elyra-ai/elyra/tree/main/docs) and hosted at [Elyra's Read the Docs](https://elyra.readthedocs.io/). The documentation is written in [Markdown](https://www.sphinx-doc.org/en/master/usage/markdown.html) and built using [Sphinx](https://www.sphinx-doc.org/en/master/).


### Contributing to the Elyra documentation

To contribute content to the Elyra documentation follow these steps:

1. Fork the [repository](https://github.com/elyra-ai/elyra).

2. Clone your fork.

   ```
   git clone https://github.com/git-id-or-org/elyra
   ```

   The documentation assets are located in the `/docs` directory.

   - To add a new document create a new markdown file in the appropriate section subdirectory (e.g. `/docs/source/getting_started`) and add an entry to that section in `/docs/source/index.rst`.
   - To update an existing document edit the corresponding markdown file.
   - Place new or updated images in the `/docs/source/images` directory. For example, images for the _pipeline components_ topic in the _user guide_ are stored in `docs/source/images/user_guide/pipeline-components/`. `PNG` is the recommended format.

3. Build the documentation assets locally.

   In the repository's _root directory_ (not the `/docs` directory) run

   ```
   make docs
   ```

4. Review the build output and verify that your updates introduced no warnings or errors.

5. Review the updated documentation assets.
    1. Navigate to the `/docs/_build/html/` directory.
    1. Open `index.html` in a web browser.

6. Commit your updates to a new branch and open a pull request.

---

## File: docs/source/developer_guide/development-workflow.md

## Development Workflow
This section describes the steps necessary to build Elyra in a development environment.

#### Requirements

* [Python 3 Miniconda](https://docs.conda.io/en/latest/miniconda.html)
* [Node.js 22](https://nodejs.org/en/)
* [Yarn](https://yarnpkg.com/lang/en/docs/install)
* [GNU Make](https://www.gnu.org/software/make/)

### Setting up your development environment

* Install Miniconda
Download and install a [Python 3 version of Miniconda](https://docs.conda.io/en/latest/miniconda.html) according to your Operating System

* Create a new Python environment using a version that is [supported by Elyra](../getting_started/installation.html#prerequisites).

    ```
    conda create -n <env-name> python
    ```

    The Python version of your environment will match the miniconda version you installed. You can override the default by explicitly setting `python=3.10`, for example.

* Activate the new environment

    ```
    conda activate <env-name>
    ```

* Verify your miniconda environment

    ```
    python --version # should yield a version that is supported by Elyra
    which python     # displays current `python` path
    pip3 --version   # should be a recent version to avoid build issues
    which pip3       # displays current `pip` path
    ```
    Python path must be under miniconda envs folder.
    Confirm pip3 location matches where miniconda is installed.

* Install a version of Node.js that is [supported by Elyra](../getting_started/installation.html#prerequisites).

    ```
    conda install -y -c conda-forge/label/main nodejs
    ```
* Verify node is installed correctly

    ```
    node --version
    ```

* Install Yarn

    ```
    conda install -y -c conda-forge/label/main yarn
    ```
* Verify yarn is installed correctly

    ```
    yarn --version
    ```

* Install GNU Make

    Refer to the following link for installation instructions:
    [GNU Make](https://www.gnu.org/software/make/)

    To verify the installation, run `make`.
    If you have yet to [set up the repository](#setting-up-your-development-environment), you should see a message like the following:
    ```
    make: *** No targets specified and no makefile found.  Stop.
    ```
    Once the repository is set up, running `make` from that location should display the available tasks that are listed in the [Build & Installation](#build-installation) section below.

### Setting up your Elyra Github repository

* Fork the [Elyra Github repository](https://github.com/elyra-ai/elyra) (if you haven't already)

* Make a local copy of Elyra fork
    ```
    git clone https://github.com/<your-github-id>/elyra.git
    cd elyra
    ```
* Set `upstream` as described in the [GitHub documentation](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork)

### Building
Elyra is divided in two parts, a collection of Jupyter Notebook backend extensions,
and their respective JupyterLab UI extensions. Our JupyterLab extensions are located in our `packages`
directory.

#### Build & Installation

Elyra uses `make` to automate some of the development workflow tasks.

Issuing a `make` command with no task specified will provide a list of the currently supported tasks.

```bash
$ make

build-ui-dev                   Build UI packages for development
build-ui-prod                  Build UI packages for production
clean                          Make a clean source tree and uninstall extensions
container-images               Build all container images
docs                           Build docs
elyra-image-env                Creates a conda env consisting of the dependencies used in images
install-all-dev                Build and install for development, including examples
install-all-prod               Build and install for production, including examples
install-dev                    Build and install for development
install-examples               Install example pipeline components
install-prod                   Build and install for production
install-server                 Build and install backend
lint                           Run linters
package-ui-dev                 Package UI for development
package-ui-prod                Package UI for production
publish-container-images       Publish all container images
release                        Build wheel file for release
test                           Run all tests (backend, frontend and cypress integration tests)
watch                          Watch packages. For use alongside jupyter lab --watch
```

You can build and install all Elyra packages with:

```bash
make clean install-prod
```

You can check that the notebook server extension was successfully installed with:
```bash
jupyter serverextension list
```

You can check that the JupyterLab extension was successfully installed with:
```bash
jupyter labextension list
```

> NOTE:
When switching between Elyra major versions, it is recommended to clean your JupyterLab environment before a build.
The `clean-jupyterlab` removes your JupyterLab packages and completely deletes your Jupyter workspace.
Make sure to backup any important data in your environment before running the script.
To clean your environment and install the latest JupyterLab:
`etc/scripts/clean-jupyterlab.sh`
To specify a JupyterLab version `x.y.z` to be installed:
`etc/scripts/clean-jupyterlab.sh --version x.y.z`

#### Parallel Development with @elyra/pipeline-editor

You can install Elyra using a local build of @elyra/pipeline-editor with:
```bash
make clean install-dev
```

### Back-end Development
After making code changes to the back-end, you can re-build Elyra's Python package with:

```bash
make install-server
```

This command builds and installs the updated  Python package independently, skipping any UI component build.

Restart JupyterLab to pick up the new code changes.

### Front-end Incremental Development

Elyra supports incremental development using `--watch`. This allows you to make code changes to
front-end packages and see them without running `make install` again.

After installation run the following to watch for code changes and rebuild automatically:
```bash
make watch
```

Then in a separate terminal, using the same Python environment, start JupyterLab in watch mode:
```bash
jupyter lab --watch
```

When in watch mode JupyterLab will watch for changes in the build of each package and rebuild.
To see your changes just refresh JupyterLab in your browser.

> NOTE: JupyterLab watch mode will not pick up changes in package dependencies like `services`.
So when making changes to services you will need to stop and restart `jupyter lab --watch` and
not just refresh your browser.

### Building the Elyra Container Image

Elyra's container image can be built in two ways (production and development):

Development:
```bash
make elyra-image
```
By default, the command above will build a container image (development) with the changes that exist in your local branch.


Production:
From main branch:
```bash
make elyra-image TAG=3.7.0
```
or after checking out a git tag e.g. `git checkout tags/v3.7.0`
```bash
make elyra-image
```
In order to build from a particular release (production), you can pass a `TAG` parameter to the make command
or you can checkout the respective tagged release and omit the `TAG` parameter.

Official container images are published on [Docker Hub](https://hub.docker.com/r/elyra/elyra/tags)
and [quay.io](https://quay.io/repository/elyra/elyra?tab=tags).

### Developing Elyra against the Jupyterlab source repo

Sometimes it is useful to develop Elyra against a local build of Jupyterlab. To use a local build of Jupyterlab use the
following steps in the same python environment.

1. Uninstall any pip installations of Jupyterlab. You can use `etc/scripts/clean-jupyterlab.sh --version dev` as
   mentioned above with `--version dev` to not reinstall Jupyterlab at the end of the script.

2. Build your local repo of Jupyterlab, step-by-step instructions can be found in the
   [Jupyterlab documentation](https://jupyterlab.readthedocs.io/en/latest/developer/contributing.html#installing-jupyterlab).
   Uninstalling in the previous step will also wipe any previous installations of a local build.

3. `cd` to the `builder/` directory in your Jupyterlab repo and run `yarn link`. The Elyra `Makefile` will use this yarn
   link in step 6.

4. In your Elyra repo, uncomment the following line in `tsconfig.base.json` to tell Typescript to use the local
   Jupyterlab packages when building:

    ```"paths": { "@jupyterlab/*": ["../jupyterlab/packages/*"] },```

5. Comment out `jupyterlab` and `jupyterlab-lsp` in the `install_requires` section of `setup.py` in your Elyra repo.
   This will prevent Jupyterlab from being pip installed during the Elyra build.
   Note: `jupyterlab-lsp` also pip installs Jupyterlab when installed

6. Run `make install-dev` to install Elyra using the linked `@jupyterlab/builder` from step 3.

7. You can now start Jupyterlab by running `jupyter lab --dev-mode --extensions-in-dev-mode`, this will automatically
   watch for changes in the Jupyterlab repo. To also watch for changes in Elyra run `make watch` in a separate terminal
   in the same Python environment.

When you want to switch back to developing Elyra against a Jupyterlab release, you just have to undo the comments in
steps 4 and 5 and rebuild with `make clean install`

## Analyzing automated test failures

The Elyra GitHub repository is configured to run automated tests whenever a pull request is opened. These tests include static [code quality analysis](https://github.com/elyra-ai/elyra/blob/main/.github/workflows/codeql-analysis.yml) and [UI, server, and integration tests](https://github.com/elyra-ai/elyra/blob/main/.github/workflows/build.yml).

The test results can be accessed from the pull request or the _actions_ tab. If the test log does not include enough details to diagnose failures, download and review test artifacts that might have been generated.

### Accessing test artifacts

1. Open the [Elyra repository actions panel](https://github.com/elyra-ai/elyra/actions) (`https://github.com/elyra-ai/elyra/actions`).
1. Locate the failing workflow.
  
1. Open the workflow.
1. Click the 'home' (summary) button.
  
1. Locate the 'Artifacts' section. If present, it should contain a download link.
  
1. Download the archive, extract it, and review the artifacts.

---

## File: docs/source/developer_guide/issue-management.md

## Issues Workflow
This section describes how Elyra issues are organized. 

### Labels used for tagging Issues and Pull Requests

| Keyword | Tags |
| --- | --- |
| Area | ![][backend] ![][community] ![][documentation] ![][front-end] |
| Impact | ![][blocker] ![][blocked]  ![][do not merge] ![][needs doc updates] ![][needs example updates] ![][needs release note] |
| Help Wanted | ![][good first issue] ![][good first project] ![][help wanted] |
| Mentoring Program | ![][jumpstart] ![][rcos] |
| Kind | ![][accessibility] ![][bug] ![][config] ![][enhancement] ![][investigate] ![][no functionality change] ![][question] ![][task] ![][user error] |
| Feedback | ![][isv] ![][proposal] ![][user] |
| External | ![][jupyter enterprise gateway] ![][jupyter server] ![][jupyter hub] ![][jupyter lab 1.x] ![][jupyter lab 2.x] ![][jupyter lab 3.x] ![][jupyter lab 4.x] ![][upstream] |
| Component | ![][application utils] ![][binder] ![][build] ![][cli tools] ![][code snippets] ![][component registry] ![][content parser] ![][docker] ![][git] ![][install] ![][lsp] ![][metadata editor] ![][metadata] ![][performance] ![][pipeline editor visual studio] ![][pipeline editor] ![][pipeline runtime] ![][pipeline validation] ![][python editor] ![][r editor] ![][resource usage] ![][s3] ![][submit notebook button] ![][submit script button] ![][test] ![][toc] ![][troubleshooting] |
| Platform | ![][open data hub] ![][OpenShift] ![][Pipeline Airflow] ![][Pipeline Kubeflow] ![][Pipeline Local] ![][Windows] |
| Browser | ![][chrome] ![][firefox] ![][safari] |
| Inactive | ![][cannot reproduce] ![][deferred] ![][duplicate] ![][invalid] ![][obsolete] ![][wontfix] |
| Status | ![][needs discussion] ![][needs triage] ![][needs update] ![][waiting for author] ![][work in progress] |
| Priority | ![][high] ![][low] ![][normal] ![][stretch] |



[isv]: https://img.shields.io/badge/-isv-d4c5F9?style=flat
[proposal]: https://img.shields.io/badge/-proposal-d4c5F9?style=flat
[user]: https://img.shields.io/badge/-user-d4c5F9?style=flat

[backend]: https://img.shields.io/badge/-backend-c5def5?style=flat
[community]: https://img.shields.io/badge/-community-c5def5?style=flat
[documentation]: https://img.shields.io/badge/-documentation-c5def5?style=flat
[front-end]: https://img.shields.io/badge/-front_end-c5def5?style=flat

[blocked]: https://img.shields.io/badge/-blocked-d93f0b?style=flat
[blocker]: https://img.shields.io/badge/-blocker-d93f0b?style=flat
[do not merge]: https://img.shields.io/badge/-do_not_merge-d93f0b?style=flat
[needs doc updates]: https://img.shields.io/badge/-needs_doc_updates-d93f0b?style=flat
[needs example updates]: https://img.shields.io/badge/-needs_example_updates-d93f0b?style=flat
[needs release note]: https://img.shields.io/badge/-needs_release_note-d93f0b?style=flat

[good first issue]: https://img.shields.io/badge/-good_first_issue-84e251?style=flat
[good first project]: https://img.shields.io/badge/-good_first_project-84e251?style=flat
[help wanted]: https://img.shields.io/badge/-help_wanted-84e251?style=flat

[jumpstart]: https://img.shields.io/badge/-jumpstart-84e251?style=flat
[rcos]: https://img.shields.io/badge/-rcos-84e251?style=flat

[accessibility]: https://img.shields.io/badge/-accessibility-ea81a2?style=flat
[bug]: https://img.shields.io/badge/-bug-ea81a2?style=flat
[config]: https://img.shields.io/badge/-config-ea81a2?style=flat
[enhancement]: https://img.shields.io/badge/-enhancement-ea81a2?style=flat
[investigate]: https://img.shields.io/badge/-investigate-ea81a2?style=flat
[no functionality change]: https://img.shields.io/badge/-no_functionality_change-ea81a2?style=flat
[question]: https://img.shields.io/badge/-question-ea81a2?style=flat
[task]: https://img.shields.io/badge/-task-ea81a2?style=flat
[user error]: https://img.shields.io/badge/-user_error-ea81a2?style=flat

[jupyter enterprise gateway]: https://img.shields.io/badge/-jupyter_enterprise_gateway-orange?style=flat
[jupyter server]: https://img.shields.io/badge/-jupyter_server-orange?style=flat
[jupyter hub]: https://img.shields.io/badge/-jupyter_hub-orange?style=flat
[jupyter lab 1.x]: https://img.shields.io/badge/-jupyter_lab_1x-orange?style=flat
[jupyter lab 2.x]: https://img.shields.io/badge/-jupyter_lab_2x-orange?style=flat
[jupyter lab 3.x]: https://img.shields.io/badge/-jupyter_lab_3x-orange?style=flat
[jupyter lab 4.x]: https://img.shields.io/badge/-jupyter_lab_4x-orange?style=flat
[upstream]: https://img.shields.io/badge/-upstream-orange?style=flat

[application utils]: https://img.shields.io/badge/-application_utils-blue?style=flat
[binder]: https://img.shields.io/badge/-binder-blue?style=flat
[build]: https://img.shields.io/badge/-build-blue?style=flat
[cli tools]: https://img.shields.io/badge/-cli_tools-blue?style=flat
[code snippets]: https://img.shields.io/badge/-code_snippets-blue?style=flat
[component registry]: https://img.shields.io/badge/-component_registry-blue?style=flat
[content parser]: https://img.shields.io/badge/-content_parser-blue?style=flat
[docker]: https://img.shields.io/badge/-docker-blue?style=flat
[git]: https://img.shields.io/badge/-git-blue?style=flat
[install]: https://img.shields.io/badge/-install-blue?style=flat
[lsp]: https://img.shields.io/badge/-lsp-blue?style=flat
[metadata editor]: https://img.shields.io/badge/-metadata_editor-blue?style=flat
[metadata]: https://img.shields.io/badge/-metadata-blue?style=flat
[performance]: https://img.shields.io/badge/-performance-blue?style=flat
[pipeline editor visual studio]: https://img.shields.io/badge/-pipeline_editor_visual_studio-blue?style=flat
[pipeline editor]: https://img.shields.io/badge/-pipeline_editor-blue?style=flat
[pipeline runtime]: https://img.shields.io/badge/-pipeline_runtime-blue?style=flat
[pipeline validation]: https://img.shields.io/badge/-pipeline_validation-blue?style=flat
[python editor]: https://img.shields.io/badge/-python_editor-blue?style=flat
[r editor]: https://img.shields.io/badge/-r_editor-blue?style=flat
[resource usage]: https://img.shields.io/badge/-resource_usage-blue?style=flat
[s3]: https://img.shields.io/badge/-s3-blue?style=flat
[submit notebook button]: https://img.shields.io/badge/-submit_notebook_button-blue?style=flat
[submit script button]: https://img.shields.io/badge/-submit_script_button-blue?style=flat
[test]: https://img.shields.io/badge/-test-blue?style=flat
[toc]: https://img.shields.io/badge/-toc-blue?style=flat
[troubleshooting]: https://img.shields.io/badge/-troubleshooting-blue?style=flat

[Open Data Hub]: https://img.shields.io/badge/-open_data_hub-darkblue?style=flat
[OpenShift]: https://img.shields.io/badge/-openshift-darkblue?style=flat
[Pipeline Airflow]: https://img.shields.io/badge/-pipeline_airflow-darkblue?style=flat
[Pipeline Kubeflow]: https://img.shields.io/badge/-pipeline_kubeflow-darkblue?style=flat
[Pipeline Local]: https://img.shields.io/badge/-pipeline_local-darkblue?style=flat
[Windows]: https://img.shields.io/badge/-windows-darkblue?style=flat

[chrome]: https://img.shields.io/badge/-chrome-006b75?style=flat
[firefox]: https://img.shields.io/badge/-firefox-006b75?style=flat
[safari]: https://img.shields.io/badge/-safari-006b75?style=flat

[needs discussion]: https://img.shields.io/badge/-needs_discussion-0e8a16?style=flat
[needs triage]: https://img.shields.io/badge/-needs_triage-0e8a16?style=flat
[needs update]: https://img.shields.io/badge/-needs_update-0e8a16?style=flat
[waiting for author]: https://img.shields.io/badge/-waiting_for_author-0e8a16?style=flat
[work in progress]: https://img.shields.io/badge/-work_in_progress-0e8a16?style=flat

[cannot reproduce]: https://img.shields.io/badge/-cannot_reproduce-grey?style=flat
[deferred]: https://img.shields.io/badge/-deferred-grey?style=flat
[duplicate]: https://img.shields.io/badge/-duplicate-grey?style=flat
[invalid]: https://img.shields.io/badge/-invalid-grey?style=flat
[obsolete]: https://img.shields.io/badge/-obsolete-grey?style=flat
[wontfix]: https://img.shields.io/badge/-wontfix-grey?style=flat

[high]: https://img.shields.io/badge/-high-d73a4a?style=flat
[low]: https://img.shields.io/badge/-low-FEF2C0?style=flat
[normal]: https://img.shields.io/badge/-normal-00cc00?style=flat
[stretch]: https://img.shields.io/badge/-stretch-FEF2C0?style=flat

---

## File: docs/source/developer_guide/logging.md

## Logging

This section will go over how to properly log messages in Elyra

### Pipelines Extension Logging

#### Python 
The Kubeflow Pipelines extension uses a custom IPython handler to handle interactions with 
the Notebook server instance. By extending the IPython handler we are able to leverage Tornado logging 
since the Jupyter Notebook server uses Tornado as its web framework. 
 
###### Pipelines -> IPythonHandler -> Tornado.log -> lib/logging 

##### When to add log messages
Use INFO level for routines(functions), like when handling requests or state changes and DEBUG for troubleshooting and diagnostics.
```
  def foo (arg):
    self.debug.info(f"doing something with this {arg}")
    do something with arg
    self.log.info(f"Result of arg is ... {result}")
```    
Use WARNING level when it is important, but not an error, for example, when a user attempts to login with an incorrect password
```
  def authenticate(username, password):
    if username and password doesn't work:
        self.log.warn(f"Failed login attempt from {username} : {password}")
```     
Use ERROR level when something is actually wrong, like when an exception is thrown, IO operation failure or connectivity 
issue (e.g. Minio client cant connect to endpoint)
``` 
    try:
        if not minio_client.bucket_exists(bucket_name):
            minio_client.make_bucket(bucket_name)
    except ResponseError:
        self.log.error("Minio error", exc_info=True)
```
##### Adding log messages
Add log messages by inserting statements as follows:
`self.log.debug("insert debug message", *args, *kwargs)`
Use the appropriate logging levels when inserting specific messages e.g. log.debug, log.warn etc. 

The log output will be color coded, legend as follows:

| Log Level | Color | 
|-----------|-------|
| DEBUG     | Blue  | 
| INFO      | Green |
| WARNING   | Yellow |
| ERROR     | Red   |
| CRITICAL  | Magenta |

##### Setting the Log Level
The default Log Level is set to INFO. If you need to increase log verbosity, you can start Jupyterlab with 
the `--debug` option for example:  
```
    jupyter lab --debug <other options>
```
If too verbose, you can filter out only the most important messages by directly setting the value
with `--log-level=` using one of the following `{DEBUG|INFO|WARN|ERROR|CRITICAL}`.  For example:
```
    jupyter lab --log-level=WARN <other options>
```
indicates that only log levels of `WARN` (or higher: `ERROR` and `CRITICAL`) will be logged.

#### TypeScript / JS

Use console.log(), console.warn() to output debug messages

---

## File: docs/source/developer_guide/metadata.md

# Metadata Services
While building enterprise extensions for the Elyra project we identified a requirement to integrate with external runtimes, data sources, and other
components that require additional metadata in order to connect
to these external components.

The **Metadata Service** provides a generic service that can be used to store metadata
that can be easily integrated with Elyra backend and/or frontend components.

## Metadata Service Overview
The core design of the metadata service is that it be primarily driven by _schema_.  That is, all metadata _instances_ must adhere to a schema.  Metadata schema are organized, both logically and physically, into _schemaspaces_ where a given schemaspace _contains_ a set of schemas and each instance of a schemaspace's schemas are co-located relative to the configured storage mechanism.

Externally, both in its persisted and serialized forms, a metadata instance is represented as a JSON structure.   Within the metadata service, however, this JSON structure is represented as a `Metadata` class instance.  The `Metadata` class can be extended via subclassing to enable custom behaviors that are triggered following load operations and/or before and after create, update and delete operations on a _per schema_ level.  That is, it is the respective schema that identifies the name of the class to use as the internal representation.

As previously mentioned, the metadata service has a pluggable storage mechanism.  By default, metadata instances are stored within the file system.  However, alternative storage mechanisms can be configured for, say, NoSQL DB or traditional, table-oriented, databases.

Schemaspaces and their schemas can be dynamically introduced via entrypoints.  When the system starts, it will identify applicable entrypoints and load their corresponding objects to seed the list of _configured_ schemaspaces and schemas.

### Metadata Components
The Elyra metadata service is composed of several components, many of which were previously referenced, and each responsible for a different task.  


The following briefly describes each of those components.

- **[Schemaspace](https://github.com/elyra-ai/elyra/blob/1536495fb61a0f4b19f658360718346e9cb4f110/elyra/metadata/schema.py#L213)**: This is the top-most component and represents a _container_ of schemas.  It has `id` (UUID), `name` (symbolic name), `display_name` (descriptive name), and `description` properties, along with a set of schemas.  Instances of a schemaspace's schemas are logically grouped together in whatever storage model is in use.  Schemaspaces are discovered and loaded via entrypoints.
- **[SchemasProvider](https://github.com/elyra-ai/elyra/blob/1536495fb61a0f4b19f658360718346e9cb4f110/elyra/metadata/schema.py#L299)**: A schemas provider is a class that is also discovered and loaded via entrypoints.  Its purpose is to provide schemas.  Since a schemaspace is always referenced within the schema, a given `SchemasProvider` instance _could_ provide schemas corresponding to different schemaspaces although a schemas provider will typically correspond to a single schemaspace.
- **[Metadata](https://github.com/elyra-ai/elyra/blob/1536495fb61a0f4b19f658360718346e9cb4f110/elyra/metadata/metadata.py#L31)**:  This class is used internally to house the JSON instance data.  It can be subclassed to enable applications to hook persistence and retrieval operations.
- **[SchemaManager](https://github.com/elyra-ai/elyra/blob/1536495fb61a0f4b19f658360718346e9cb4f110/elyra/metadata/schema.py#L41)**: This class is a singleton instance that loads and houses all schemas organized by their respective `Schemaspace`.  All schemas provided by a `SchemasProvider` must adhere to the _meta-schema_ and the `SchemaManager` is responsible for validating provided schemas.
- **[MetadataManager](https://github.com/elyra-ai/elyra/blob/1536495fb61a0f4b19f658360718346e9cb4f110/elyra/metadata/manager.py#L34)**: This class operates on `Metadata` instances. Prior to each persistence operation, and following each load, the `MetadataManager` is responsible for validating the instance against its corresponding schema, which it obtains from the `SchemaManager`.  It is also responsible for driving the persistence and retrieval of instances against the configured `MetadataStore`.
- **[MetadataStore](https://github.com/elyra-ai/elyra/blob/1536495fb61a0f4b19f658360718346e9cb4f110/elyra/metadata/storage.py#L42)**: This class is an abstract base class that defines methods relative to the persistence and retrieval of metadata instances.  It can be subclassed such that each subclass typically represents an alternative storage mechanism.  The system provides a [`FileMetadataStore`](https://github.com/elyra-ai/elyra/blob/a31afee97c716feacec1b43feee66b119de71397/elyra/metadata/storage.py#L174) instance - which is configured as the default storage mechanism.
- **REST/CLI API**: The metadata service entities can be manipulated via a REST API consisting of `/elyra/metadata`, `/elyra/schema` and `/elyra/schemaspace` endpoints.  Similar capabilities can also be performed using the command-line tool (CLI) `elyra-metadata`.

#### Metadata Services structure using the default file system store

The default implementation is for the metadata services to store metadata files in the file system, grouped
by directories according to the metadata instance's _schemaspace_.

The root directory for metadata is relative to the 'Jupyter Data directory' (e.g. `jupyter --data-dir`)

```
/Users/jovyan/Library/Jupyter/metadata/
```

Each instance of metadata is then stored in a child directory representing the instance's schemaspace.

As an example, `runtimes` is the schemaspace for runtime metadata instances that reside in the following directory:

```
/Users/jovyan/Library/Jupyter/metadata/runtimes
```

The contents of this folder would then include multiple JSON files, each associated with a schema corresponding to the desired runtime platform.

For example, the following contains runtime metadata instances for two runtime platforms, airflow and kfp,
where each runtime has 1 and 2 runtime instances defined, each of which adheres to their respective schemas.

```
/Users/jovyan/Library/Jupyter/metadata/runtimes/airflow-cloud.json
/Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-fyre.json
/Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-qa.json
```

And each metadata file contains information similar to the following (note the reference to its schema_name):

```json
{
  "display_name": "Kubeflow Pipeline - Fyre",
  "schema_name": "kfp",
  "metadata": {
    "api_endpoint": "http://weakish1.fyre.ibm.com:32488/pipeline",
    "api_username": "username@email.com",
    "api_password": "mypassword",
    "cos_endpoint": "http://weakish1.fyre.ibm.com:30427",
    "cos_username": "minio",
    "cos_password": "minio123",
    "cos_bucket": "test-bucket",
    "tags": [
      "kfp", "v1.1"
    ]
  }
}
```

#### Metadata Service REST API

A REST API is available for easy integration with frontend components:

Retrieve all metadata for a given schemaspace:

```REST
GET /elyra/metadata/<schemaspace>
```

Retrieve a given metadata resource from a given schemaspace:

```REST
GET /elyra/metadata/<schemaspace>/<resource>
```

#### Metadata Client Tool

Users can easily manipulate metadata via the Client Tool

```bash
elyra-metadata list runtimes
```

```
Available metadata instances for runtimes (includes invalid):

Schema   Instance       Resource  
------   --------       -------- 
kfp      kfp-fyre       /Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-fyre.json
kfp      kfp-qa         /Users/jovyan/Library/Jupyter/metadata/runtimes/kfp-qa.json
airflow  airflow-cloud  /Users/jovyan/Library/Jupyter/metadata/runtimes/airflow-cloud.json
```


#### Metadata APIs
A Python API is also available for accessing and manipulating metadata.  This is accomplished using the `MetadataManager` along with a corresponding storage class.  The default storage class is `FileMetadataStore`.

```Python
from elyra.metadata import MetadataManager, FileMetadataStore

metadata_manager = MetadataManager(schemaspace="runtimes",
                                   store=FileMetadataStore(schemaspace='runtimes'))

runtime_configuration = metadata_manager.get('kfp')

if not runtime_configuration:
    raise RuntimeError("Runtime metadata not available.")

api_endpoint = runtime_configuration.metadata['api_endpoint']
api_username = runtime_configuration.metadata['api_username']
api_password = runtime_configuration.metadata['api_password']
cos_endpoint = runtime_configuration.metadata['cos_endpoint']
cos_username = runtime_configuration.metadata['cos_username']
cos_password = runtime_configuration.metadata['cos_password']
bucket_name = runtime_configuration.metadata['cos_bucket']
tags = runtime_configuration.metadata['tags']

```

## Bring Your Own Schemas
The metadata service enables the ability to introduce your own schemaspaces and bring custom schemas to existing schemaspaces via [entrypoints](https://entrypoints.readthedocs.io/en/latest/).  It is necessary to introduce Schemaspaces when developing a new extension or adding functionality in which it makes sense to group instances of the functionality together.  For example, Code Snippets consist of their own Schemaspace and, currently, are based on a single schema.
Applications extending _existing_ functionality do not need to introduce a new Schemaspace, since one will already exist, but must be able to introduce their custom schema.  [Introducing your own _runtime_](../developer_guide/pipelines.html#pipeline-processor-customization) would be an example where a Schemaspace is not required.  In this case, only a _`SchemasProvider`_ would be necessary.

### Schemaspace
A Schemaspace instance is used to represent a set of schemas.  Custom schemaspace classes must derive from the `Schemaspace` base class and are responsible for providing basic information like `id` (UUID), `name` (used as a symbolic name for lookups), `display_name` (a human-readible, descriptive name) and `description`.  They also contain a dictionary of JSON schemas, indexed by schema name.  However, the dictionary is empty at construction and filled during system initialization via the `SchemasProvider` mechanism.  By convention, schemaspace names are plural, while schema names are singular.

#### Registration
Schemaspaces are discovered via the entrypoints functionality.  When the `SchemaManager` starts, it fetches all entrypoint instances named by the group `'metadata.schemaspaces'`, and uses the entrypoint data to load each found instance.
Entrypoints are registered in the package's setup configuration.  

Here is an example of the schemaspace entrypoints defined in Elyra:
```
entry_points={
  'metadata.schemaspaces': [
    'runtimes = elyra.metadata.schemaspaces:Runtimes',
    'runtimes-images = elyra.metadata.schemaspaces:RuntimeImages',
    'code-snippets = elyra.metadata.schemaspaces:CodeSnippets',
    'component-registries = elyra.metadata.schemaspaces:ComponentRegistries'
  ]
}
```
As you can see, each entry has a _name_ that equates to a module and object/class.

### SchemasProvider
As mentioned above, `Schemaspace` instances do not know their set of schemas at the time of their construction.  Instead, the `SchemaManager` loads a different entrypoints group following the loading of schemaspaces named `'metadata.schemas_providers'`.  These entrypoints identify a `SchemasProvider` object and this object is responsible for returning schemas when called upon by the `SchemaManager`.
`SchemasProvider` instances are concrete instances of the abstract base class `SchemasProvider`, which requires an implementation of the `get_schemas()` method:
```python
class SchemasProvider(ABC):
    """Abstract base class used to obtain schema definitions from registered schema providers."""

    @abstractmethod
    def get_schemas(self) -> List[Dict]:
        """Returns a list of schemas"""
        pass
```
When called, the `SchemasProvider` instance is responsible for producing a list of schemas.  These schemas must conform to the metadata service's _meta-schema_ and, as a result, will include the schemaspace name and schemaspace id to which they correspond, in addition to a `name` property representing that schema's name.  The `SchemaManager` uses this meta-information to confirm the schemaspace has been loaded and register that schema with the schemaspace following its successful validation against the meta-schema.

Some `SchemasProvider` instances may have schema information which is dynamic, based on the active configuration of the system.  In such cases, they are responsible for returning information pertinent to the current configuration.  For example, a schema may use an `enum` that holds a set of values pertinent to the system's configuration.  In this case, the `SchemasProvider` may alter its static schema to dynamically update its enumerated property corresponding to the current runtime environment.  Note however, that care should be taken when dynamically altering schema since it could lead to instances that will no longer load because they (now) fail schema validation if the contents of the enum no longer satisfy previous instance values.  Of course, this may very well be proper behavior, depending on the intention.

#### Validators
A [_validator_](https://python-jsonschema.readthedocs.io/en/stable/creating/) is used
to validate instances of a given schema.  Within the metadata service, instances
are validated whenever they are retrieved, and immediately prior to their persistence.

The `SchemasProvider` base class provides an instance of the `Draft7Validator` which should
be sufficient for most schemas.  Implementations requiring their own validator, or that wish to
extend the `Draft7Validator`, must override the `get_validator()` method in their
 subclass of `SchemasProvider`:
```python
    def get_validator(self, schema: dict) -> Any:
        """Returns an instance of a validator used to validate instances of the given schema.

        The default implementation uses the `Draft7Validator`.  SchemasProviders requiring
        the use of a different validator must override this method.
        """
        return Draft7Validator(schema, format_checker=draft7_format_checker)
```

#### Registration
Like `Schemaspaces`, `SchemasProvider` implementations are discovered via the entrypoints functionality.  When the `SchemaManager` starts, and after it loads all `Schemaspaces`, it fetches all entrypoint instances named by the group `'metadata.schemas_providers'`, and uses the entrypoint data to load each found instance.

Here is an example of the schemas_providers entrypoints defined in Elyra:
```
entry_points={
  'metadata.schemas_providers': [
    'runtimes = elyra.metadata.schemasproviders:RuntimesSchemas',
    'runtimes-images = elyra.metadata.schemasproviders:RuntimeImagesSchemas',
    'code-snippets = elyra.metadata.schemasproviders:CodeSnippetsSchemas',
    'component-registries = elyra.metadata.schemasproviders:ComponentRegistriesSchemas'  ]
}
```
## Implementing Custom Metadata Classes
Applications can also influence behavior by introducing their own sublcass of `Metadata` to wrap instance data.  When this is done, those implementations will be called prior to and after any create, update and delete operations, in addition to following any load (get) operations.  As a result, applications can influence what is persisted and what is retrieved.

It should be noted that any exceptions raised from the method overrides will affect the overall operation.  This may be warranted in some cases in which the application finds some invariant condition violations, perhaps those that can't be expressed in the schema, in which the operation should be terminated.  If the exceptions are raised in either `post_save()` or `post_delete()`, the `MetadataManager` will also apply a _compensating transaction_ corresponding to the current operation.  For example, if an instance is being created and the `post_save()` method raises an exception, the `MetadataManager` will apply a compensating transaction that _deletes_ the newly-created instance.

#### Registration
Unlike `Schemaspaces` and `SchemasProvider` registrations, metadata class overrides are registered within their respective schema file via the `metadata_class_name` property.  The value of this property denotes the dotted notation corresponding to the location of the implementation, the hosting package of which must be installed and locatable.  (Note that someday, we may choose to change this value to be an entrypoint name of a well-documented entrypoint group.  In such cases, the value would not be "dotted" and, therefore, backwards compatible with the current approach.)

Here's an example of what the component-registry schema provides:
```python
"metadata_class_name": "elyra.pipeline.component_metadata.ComponentRegistryMetadata"
```
In this case, the `ComponentRegistry` module overrides the `post_save()` and `post_delete()` methods to update its component's cache.

### Migration Strategies
It's inevitable that schemas will change over time.  When such changes occur, the existing instances may be impacted such that they no longer conform to the schema or should belong to a different schema entirely.  That is, the instances need to be migrated to adhere to the updated schema.  This section covers two migration strategies, _Implicit Migration_ and _Explicit Migration_.

#### Implicit Migration
Implicit migration occurs _on the fly_, requiring no user intervention.  This form of migration typically occurs when the schema is altered to include a new required property or a schema within the schemaspace has been deprecated and existing instances should be associated with a different schema.  With implicit migrations, the instances will typically remain in their current schemaspace.

The recommended approach in these scenarios is to leverage the `Metadata` subclass that can be associated with each instance.  By introducing a subclass via the `metadata_class_name` _meta-property_, applications can then use the `on_load()` method to hook the loading of instances, apply the necessary alterations, persist the updates, and return.

For example, let's say that schema `foo` has been deprecated in favor of schema `bar` and that all instances of `foo` should be updated to instances of `bar`.  One could perform the following steps to ensure instances are migrated from `foo` to `bar`:
1. Introduce a `metadata_class_name` to schema `foo`(if one doesn't exist already).
2. Implement (or alter) the `on_load()` method.  This method needs to detect pre-migrated instances.  In this case, we're changing the instance's schema, so _any_ instance triggering this method is considered _pre-migrated_.  However, if the instance will remain under the schema, this is probably a good time to introduce an application-level _version_ indicator to determine if the instance is current or requires migration.
3. Make the necessary changes, including the update of the `schema_name` field, and persist the changes using an appropriate `MetadataManager` instance (tied to the schemaspace), being sure to include the `for_migration=True` property in the `update()` method.

```python
    def on_load(self, **kwargs: Any) -> None:
        """Perform implicit migration and update schema from 'foo' to 'bar' """

        prev_schema = self.schema_name
        self.schema_name = "bar"

        getLogger('ServerApp').info(f"Migrating '{prev_schema}' instance '{self.name}' "
                                    f"to schema '{self.schema_name}'...")
        MetadataManager(schemaspace="my_schemaspace").update(self.name, self, for_migration=True)
```
With implicit migration, the application migrates the instance on load, so you should ensure the instances are "touched".  If that is an issue, you could use the `elyra-metadata` CLI to list the instances - triggering their migration.  For example: `elyra-metadata list my_schemaspace`

#### Explicit Migration
Explicit migration is typically necessary when the changes are more widespread.  Examples of such migration will be when instances need to _move_ from one schemaspace to another.  In such cases, implicit migration is not practical.

An example of an explicit migration occurred in the Elyra 3.3 release when instances of the `component-registry` schema in schemaspace `component-registries` were migrated to different schemas (depending on a field's value) in a new schemaspace named `component-catalogs`.  In this case, we chose to use an explicit migration because there was no _application-level_ trigger to "touch" existing instances because the `component-registries` schemaspace was considered deprecated.

Explicit migration centers around the schemaspace from which instances are being migrated.  This is because it is the only entity that knows it has instances.  This migration is accomplished via the base `Schemaspace` `migrate()` method.  This method is marked as `final` meaning that it cannot be overridden by `Schemaspace` subclasses, and it merely enumerates the schemas (and their providers) calling the `migrate()` method on the corresponding `SchemasProvider` subclass.

When called `SchemasProvider.migrate()` enumerates instances of the given schema, updates those instances, persisting changes as necessary, and returns the list of migrated instances to the caller (i.e., the `migrate()` method on `Schemaspace`). (The code for the migration of `component-registry` instances to the `component-catalogs` schemaspace can be found [here](https://github.com/elyra-ai/elyra/blob/05bbdf22fa25b0a65f72c9054337f32fe5fde460/elyra/metadata/schemasproviders.py#L144-L212).)

To drive explicit migration, the `elyra-metadata` CLI tool has been updated with a `migrate` command - which acts on the _pre-migration_ schemaspace.  This option essentially calls the `migrate()` method on the given `Schemaspace` instance - which then invokes the appropriate `SchemasProvider` to migrate its schema instances.  See [Migrating user-defined component registries to 3.3](../user_guide/pipeline-components.html#migrating-user-defined-component-registries-to-3-3) for details.

---

## File: docs/source/developer_guide/pipeline-component-connectors.md

## How to develop a component catalog connector

Elyra uses connectors to retrieve [pipeline components](../user_guide/pipeline-components.md) from local or remote component catalogs.



The examples repository includes a [list of available catalog connectors](https://github.com/elyra-ai/examples/tree/main/component-catalog-connectors) and instructions on [how to create a custom connector](https://github.com/elyra-ai/examples/tree/main/component-catalog-connectors/build-a-custom-connector.md).

---

## File: docs/source/developer_guide/pipelines.md

# Pipelines
The following sections cover the details relative to working with pipelines in a development capacity.

## Pipeline definition

Below is a json representation of the pipeline that is generated by the **Pipeline Editor** and referred to as the _pipeline definition_. 

The pipeline is then submitted to the **Elyra Scheduler** which parses, properly packages and 
submits the pipeline definition to the chosen target runtime (e.g. Kubeflow Pipelines).

For more details on the **pipeline json definition** see it's [json schema](https://github.com/elyra-ai/pipeline-schemas/blob/master/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json)
 
### Pipeline definition json example 

```
/* Detailed source-code truncated for AI context efficiency. */
```


## Pipeline Processor Customization
Elyra implements an extensible **pipeline processor engine**, which enables the addition of new pipeline processors utilizing
a service discovery mechanism.  The pipeline processor class hierarchy is depicted here:

This section outlines what is needed to introduce your own runtime for integration with Elyra.  In essence, two criteria must be fulfilled to introduce a new runtime: 
1. A schema describing the necessary runtime metadata
1. A pipeline processor implementation appropriately associated to the runtime

### Custom Runtime Schema
The first requirement for introducing a new runtime for use in Elyra is to define the necessary metadata corresponding to the runtime.  This is accomplished via a JSON schema file describing the necessary metadata used to integrate with the targeted runtime platform.

The schema will be made available to the application via a `SchemasProvider` (See [_Bring Your Own Schema_](../developer_guide/metadata.html#bring-your-own-schemas) in [_Metadata Services_](../developer_guide/metadata.html#metadata-services)) which implements a single method `get_schemas()`.

The schema name, represented by the value of the top-level `name` property is what the pipeline engine uses to locate the appropriate pipeline processor implementation.  This value is also set into the pipeline definition's `runtime` property in Elyra's UI, thereby tying the pipeline to the appropriate processor.

The schema should minimally include property definitions for the Cloud Object Storage (cos) properties found in the built-in schema definitions for [Kubeflow Pipelines](https://github.com/elyra-ai/elyra/blob/62e1964244ec8ada3e63c9c6d39befd7c046df08/elyra/metadata/schemas/kfp.json#L83-L129) and [Apache Airflow](https://github.com/elyra-ai/elyra/blob/62e1964244ec8ada3e63c9c6d39befd7c046df08/elyra/metadata/schemas/airflow.json#L93-L139).
    
### Custom Runtime Pipeline Processor Implementation
The pipeline processor implementation that corresponds to the targeted runtime platform must be a subclass of `elyra.pipeline.processor.RuntimePipelineProcessor` - which, itself, derives from `elyra.pipeline.processor.PipelineProcessor`.

To facilitate discovery by the pipeline engine, this implementation must use a value for its [`type` property](https://github.com/elyra-ai/elyra/blob/62e1964244ec8ada3e63c9c6d39befd7c046df08/elyra/pipeline/processor.py#L156) identical to its schema name.

#### Processor Registration
The pipeline processor should be registered using **entry_points** with a _name_ that matches both the schema name corresponding to the new runtime and the pipeline processor's `type` property value.  Although this is not technically required, we would like to possibly utilize this during discovery in the future (and is true relative to the built-in processors).

Entrypoint definitions are typically found in the package's **setup.py** or **setup.cfg** files similar to the following example:

```python
    entry_points = {
        'elyra.pipeline.processors': [
            'my_runtime = acme.my_runtime:MyRuntimePipelineProcessor'
        ]
    },
```
In this example, and corresponding to the criteria above, pipeline processor `acme.my_runtime.MyRuntimePipelineProcessor` will:
- derive from `elyra.pipeline.processor.RuntimePipelineProcessor`
  
- use a value of `my_runtime` for its `type` property value
  
- implement a `SchemasProvider` that will return the JSON schema for `my_runtime` via its `get_schemas()` method.

---

## File: docs/source/developer_guide/refresh-elyra-on-odh.md

# Elyra refresh on Open Data Hub 

A custom Elyra container image is included with [Open Data Hub](https://opendatahub.io/).  This document outlines how to upgrade that image to a newer version. This task is typicaly only performed by Elyra maintainers.

## Update Elyra image stream

The [s2i-lab-elyra](https://github.com/opendatahub-io/s2i-lab-elyra) repository contains the artifacts that make Elyra available as an image stream.

- Install pipenv (version <= 2022.1.8) from PyPI.
  ```
  $ pip install pipenv==2022.1.8
  ```
- Fork the `https://github.com/opendatahub-io/s2i-lab-elyra` repository.
- Git clone the fork. 
  ```
  $ git clone git@github.com:<your-fork>/s2i-lab-elyra.git
  ```
- Create a new branch named `update-to-vX.Y.Z` (replacing X, Y, and Z with the latest major, minor, and patch version number for Elyra).

- Update the following as needed:
  - `.aicoe-ci.yaml` if any version changes to the base container image are needed. Only updated periodically after a new version of the [base image](https://quay.io/repository/thoth-station/s2i-minimal-py38-notebook?tab=tags&tag=latest) has been released and tested and vetted by the upstream team for ODH.
  - `.s2i/environment` if any additional environment variables need to be changed or set
  - `.s2i/run` if any additional bash setup commands need to be added/changed prior to starting Elyra

- Update `Pipfile`:
  - Replace the Elyra version number
    ```
    "elyra[all]" = "==X.Y.Z"
    ```

- Regenerate the `pipfile.lock` file
  ```
  $ pipenv lock --pre
  ```

- Create a pull request providing the following information:
  - Title: `Update elyra image to vX.Y.Z`
  - Description: add links to the [release notes](https://github.com/elyra-ai/elyra/releases) for every version that isn't available yet on Open Data Hub. For example, if the last version on ODH is 3.9.1 and you are bumping Elyra to 3.10.1, add links to the release notes for 3.10.0 and 3.10.1. This enables reviewers and users to determine what is new and why an upgrade to this release should be considered.

  Example pull request: https://github.com/opendatahub-io/s2i-lab-elyra/pull/43 

- After the pull request was merged a [new image release](https://github.com/opendatahub-io/s2i-lab-elyra/releases) is created for you. Take note of the new release `vY.Y.Y`. 
  > The release version is different from the Elyra version! 

## Update manifest files

Once the image stream pull request was merged you can update Open Data Hub manifest files.

- Fork the `https://github.com/opendatahub-io/odh-manifests` repository.
- Git clone the fork. 
  ```
  $ git clone git@github.com:<your-fork>/odh-manifests.git
  ```
- Create a new branch named `update-to-vY.Y.Y` replacing YYY with the image release number.

- Update `jupyterhub/notebook-images/overlays/additional/elyra-notebook-imagestream.yaml`
    - Change the existing `opendatahub.io/notebook-python-dependencies` annotation from `"version":"X.X.X"` to `"version":"X.Y.Z"`.
    - Change all existing occurrences of the old image release `vX.X.X` to the new image release `vY.Y.Y`.
- Update `jupyterhub/notebook-images/overlays/build/elyra-notebook-buildconfig.yaml`
    - Change the existing image stream tag name from `s2i-lab-elyra:vX.X.X` to `s2i-lab-elyra:vY.Y.Y`.
    - Change the git ref from `ref: vX.X.X` to `ref: vY.Y.Y`.
- Create a pull request providing the following information:
  - Title: `Update Elyra notebook image to vY.Y.Y`
  - Description: add links to the [release notes](https://github.com/elyra-ai/elyra/releases) for every version that isn't available yet on Open Data Hub. For example, if the last referenced version in the ODH manifest is 3.9.1 and you are bumping Elyra to 3.10.1, add links to the release notes for 3.10.0 and 3.10.1. This enables reviewers and users to determine what is new and why an upgrade to this release should be considered.

  Example pull request: https://github.com/opendatahub-io/odh-manifests/pull/579