{"owner":"ansible","repo":"awx","hasSkills":true,"totalSkillsCount":27,"totalTokensCount":23539,"categories":["plugin-manifest","anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":["tools/community-bugscrub/README.md","awx_collection/tools/README.md","tools/loki/local-config.yaml","tools/grafana/README.md","tools/otel/otel-collector-config.yaml","tools/docker-compose/logstash-override.yaml","tools/docker-compose/README.md","tools/docker-compose/docs/data_migration.md","tools/docker-compose/docs/logstash.md","tools/docker-compose/editable_dependencies/README.md","tools/sosreport/TESTING.md","tools/grafana/dashboards/demo_dashboard.json","tools/grafana/dashboards/services_dashboard.json","tools/scripts/ig-hotfix/README.md","tools/scripts/webhook_examples/github_push.json","tools/scripts/webhook_examples/gitlab_push.json","awx/main/tests/data/inventory/plugins/vmware/env.json","awx/main/tests/data/inventory/plugins/terraform/env.json","awx/main/tests/data/inventory/plugins/satellite6/env.json","awx/main/tests/data/inventory/plugins/rhv/env.json","awx/main/tests/data/inventory/plugins/openstack/env.json","awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json","awx/main/tests/data/inventory/plugins/insights/env.json","awx/main/tests/data/inventory/plugins/gce/env.json","awx/main/tests/data/inventory/plugins/ec2/env.json","awx/main/tests/data/inventory/plugins/controller/env.json","awx/main/tests/data/inventory/plugins/azure_rm/env.json"],"skills":{"tools/community-bugscrub/README.md":"# Community BugScrub tooling\n\nSmall python script that automatically distributes PRs and Issues given a list of `people` and dumps the contents in a Spreadsheet.\n\nTo be used when distributing the work of reviewing community contributions.\n\n## Usage\n\nInstall requirements.\n\n```\npip install -r requirements.txt\n```\n\nGet the usage.\n\n```\npython generate-sheet.py -h\n```\n\n## Adding a github Personal Access Token\nThe scripts looks first for a github personal access token to use to avoid having the scripts calls rate limited, you can create one or use an existing one if you have. The script looks for the PAT under the environment var `GITHUB_ACCESS_TOKEN`. \n\n\n# For internal spreadsheet usage\nAWX engineers will need to import the data generated from the script into a spreadshet manager. Please make sure that you do not replace the existing sheets but make a new one or create a new sheet inside the existing spreadsheet upon import. ","awx_collection/tools/README.md":"## Collection tools\n\nTools used for building, maintaining, and testing the collection.\n\n### Template Galaxy\n\nThe `template_galaxy.yml` playbook ran as a pre-requisite for building the collection.\n\n```\nmake awx_collection_build\n```\n\n### Generate\n\nThis will template resource modules (like `group`, for groups in inventory) from a boilerplate template.\nIt is intended as a tool for writing new modules or enforcing consistency.\n\n### Integration Testing\n\nThese instructions assume you have ansible-core and the collection installed.\nTo install the collection in-place (to pick up any local changes to source)\nthe `make symlink_collection` will symlink the `awx_collection/` folder to\nthe appropriate place under `~/.ansible/collections`.\n\nThis is a shortcut for quick validation of tests that bypasses `ansible-test`.\nTo use this, you need the `~/.tower_cli.cfg` config file populated,\nwhich can be done via the deprecated `tower-cli login ` or manually\nwriting it, where the format looks like:\n\n```\n[general]\nhost = https://localhost:8043/\nverify_ssl = false\nusername = admin\npassword = password\n```\n\nTODO: adjust playbook to allow using environment variables as well.\n\nTo run some sample modules:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml\n```\n\nTo run just one module (the most common use case), use the `-e test=`.\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=host\n```\n\nIf you want to run _all_ the tests, then you need to pass in the whole list.\nThis will take significant time and is not ideal from an error-handling perspective,\nbut this is a way to do it:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=$(ls -1Am awx_collection/tests/integration/targets/ | tr -d '[:space:]')\n```\n\nDepending on the module, you may need special dependencies.\nFor instance, the rrule lookup plugins need `pytz`.\nThese will be satisfied if you install requirements in `awx_collection/requirements.txt`.\n","tools/loki/local-config.yaml":"auth_enabled: false\n\nserver:\n http_listen_port: 3100\n grpc_server_max_recv_msg_size: 524288000 # 500 MB\n grpc_server_max_send_msg_size: 524288000 # 500 MB, might be too much, be careful\n\nfrontend_worker:\n match_max_concurrent: true\n grpc_client_config:\n max_send_msg_size: 524288000 # 500 MB\n\n\ningester:\n max_chunk_age: 8766h\n\ncommon:\n path_prefix: /loki\n storage:\n filesystem:\n chunks_directory: /loki/chunks\n rules_directory: /loki/rules\n replication_factor: 1\n ring:\n kvstore:\n store: inmemory\n\n# compactor:\n# retention_enabled: true\n# # cmeyers: YOLO. 1s seems wrong but it works so right\n# compaction_interval: 1s # default 10m\n\nschema_config:\n configs:\n - from: 2020-10-24\n store: boltdb-shipper\n object_store: filesystem\n schema: v11\n index:\n prefix: index_\n period: 24h\n\nstorage_config:\n boltdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/boltdb-cache\n\nruler:\n alertmanager_url: http://localhost:9093\n\nlimits_config:\n retention_period: 3y\n # cmeyers: The default of 30m triggers a loop of queries that take a long time\n # to complete and the UI times out\n split_queries_by_interval: 1d\n # cmeyers: Default of 30d1h limits grafana time queries. Can't, for example,\n # query last 90 days\n max_query_length: 3y\n # cmeyers: Made the batch post request succeed.\n reject_old_samples: false\n reject_old_samples_max_age: 365d\n\n ingestion_rate_mb: 32\n ingestion_burst_size_mb: 32\n per_stream_rate_limit: 32M\n per_stream_rate_limit_burst: 32M\n ingestion_rate_strategy: local # Default: global\n max_global_streams_per_user: 100000000\n max_entries_limit_per_query: 100000000\n max_query_series: 1000000\n max_query_parallelism: 32 # Old Default: 14\n max_streams_per_user: 100000000 # Old Default: 10000\n\n# Taken from aap-log-visualizer\nfrontend:\n max_outstanding_per_tenant: 2048\n\nquery_scheduler:\n max_outstanding_requests_per_tenant: 2048\n\nquery_range:\n parallelise_shardable_queries: false\n split_queries_by_interval: 0\n\n# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration\n# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/\n#\n# Statistics help us better understand how Loki is used, and they show us performance\n# levels for most users. This helps us prioritize features and documentation.\n# For more information on what's sent, look at\n# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go\n# Refer to the buildReport method to see what goes into a report.\n#\n# If you would like to disable reporting, uncomment the following lines:\n#analytics:\n# reporting_enabled: false\n","tools/grafana/README.md":"### Prometheus and Grafana\nPrometheus is a metrics collecting tool, and we support prometheus formatted data at the`api/v2/metrics` endpoint.\n\nPrometheus is configured to poll targets on some interval and can use various authentication methods.\nWe configure prometheus to use basic auth and poll AWX every 5 seconds by default. This configuration is created using the `docker-compose/ansible/roles/sources/templates/prometheus.yml.j2` template. To change the values, modify the template or set the ansible variables used in that template (see use of `EXTRA_SOURCES_ANSIBLE_OPTS` in the next section).\n\nGrafana provides dashboards to visualize this data as well as alerting features.\n\nWe use the \"provisioning files\" functionality in Grafana to be able to deploy alert and dashboard configs from file.\n\nThese config files are located in directories that are mounted into the container in `tools/grafana`.\n\n### How to run docker-compose with Grafana and Prometheus\n\nTo run the development environment (see [docs](https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md)) with Prometheus and Grafana enabled, set the following variables:\n\n```\n$ PROMETHEUS=yes GRAFANA=yes make docker-compose\n```\n\nIf we want to pass additional ansible vars used in the deployment process, we can pass `EXTRA_SOURCES_ANSIBLE_OPTS`:\n\n```\nGRAFANA=true PROMETHEUS=true EXTRA_SOURCES_ANSIBLE_OPTS=\"-e scrape_interval=1 admin_password=foobar\" make docker-compose\n```\n\n### Where to view grafana and prometheus\n\n1. navigate to Prometheus at `http://localhost:9090/targets` and check that the metrics endpoint State is Up.\n2. Click the Graph tab, start typing a metric name, or use the Open metrics explorer button to find a metric to display (next to `Execute` button)\n3. Navigate to Grafana at `http://localhost:3001`. Sign in, using `admin` for both username and password.\n4. In the left navigation menu go to Dashboards->Browse, find the \"awx-demo\" and click. These should have graphs.\n5. Now you can modify these and add panels for whichever metrics you like.\n\n### Alerts in Grafana\n\nWe are configuring alerts in grafana using the provisioning files method. This feature is new in Grafana as of August, 2022. Documentation can be found: https://grafana.com/docs/grafana/latest/administration/provisioning/#alerting however it does not fully show all parameters to the config.\n\nOne way to understand how to build rules is to build them in the UI and use chrometools to inspect the payload as you save the rules. It appears that the \"data\" portion of the payload for each rule is the same syntax as needed in the provisioning file config. To reload the alerts without restarting the container, from your terminal you can send a POST with `curl -X POST http://admin:admin@localhost:3001/api/admin/provisioning/alerting/reload`.\n\nAnother way to export rules is explore the api.\n1. Get all the folders: `GET` to `/api/folders`\n2. Get the rules `GET` to `/api/ruler/grafana/api/v1/rules/{{ Folder }}`\n\nYou can do this via curl or in the web browser.\n\n### Included Alerts\n\n#### Alert if remaining capacity low and pending jobs exist\n\nWe want to know if jobs are in pending but we lack capacity in the cluster to run them. Our approach is to sum all remaining capacity in the cluster and compare it to the total capacity of the cluster. If less than 10% of our capacity is remaining and we have pending jobs, and this is true for more than 180s, we will fire the alert.\n\nThis alert is named \"capacity_below_10_percent\" and can be found in this directory in https://github.com/ansible/awx/blob/devel/tools/grafana/alerting/alerts.yml\n","tools/otel/otel-collector-config.yaml":"receivers:\n otlp:\n protocols:\n grpc:\n http:\n\nexporters:\n debug:\n verbosity: detailed\n\n file:\n path: /awx-logs/awx-logs.json.zstd\n rotation:\n max_days: 14\n localtime: false\n max_megabytes: 300\n max_backups: 200\n format: json\n compression: zstd\n\n loki:\n endpoint: http://loki:3100/loki/api/v1/push\n tls:\n insecure: true\n headers:\n \"X-Scope-OrgID\": \"1\"\n default_labels_enabled:\n exporter: true\n job: true\n instance: true\n level: true\n\nprocessors:\n batch:\n\nextensions:\n health_check:\n zpages:\n endpoint: \":55679\"\n\nservice:\n pipelines:\n logs:\n receivers: [otlp]\n processors: [batch]\n exporters: [file, loki]\n\n extensions:\n - health_check\n - zpages\n","tools/docker-compose/logstash-override.yaml":"---\nservices:\n # A useful container that simply passes through log messages to the console\n # helpful for testing awx/tower logging\n logstash:\n build:\n context: ../\n dockerfile: Dockerfile-logstash\n container_name: tools_logstash_1\n hostname: logstash\n networks:\n - awx\n","tools/docker-compose/README.md":"# Docker Compose for Development\n\n## Getting started\n\n### Clone the repo\n\nIf you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). We generally recommend that you view the releases page:\n\nhttps://github.com/ansible/awx/releases/latest\n\n...and clone the latest stable tag, e.g.,\n\n`git clone -b x.y.z https://github.com/ansible/awx.git`\n\nPlease note that deploying from `HEAD` (or the latest commit) is **not** stable, and that if you want to do this, you should proceed at your own risk.\n\nFor more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).\n\nOnce you have a local copy, run the commands in the following sections from the root of the project tree.\n\n## Overview\n\nHere are the main `make` targets:\n\n- `docker-compose-build` - used for building the development image, which is used by the `docker-compose` target. You can skip this target if you want to use the latest [ghcr.io/ansible/awx_devel:devel](https://github.com/ansible/awx/pkgs/container/awx_devel) image rather than build a new one.\n- `docker-compose` - make target for development, passes awx_devel image and tag\n\nNotable files:\n\n- `tools/docker-compose/inventory` file - used to configure the AWX development environment.\n- `tools/docker-compose/ansible/migrate.yml` - playbook for migrating data from Local Docker to the Development Environment\n\n### Prerequisites\n\n- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the `docker` group, refer to the documentation for details)\n- [Docker Compose](https://docs.docker.com/compose/install/).\n- [Ansible](https://docs.ansible.com/projects/ansible/latest/installation_guide/intro_installation.html) will need to be installed as we use it to template files needed for the docker-compose.\n- OpenSSL.\n\n### Tested Operating Systems\n\nThe docker-compose development environment is regularly used and should work on x86_64 systems running:\n\n- Fedora (maintained versions)\n- Ubuntu LTS (18, 20)\n- Red Hat Enterprise Linux 8, CentOS Stream 8\n- macOS 11\n\nUse on other platforms is untested, and may require local changes.\n\n### How to Disable Color?\n\nThere are several layers that might apply color:\n - docker compose coloring based on what container log comes from\n - supervisord coloring based on what server it comes from\n - general coloration from Django management commands\n\nThese can have color turned off by adding things to the\nenvironment variable to modify the call options.\n\n```\nDJANGO_COLORS=nocolor COMPOSE_UP_OPTS=\"--no-color\" SUPERVISOR_ARGS=\"-n -t\" make docker-compose\n```\n\nThis can be useful if this is ran in CI in any context.\n\n## Starting the Development Environment\n\n### Build the Image\n\nThe AWX base container image (defined in the Dockerfile templated from [Dockerfile.j2](./../ansible/roles/dockerfile/templates/Dockerfile.j2)) contains basic OS dependencies and symbolic links into the development environment that make running the services easy.\n\nRun the following to build the image:\n\n```bash\n$ make docker-compose-build\n```\n\n> The image will need to be rebuilt if there are any changes to Dockerfile.j2 or any of the files used by the templated Dockerfile.\n\nOnce the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows:\n\n```bash\n(host)$ docker images\n\nREPOSITORY TAG IMAGE ID CREATED SIZE\nansible/awx_devel latest ba9ec3e8df74 26 minutes ago 1.42GB\n```\n\n> By default, this image will be tagged with your branch name. You can specify a custom tag by setting an environment variable, for example: `DEVEL_IMAGE_NAME=quay.io/your_user/awx_devel:17.0.1`\n\n#### Customizing the Receptor Image\n\nBy default, the development environment will use the `devel` image from receptor.\nThis is used directly in `docker-compose.yml` for the hop nodes.\nThe receptor binary is also copied over to the main awx_devel image, used in all other AWX nodes.\n\nBecause of this, the `RECEPTOR_IMAGE` environment variable must be set when running\nboth docker-compose-build and docker-compose in order to use the correct receptor in all containers.\n\nIf you need to create a new receptor image, you can check out receptor and build it like this:\n\n```bash\nCONTAINERCMD=docker TAG=quay.io/ansible/receptor:release_1.1 make container\n```\n\nThen that can be used by AWX like this:\n\n```bash\nexport RECEPTOR_IMAGE=quay.io/ansible/receptor:release_1.1\nmake docker-compose-build\nmake docker-compose\n```\n\n### Run AWX\n\n##### Start the containers\n\nRun the awx, postgres and redis containers. This utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the awx container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process.\n\n```bash\n$ make docker-compose\n```\n\n> The make target assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG= `in your environment. For example, you might be working in a feature branch, but you want to run the containers using the devel image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose`\n\n> For running docker-compose detached mode, start the containers using the following command: `$ make docker-compose COMPOSE_UP_OPTS=-d`\n\n###### Solving database-related issues during initial startup\n\nIf you have encountered the infinitely-repeating `Waiting for postgres to be ready to accept connections` message during the execution, try to do the following:\n\n1. Stop and delete AWX-related docker containers.\n2. Delete all associated docker volumes.\n3. Delete all associated docker networks.\n4. Repeat the process from scratch.\n\nIf you have **only** AWX-related container entities in your system, you can simply stop and delete everything using the following commands:\n\n```bash\ndocker stop $(docker ps -a -q)\ndocker system prune -a\ndocker volume prune\ndocker network prune\n```\n\n##### _(alternative method)_ Spin up a development environment with customized mesh node cluster\n\nA cluster (of containers) with execution nodes and a hop node can be created by the docker-compose Makefile target.\nBy default, it will create 1 hybrid node.\nYou can switch the type of AWX nodes between hybrid and control with `MAIN_NODE_TYPE`.\n\n```\nMAIN_NODE_TYPE=control EXECUTION_NODE_COUNT=2 COMPOSE_TAG=devel make docker-compose\n```\n\nRunning the above command will create a cluster of 1 control node, 1 hop node, and 2 execution nodes.\nA hop node is automatically created whenever there are 1 or more execution nodes.\n\nThe number of nodes can be changed:\n\n```\nCONTROL_PLANE_NODE_COUNT=2 EXECUTION_NODE_COUNT=3 COMPOSE_TAG=devel make docker-compose\n```\n\nThis will spin up a topology represented below.\n(names are the receptor node names, which differ from the AWX Instance names and network address in some cases)\n\n```\n ┌──────────────┐\n │ │\n┌──────────────┐ ┌──────────┤ receptor-1 │\n│ │ │ │ │\n│ awx_1 │◄──────────┐ │ └──────────────┘\n│ │ │ ▼\n└──────┬───────┘ ┌──────┴───────┐ ┌──────────────┐\n │ │ │ │ │\n │ │ receptor-hop │◄───────┤ receptor-2 │\n ▼ │ │ │ │\n┌──────────────┐ └──────────────┘ └──────────────┘\n│ │ ▲\n│ awx_2 │ │ ┌──────────────┐\n│ │ │ │ │\n└──────────────┘ └──────────┤ receptor-3 │\n │ │\n └──────────────┘\n```\n\nAll execution (`receptor-*`) nodes connect to the hop node.\nOnly the `awx_1` node connects to the hop node out of the AWX cluster.\n`awx_1` connects to `awx_2`, fulfilling the requirement that the AWX cluster is fully connected.\n\nFor example, if a job is launched with `awx_2` as the `controller_node` and `receptor-3` as the `execution_node`,\nthen `awx_2` communicates to `receptor-3` via `awx_1` and then `receptor-hop`.\n\n\n##### Wait for migrations to complete\n\nThe first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following:\n\n```bash\nawx_1 | Operations to perform:\nawx_1 | Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic\nawx_1 | Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, contenttypes, auth, conf, main\nawx_1 | Synchronizing apps without migrations:\nawx_1 | Creating tables...\nawx_1 | Running deferred SQL...\nawx_1 | Installing custom SQL...\nawx_1 | Running migrations:\nawx_1 | Rendering model states... DONE\nawx_1 | Applying contenttypes.0001_initial... OK\nawx_1 | Applying contenttypes.0002_remove_content_type_name... OK\nawx_1 | Applying auth.0001_initial... OK\n...\n```\n\n##### Clean and build the UI\n\nPrerequisites (on your local machine)\n- npm\n- nodejs\n\nRequired versions listed here https://github.com/ansible/ansible-ui/blob/main/README.md\n\nOn your local machine (not in awx container)\n\n```bash\nmake clean/ui ui\n```\n\nThis will clone the ansible-ui into the `awx/ui/src` directory and build the static files. Then when the containers come up, awx-manage collectstatic will copy those files into the proper place.\n\nYou can also use `UI_LOCAL` to build from a locally cloned ansible-ui repo.\n\nSee [the ui development documentation](https://github.com/ansible/ansible-ui/blob/main/CONTRIBUTING.md) for more information on using the frontend development, build, and test tooling.\n\nOnce migrations are completed and the UI is built, you can begin using AWX. The UI can be reached in your browser at `https://localhost:8043/`, and the API can be found at `https://localhost:8043/api/v2`.\n\n##### Create an admin user\n\nBefore you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command:\n\n```bash\n$ docker exec -ti tools_awx_1 awx-manage createsuperuser\n```\n\n> Remember the username and password, as you will use them to log into the web interface for the first time.\n\n##### Load demo data\n\nOptionally, you may also want to load some demo data. This will create a demo project, inventory, and job template.\n\n```bash\n$ docker exec tools_awx_1 awx-manage create_preload_data\n```\n\n> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate\n> this container, and thus the database, if the database schema changes in an upstream commit.\n\n## Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior), you can\nmigrate your data to the development environment via the migrate.yml playbook using the steps described [here](./docs/data_migration.md).\n\n## Upgrading the Development Environment\n\nUpgrading AWX involves checking out the new source code and re-running the make target. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables.\n\nAfter updating the inventory file with any custom values, run the make target from the root of your AWX clone.\n\n```bash\n$ make docker-compose\n```\n\n## Extras\n\n- [Start a shell](#start-a-shell)\n- [Start AWX from the container shell](#start-awx-from-the-container-shell)\n- [Using Logstash](./docs/logstash.md)\n- [Start a Cluster](#start-a-cluster)\n- [Start with Minikube](#start-with-minikube)\n- [Splunk Integration](#splunk-integration)\n\n### Start a Shell\n\nTo run `awx-manage` commands and modify things inside the container, you will want to start a shell session on the _awx_ container. In a new terminal session, use the `docker exec` command to start the shell session:\n\n```bash\n(host)$ docker exec -it tools_awx_1 bash\n```\n\nThis creates a session in the _awx_ containers, just as if you were using `ssh`, and allows you execute commands within the running container.\n\n### Start AWX from the container shell\n\nOften times you'll want to start the development environment without immediately starting all of the services in the _awx_ container, and instead be taken directly to a shell. You can do this with the following:\n\n```bash\n(host)$ make docker-compose-test\n```\n\nUsing `docker exec`, this will create a session in the running _awx_ container, and place you at a command prompt, where you can run shell commands inside the container.\n\nIf you want to start and use the development environment, you'll first need to bootstrap it by running the following command:\n\n```bash\n(container)# /usr/bin/bootstrap_development.sh\n```\n\nThe above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it\nwill drop you back to the shell.\n\nIn order to launch all developer services:\n\n```bash\n(container)# /usr/bin/launch_awx.sh\n```\n\n`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't\nneed to call `bootstrap_development.sh` first.\n\n### Start a Cluster\n\nCertain features or bugs are only applicable when running a cluster of AWX nodes. To bring up a 3 node cluster development environment simply run the below command.\n\n```bash\n(host)$ CONTROL_PLANE_NODE_COUNT=3 make docker-compose\n```\n\n`CONTROL_PLANE_NODE_COUNT` is configurable and defaults to 1, effectively a non-clustered AWX.\n\nNote that you may see multiple messages of the form `2021-03-04 20:11:47,666 WARNING [-] awx.main.wsrelay Connection from awx_2 to awx_5 failed: 'Cannot connect to host awx_5:8013 ssl:False [Name or service not known]'.`. This can happen when you bring up a cluster of many nodes, say 10, then you bring up a cluster of less nodes, say 3. In this example, there will be 7 `Instance` records in the database that represent AWX instances. The AWX development environment mimics the VM deployment (vs. kubernetes) and expects the missing nodes to be brought back to healthy by the admin. The warning message you are seeing is all of the AWX nodes trying to connect the websocket backplane. You can manually delete the `Instance` records from the database i.e. `Instance.objects.get(hostname='awx_9').delete()` to stop the warnings.\n\n### Start with Minikube\n\nTo bring up a 1 node AWX + minikube that is accessible from AWX run the following.\n\nStart minikube\n\n```bash\n(host)$minikube start --cpus=4 --memory=8g --addons=ingress\n```\n\nStart AWX\n\n```bash\n(host)$ make docker-compose-container-group\n```\n\nAlternatively, you can set the env var `MINIKUBE_CONTAINER_GROUP=true` to use the default dev env bring up. his way you can use other env flags like the cluster node count. Set `MINIKUBE_SETUP=true` to make the roles download, install and run minikube for you, but if you run into issues with this just start minikube yourself.\n\n```bash\n(host)$ MINIKUBE_CONTAINER_GROUP=true make docker-compose\n```\n\nIf you want to clean all things once your are done, you can do:\n\n```bash\n(host)$ make docker-compose-container-group-clean\n```\n\n### Splunk Integration\n\nSplunk is a log aggregation tool that can be used to test AWX with external logging integration. This section describes how to build a reference Splunk instance and plumb it with your AWX for testing purposes.\n\nFirst, be sure that you have the awx.awx collection installed by running `make install_collection`.\n\nNext, install the splunk.es collection by running `ansible-galaxy collection install splunk.es`.\n\nAnytime you want to run a Splunk instance alongside AWX we can start docker-compose with the SPLUNK option to get a Splunk instance with the command:\n```bash\nSPLUNK=true make docker-compose\n```\n\nOnce the containers come up three new ports (8000, 8089 and 9199) should be exposed and the Splunk server should be running on some of those ports (the 9199 will be created later by the plumbing playbook). The first port (8000) is the non-SSL admin port and you can log into splunk with the credentials admin/splunk_admin. The url will be like http://:8000/ this will be referenced below. The 8089 is the API port that the ansible modules will use to connect to and configure splunk. The 9199 port will be used to construct a TCP listener in Splunk that AWX will forward messages to.\n\nOnce the containers are up we are ready to configure and plumb Splunk with AWX. To do this we have provided a playbook which will:\n* Backup and configure the External Logging adapter in AWX. NOTE: this will back up your existing settings but the password fields can not be backed up through the API, you need a DB backup to recover this.\n* Create a TCP port in Splunk for log forwarding\n\nFor routing traffic between AWX and Splunk we will use the internal docker compose network. The `Logging Aggregator` will be configured using the internal network machine name of `splunk`.\n\nOnce you have have the collections installed (from above) you can run the playbook like:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_splunk.yml\n```\n\nOnce the playbook is done running Splunk should now be setup in your development environment. You can log into the admin console (see above for username/password) and click on \"Searching and Reporting\" in the left hand navigation. In the search box enter `source=\"http:tower_logging_collections\"` and click search.\n\n### HashiVault Integration\n\nRun a HashiVault container alongside of AWX.\n\n```bash\nVAULT=true make docker-compose\n```\n\nYou can find the initialization data at `tools/docker-compose/_sources/secrets/vault_init.yml`,\nThis includes the unseal keys and a root token.\n\nYou will need to unseal the HashiVault each time the container is started.\nThe easiest way to do that is to run:\n```bash\nansible-playbook tools/docker-compose/ansible/unseal_vault.yml\n```\nThis will perform the unseal and also display the root token for login.\n\nFor demo purposes, Vault will be auto-configured to include a Key Value (KV) vault called `my_engine` along with a secret called `my_key` in `/my_engine/my_root/my_folder`.\nThe secret value is `this_is_the_secret_value`.\n\nTo create a secret connected to this vault in AWX you can run the following playbook:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_vault.yml\n```\n\nThis will create the following items in your AWX instance:\n* A credential called `Vault Lookup Cred` tied to the vault instance.\n* A credential called `Vault UserPass Lookup Cred` tied to the vault instance.\n* A custom credential type called `Vault Custom Cred Type`.\n* A credential called `Credential From HashiCorp Vault via Token Auth` which is of the created type using the `Vault Lookup Cred` to get the secret.\n* A credential called `Credential From HashiCorp Vault via UserPass Auth` which is of the created type using the `Vault Userpass Lookup Cred` to get the secret.\n\nThe custom credential type adds a variable when used in a playbook called `the_secret_from_vault`.\nIf you have a playbook like:\n```\n---\n- name: Show a vault secret\n hosts: localhost\n connection: local\n gather_facts: False\n tasks:\n - debug:\n var: the_secret_from_vault\n```\n\nAnd run it through AWX with the credential `Credential From Vault via Token Auth` tied to it, the debug should result in `this_is_the_secret_value`. If you run it through AWX with the credential `Credential From Vault via Userpass Auth`, the debug should result in `this_is_the_userpass_secret_value`.\n\n### Prometheus and Grafana integration\n\nSee docs at https://github.com/ansible/awx/blob/devel/tools/grafana/README.md\n\n### OpenTelemetry Integration\n\n```bash\nOTEL=true GRAFANA=true LOKI=true PROMETHEUS=true make docker-compose\n```\n\nThis will start the sidecar container `tools_otel_1` and configure AWX logging to send to it. The OpenTelemetry Collector is configured to export logs to Loki. Grafana is configured with Loki as a datasource. AWX logs can be viewed in Grafana.\n\n`http://localhost:3001` grafana\n","tools/docker-compose/docs/data_migration.md":"# Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior) to AWX 18.0 or higher, you can\nmigrate your data to the development environment via the migrate.yml playbook.\n\n> Note: This will also convert your postgresql bind-mount into a docker volume.\n\n### Migrate data with migrate.yml\n\nFirst, in the [`inventory` file](../inventory), set your `pg_password`, `broadcast_websocket_secret`, `secret_key`, and any other settings you need for your deployment. **Make sure you use the same values from the `inventory` file in your existing pre-18.0.0 Docker deployment.**\n\nIf you used a custom pgdocker or awxcompose location, you will need to set the `postgres_data_dir` and `old_docker_compose_dir` variables.\n\n1. Run the [migrate playbook](../ansible/migrate.yml) to migrate your data to the new postgresql container and convert the data directory to a volume mount.\n```bash\n$ ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/migrate.yml -e \"migrate_local_docker=true\" -e \"postgres_data_dir=~/.awx/pgdocker\" -e \"old_docker_compose_dir=~/.awx/awxcompose\"\n```\n\n2. Change directory to the top of your awx checkout, and follow the [development environment installation instructions](../README.md) to start your containers:\n```bash\n$ make docker-compose-build\n$ make docker-compose\n$ make ui\n```\n\n3. After ensuring your data has been successfully migrated, you may delete your old data directory (typically stored at `~/.awx/pgdocker`).\n\n4. Note that after migration, the development environment will be available at https://localhost:8043/ (not http://localhost).\n","tools/docker-compose/docs/logstash.md":"# How to use the logstash container\n\n#### Modify the docker-compose.yml\n\nYou can run the logstash container by adding another compose file to the docker-compose target.\n\n```\nCOMPOSE_OPTS=\"-f tools/docker-compose/logstash-override.yaml\" COMPOSE_TAG=devel make docker-compose\n```\n\nPOST the following content to `/api/v2/settings/logging/` (this uses\nauthentication set up inside of the logstash configuration file).\n\n```\n{\n \"LOG_AGGREGATOR_HOST\": \"http://logstash\",\n \"LOG_AGGREGATOR_PORT\": 8085,\n \"LOG_AGGREGATOR_TYPE\": \"logstash\",\n \"LOG_AGGREGATOR_USERNAME\": \"awx_logger\",\n \"LOG_AGGREGATOR_PASSWORD\": \"workflows\",\n \"LOG_AGGREGATOR_LOGGERS\": [\n \"awx\",\n \"activity_stream\",\n \"job_events\",\n \"system_tracking\",\n \"job_lifecycle\"\n ],\n \"LOG_AGGREGATOR_INDIVIDUAL_FACTS\": false,\n \"LOG_AGGREGATOR_TOWER_UUID\": \"991ac7e9-6d68-48c8-bbde-7ca1096653c6\",\n \"LOG_AGGREGATOR_ENABLED\": true\n}\n```\n\n> Note: HTTP must be specified in the `LOG_AGGREGATOR_HOST` if you are using the docker development environment. \n\nAn example of how to view the most recent logs from the container:\n\n```\ndocker exec -i -t $(docker ps -aqf \"name=tools_logstash_1\") tail -n 50 /logstash.log\n```\n\n#### How to add logstash plugins\n\nAdd any plugins you need in `tools/elastic/logstash/Dockerfile` before running the container. \n","tools/docker-compose/editable_dependencies/README.md":"# Editable dependencies in AWX Docker Compose Development Environment\n\nThis folder contains the symlink to editable dependencies for AWX\n\nDuring the bootstrap of awx development environment we will try to crawl through the symlinks and mount (the source of the symlink) to `tools_awx_` containers and `init_awx` containers then install all the dependencies in editable mode\n\n## How to enable/disable editable dependnecies\n\n### Enable\n\nSet `EDITABLE_DEPENDENCIES=true` either as an Environment Variable before invoking `make docker-compose`\n\n```bash\nexport EDITABLE_DEPENDENCIES=true\n```\n\nor during invocation of `make docker-compose`\n\n```bash\n\nEDITABLE_DEPENDENCIES=true make docker-compose\n```\n\nThis will cause the `make docker-compose-source` to template out docker-compose file with editable dependencies.\n\n### Disable\n\nTo disable editable dependency simply `unset EDITABLE_DEPENDENCIES`\n\n## How to add editable dependencies\n\nAdding symlink to the directory that contains the source of the editable dependencies will cause the dependency to be mounted and installed in the docker-compose development environment.\n\nBoth relative path or absolute path will work.\n\n### Examples\n\nI have `awx` checked out at `~/projects/src/github.com/TheRealHaoLiu/awx`\nI have `django-ansible-base` checked out at `~/projects/src/github.com/TheRealHaoLiu/django-ansible-base`\n\nFrom root of AWX project `~/projects/src/github.com/TheRealHaoLiu/awx`\n\nI can either do\n\n```bash\nln -s ~/projects/src/github.com/TheRealHaoLiu/django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\nor\n\n```bash\nln -s ../django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\n## How to remove indivisual editable dependencies\n\nSimply removing the symlink from `tools/docker-compose/editable_dependencies` **will cause problem**!\n\nand the volume `tools_var_lib_awx` needs to be deleted as well with\n\n```bash\nmake docker-compose-down\ndocker volume rm tools_var_lib_awx\n```\n\n","tools/sosreport/TESTING.md":"Create a RHEL box and then do the following.\n```bash\nsudo mkdir -p /usr/share/sosreport/sos/plugins\nsudo yum install sos\ncp controller.py /usr/share/sosreport/sos/plugins\nsudo chmod 644 /usr/share/sosreport/sos/plugins/controller.py\nln -s /usr/share/sosreport/sos/plugins/controller.py `find `find /usr/lib -name sos` -name plugins`\nsosreport -l | grep controller\n```\n\nThe results should be:\n```bash\n# sosreport -l | grep controller\n controller Ansible Automation Platform controller information\n```\n\nTo run only the controller plugin run: `sosreport --only-plugins controller`\n\n","tools/grafana/dashboards/demo_dashboard.json":"{\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": {\n \"type\": \"grafana\",\n \"uid\": \"-- Grafana --\"\n },\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"fiscalYearStartMonth\": 0,\n \"graphTooltip\": 0,\n \"id\": 2,\n \"links\": [],\n \"liveNow\": false,\n \"panels\": [\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 38,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"Fraction of time dispatcher is listening for new messages\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 39,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_availability\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Availability\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 1\n },\n \"id\": 40,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_max_worker_count\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_active_task_count\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Dispatcher Workers\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 9\n },\n \"id\": 41,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_scale_up_events\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Pool Scale-Up Events\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Dispatcher\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 37,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 14,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_database_connections_total\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Database\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {},\n \"fieldConfig\": {\n \"defaults\": {\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"short\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 25,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"/^tower_version$/\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"000000021\"\n },\n \"editorMode\": \"code\",\n \"exemplar\": false,\n \"expr\": \"awx_system_info\",\n \"format\": \"table\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Version\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"displayName\": \"Instances\",\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 30\n },\n \"id\": 13,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"center\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"count(awx_instance_info)\",\n \"interval\": \"\",\n \"legendFormat\": \" \",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Node Count\",\n \"type\": \"stat\"\n }\n ],\n \"title\": \"System\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 2\n },\n \"id\": 35,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"running\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"canceled\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"super-light-yellow\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 8,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"awx_status_total\",\n \"legendFormat\": \"{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"job status\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-error\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-successful\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-purple\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_3-failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-orange\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 29,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_status_total\",\n \"legendFormat\": \"{{node}}-{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Job Status per Instance\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 34\n },\n \"id\": 16,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_generate_dependencies_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"D\"\n }\n ],\n \"title\": \"Dependency Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 34\n },\n \"id\": 18,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"workflow_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Workflow Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 6,\n \"w\": 12,\n \"x\": 0,\n \"y\": 42\n },\n \"id\": 27,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_remaining_capacity\",\n \"legendFormat\": \"remaining_capacity_{{hostname}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Remaining Instance Capacity\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 42\n },\n \"id\": 20,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_consumed_capacity\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Consumed Instance Capacity\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Jobs and Capacity\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 3\n },\n \"id\": 33,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"__systemRef\": \"hideSeriesFrom\",\n \"matcher\": {\n \"id\": \"byNames\",\n \"options\": {\n \"mode\": \"exclude\",\n \"names\": [\n \"task_manager_pending_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_running_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_tasks_started{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\"\n ],\n \"prefix\": \"All except:\",\n \"readOnly\": true\n }\n },\n \"properties\": [\n {\n \"id\": \"custom.hideFrom\",\n \"value\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": true\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 27\n },\n \"id\": 12,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_running_processed\",\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_pending_processed\",\n \"hide\": false,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_blocked\",\n \"hide\": false,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_started\",\n \"hide\": false,\n \"refId\": \"C\"\n }\n ],\n \"title\": \"Task manager workload\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 27\n },\n \"id\": 10,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_pending_tasks_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_running_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_commit_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager__schedule_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"E\"\n }\n ],\n \"title\": \"Task manager timings\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Task Manager\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 4\n },\n \"id\": 31,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 18,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"smooth\",\n \"lineStyle\": {\n \"fill\": \"solid\"\n },\n \"lineWidth\": 1,\n \"pointSize\": 6,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 28\n },\n \"id\": 26,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_event_processing_avg_seconds\",\n \"legendFormat\": \"{{node}}-processing-avg-seconds\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Avg Job Event Processing Lag Time\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 31,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 1,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_2-job-event-processing-rate\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-redis-queue\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 28\n },\n \"id\": 24,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_events_queue_size_redis\",\n \"legendFormat\": \"{{node}}-redis-queue\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"irate(callback_receiver_events_insert_db[1m])*60\",\n \"hide\": ","tools/grafana/dashboards/services_dashboard.json":"# Dashboards Skill\n\nPath: `tools/grafana/dashboards/services_dashboard.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/grafana/dashboards/services_dashboard.json)","tools/scripts/ig-hotfix/README.md":"# ig-hotfix Documentation\n\nPath: `tools/scripts/ig-hotfix/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/ig-hotfix/README.md)","tools/scripts/webhook_examples/github_push.json":"# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/github_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/github_push.json)","tools/scripts/webhook_examples/gitlab_push.json":"# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/gitlab_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/gitlab_push.json)","awx/main/tests/data/inventory/plugins/vmware/env.json":"# Vmware Skill\n\nPath: `awx/main/tests/data/inventory/plugins/vmware/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/vmware/env.json)","awx/main/tests/data/inventory/plugins/terraform/env.json":"# Terraform Skill\n\nPath: `awx/main/tests/data/inventory/plugins/terraform/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/terraform/env.json)","awx/main/tests/data/inventory/plugins/satellite6/env.json":"# Satellite6 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/satellite6/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/satellite6/env.json)","awx/main/tests/data/inventory/plugins/rhv/env.json":"# Rhv Skill\n\nPath: `awx/main/tests/data/inventory/plugins/rhv/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/rhv/env.json)","awx/main/tests/data/inventory/plugins/openstack/env.json":"# Openstack Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openstack/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openstack/env.json)","awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json":"# Openshift_virtualization Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json)","awx/main/tests/data/inventory/plugins/insights/env.json":"# Insights Skill\n\nPath: `awx/main/tests/data/inventory/plugins/insights/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/insights/env.json)","awx/main/tests/data/inventory/plugins/gce/env.json":"# Gce Skill\n\nPath: `awx/main/tests/data/inventory/plugins/gce/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/gce/env.json)","awx/main/tests/data/inventory/plugins/ec2/env.json":"# Ec2 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/ec2/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/ec2/env.json)","awx/main/tests/data/inventory/plugins/controller/env.json":"# Controller Skill\n\nPath: `awx/main/tests/data/inventory/plugins/controller/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/controller/env.json)","awx/main/tests/data/inventory/plugins/azure_rm/env.json":"# Azure_rm Skill\n\nPath: `awx/main/tests/data/inventory/plugins/azure_rm/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/azure_rm/env.json)"},"files":{"tools/community-bugscrub/README.md":"# Community BugScrub tooling\n\nSmall python script that automatically distributes PRs and Issues given a list of `people` and dumps the contents in a Spreadsheet.\n\nTo be used when distributing the work of reviewing community contributions.\n\n## Usage\n\nInstall requirements.\n\n```\npip install -r requirements.txt\n```\n\nGet the usage.\n\n```\npython generate-sheet.py -h\n```\n\n## Adding a github Personal Access Token\nThe scripts looks first for a github personal access token to use to avoid having the scripts calls rate limited, you can create one or use an existing one if you have. The script looks for the PAT under the environment var `GITHUB_ACCESS_TOKEN`. \n\n\n# For internal spreadsheet usage\nAWX engineers will need to import the data generated from the script into a spreadshet manager. Please make sure that you do not replace the existing sheets but make a new one or create a new sheet inside the existing spreadsheet upon import. ","awx_collection/tools/README.md":"## Collection tools\n\nTools used for building, maintaining, and testing the collection.\n\n### Template Galaxy\n\nThe `template_galaxy.yml` playbook ran as a pre-requisite for building the collection.\n\n```\nmake awx_collection_build\n```\n\n### Generate\n\nThis will template resource modules (like `group`, for groups in inventory) from a boilerplate template.\nIt is intended as a tool for writing new modules or enforcing consistency.\n\n### Integration Testing\n\nThese instructions assume you have ansible-core and the collection installed.\nTo install the collection in-place (to pick up any local changes to source)\nthe `make symlink_collection` will symlink the `awx_collection/` folder to\nthe appropriate place under `~/.ansible/collections`.\n\nThis is a shortcut for quick validation of tests that bypasses `ansible-test`.\nTo use this, you need the `~/.tower_cli.cfg` config file populated,\nwhich can be done via the deprecated `tower-cli login ` or manually\nwriting it, where the format looks like:\n\n```\n[general]\nhost = https://localhost:8043/\nverify_ssl = false\nusername = admin\npassword = password\n```\n\nTODO: adjust playbook to allow using environment variables as well.\n\nTo run some sample modules:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml\n```\n\nTo run just one module (the most common use case), use the `-e test=`.\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=host\n```\n\nIf you want to run _all_ the tests, then you need to pass in the whole list.\nThis will take significant time and is not ideal from an error-handling perspective,\nbut this is a way to do it:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=$(ls -1Am awx_collection/tests/integration/targets/ | tr -d '[:space:]')\n```\n\nDepending on the module, you may need special dependencies.\nFor instance, the rrule lookup plugins need `pytz`.\nThese will be satisfied if you install requirements in `awx_collection/requirements.txt`.\n","tools/loki/local-config.yaml":"auth_enabled: false\n\nserver:\n http_listen_port: 3100\n grpc_server_max_recv_msg_size: 524288000 # 500 MB\n grpc_server_max_send_msg_size: 524288000 # 500 MB, might be too much, be careful\n\nfrontend_worker:\n match_max_concurrent: true\n grpc_client_config:\n max_send_msg_size: 524288000 # 500 MB\n\n\ningester:\n max_chunk_age: 8766h\n\ncommon:\n path_prefix: /loki\n storage:\n filesystem:\n chunks_directory: /loki/chunks\n rules_directory: /loki/rules\n replication_factor: 1\n ring:\n kvstore:\n store: inmemory\n\n# compactor:\n# retention_enabled: true\n# # cmeyers: YOLO. 1s seems wrong but it works so right\n# compaction_interval: 1s # default 10m\n\nschema_config:\n configs:\n - from: 2020-10-24\n store: boltdb-shipper\n object_store: filesystem\n schema: v11\n index:\n prefix: index_\n period: 24h\n\nstorage_config:\n boltdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/boltdb-cache\n\nruler:\n alertmanager_url: http://localhost:9093\n\nlimits_config:\n retention_period: 3y\n # cmeyers: The default of 30m triggers a loop of queries that take a long time\n # to complete and the UI times out\n split_queries_by_interval: 1d\n # cmeyers: Default of 30d1h limits grafana time queries. Can't, for example,\n # query last 90 days\n max_query_length: 3y\n # cmeyers: Made the batch post request succeed.\n reject_old_samples: false\n reject_old_samples_max_age: 365d\n\n ingestion_rate_mb: 32\n ingestion_burst_size_mb: 32\n per_stream_rate_limit: 32M\n per_stream_rate_limit_burst: 32M\n ingestion_rate_strategy: local # Default: global\n max_global_streams_per_user: 100000000\n max_entries_limit_per_query: 100000000\n max_query_series: 1000000\n max_query_parallelism: 32 # Old Default: 14\n max_streams_per_user: 100000000 # Old Default: 10000\n\n# Taken from aap-log-visualizer\nfrontend:\n max_outstanding_per_tenant: 2048\n\nquery_scheduler:\n max_outstanding_requests_per_tenant: 2048\n\nquery_range:\n parallelise_shardable_queries: false\n split_queries_by_interval: 0\n\n# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration\n# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/\n#\n# Statistics help us better understand how Loki is used, and they show us performance\n# levels for most users. This helps us prioritize features and documentation.\n# For more information on what's sent, look at\n# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go\n# Refer to the buildReport method to see what goes into a report.\n#\n# If you would like to disable reporting, uncomment the following lines:\n#analytics:\n# reporting_enabled: false\n","tools/grafana/README.md":"### Prometheus and Grafana\nPrometheus is a metrics collecting tool, and we support prometheus formatted data at the`api/v2/metrics` endpoint.\n\nPrometheus is configured to poll targets on some interval and can use various authentication methods.\nWe configure prometheus to use basic auth and poll AWX every 5 seconds by default. This configuration is created using the `docker-compose/ansible/roles/sources/templates/prometheus.yml.j2` template. To change the values, modify the template or set the ansible variables used in that template (see use of `EXTRA_SOURCES_ANSIBLE_OPTS` in the next section).\n\nGrafana provides dashboards to visualize this data as well as alerting features.\n\nWe use the \"provisioning files\" functionality in Grafana to be able to deploy alert and dashboard configs from file.\n\nThese config files are located in directories that are mounted into the container in `tools/grafana`.\n\n### How to run docker-compose with Grafana and Prometheus\n\nTo run the development environment (see [docs](https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md)) with Prometheus and Grafana enabled, set the following variables:\n\n```\n$ PROMETHEUS=yes GRAFANA=yes make docker-compose\n```\n\nIf we want to pass additional ansible vars used in the deployment process, we can pass `EXTRA_SOURCES_ANSIBLE_OPTS`:\n\n```\nGRAFANA=true PROMETHEUS=true EXTRA_SOURCES_ANSIBLE_OPTS=\"-e scrape_interval=1 admin_password=foobar\" make docker-compose\n```\n\n### Where to view grafana and prometheus\n\n1. navigate to Prometheus at `http://localhost:9090/targets` and check that the metrics endpoint State is Up.\n2. Click the Graph tab, start typing a metric name, or use the Open metrics explorer button to find a metric to display (next to `Execute` button)\n3. Navigate to Grafana at `http://localhost:3001`. Sign in, using `admin` for both username and password.\n4. In the left navigation menu go to Dashboards->Browse, find the \"awx-demo\" and click. These should have graphs.\n5. Now you can modify these and add panels for whichever metrics you like.\n\n### Alerts in Grafana\n\nWe are configuring alerts in grafana using the provisioning files method. This feature is new in Grafana as of August, 2022. Documentation can be found: https://grafana.com/docs/grafana/latest/administration/provisioning/#alerting however it does not fully show all parameters to the config.\n\nOne way to understand how to build rules is to build them in the UI and use chrometools to inspect the payload as you save the rules. It appears that the \"data\" portion of the payload for each rule is the same syntax as needed in the provisioning file config. To reload the alerts without restarting the container, from your terminal you can send a POST with `curl -X POST http://admin:admin@localhost:3001/api/admin/provisioning/alerting/reload`.\n\nAnother way to export rules is explore the api.\n1. Get all the folders: `GET` to `/api/folders`\n2. Get the rules `GET` to `/api/ruler/grafana/api/v1/rules/{{ Folder }}`\n\nYou can do this via curl or in the web browser.\n\n### Included Alerts\n\n#### Alert if remaining capacity low and pending jobs exist\n\nWe want to know if jobs are in pending but we lack capacity in the cluster to run them. Our approach is to sum all remaining capacity in the cluster and compare it to the total capacity of the cluster. If less than 10% of our capacity is remaining and we have pending jobs, and this is true for more than 180s, we will fire the alert.\n\nThis alert is named \"capacity_below_10_percent\" and can be found in this directory in https://github.com/ansible/awx/blob/devel/tools/grafana/alerting/alerts.yml\n","tools/otel/otel-collector-config.yaml":"receivers:\n otlp:\n protocols:\n grpc:\n http:\n\nexporters:\n debug:\n verbosity: detailed\n\n file:\n path: /awx-logs/awx-logs.json.zstd\n rotation:\n max_days: 14\n localtime: false\n max_megabytes: 300\n max_backups: 200\n format: json\n compression: zstd\n\n loki:\n endpoint: http://loki:3100/loki/api/v1/push\n tls:\n insecure: true\n headers:\n \"X-Scope-OrgID\": \"1\"\n default_labels_enabled:\n exporter: true\n job: true\n instance: true\n level: true\n\nprocessors:\n batch:\n\nextensions:\n health_check:\n zpages:\n endpoint: \":55679\"\n\nservice:\n pipelines:\n logs:\n receivers: [otlp]\n processors: [batch]\n exporters: [file, loki]\n\n extensions:\n - health_check\n - zpages\n","tools/docker-compose/logstash-override.yaml":"---\nservices:\n # A useful container that simply passes through log messages to the console\n # helpful for testing awx/tower logging\n logstash:\n build:\n context: ../\n dockerfile: Dockerfile-logstash\n container_name: tools_logstash_1\n hostname: logstash\n networks:\n - awx\n","tools/docker-compose/README.md":"# Docker Compose for Development\n\n## Getting started\n\n### Clone the repo\n\nIf you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). We generally recommend that you view the releases page:\n\nhttps://github.com/ansible/awx/releases/latest\n\n...and clone the latest stable tag, e.g.,\n\n`git clone -b x.y.z https://github.com/ansible/awx.git`\n\nPlease note that deploying from `HEAD` (or the latest commit) is **not** stable, and that if you want to do this, you should proceed at your own risk.\n\nFor more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).\n\nOnce you have a local copy, run the commands in the following sections from the root of the project tree.\n\n## Overview\n\nHere are the main `make` targets:\n\n- `docker-compose-build` - used for building the development image, which is used by the `docker-compose` target. You can skip this target if you want to use the latest [ghcr.io/ansible/awx_devel:devel](https://github.com/ansible/awx/pkgs/container/awx_devel) image rather than build a new one.\n- `docker-compose` - make target for development, passes awx_devel image and tag\n\nNotable files:\n\n- `tools/docker-compose/inventory` file - used to configure the AWX development environment.\n- `tools/docker-compose/ansible/migrate.yml` - playbook for migrating data from Local Docker to the Development Environment\n\n### Prerequisites\n\n- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the `docker` group, refer to the documentation for details)\n- [Docker Compose](https://docs.docker.com/compose/install/).\n- [Ansible](https://docs.ansible.com/projects/ansible/latest/installation_guide/intro_installation.html) will need to be installed as we use it to template files needed for the docker-compose.\n- OpenSSL.\n\n### Tested Operating Systems\n\nThe docker-compose development environment is regularly used and should work on x86_64 systems running:\n\n- Fedora (maintained versions)\n- Ubuntu LTS (18, 20)\n- Red Hat Enterprise Linux 8, CentOS Stream 8\n- macOS 11\n\nUse on other platforms is untested, and may require local changes.\n\n### How to Disable Color?\n\nThere are several layers that might apply color:\n - docker compose coloring based on what container log comes from\n - supervisord coloring based on what server it comes from\n - general coloration from Django management commands\n\nThese can have color turned off by adding things to the\nenvironment variable to modify the call options.\n\n```\nDJANGO_COLORS=nocolor COMPOSE_UP_OPTS=\"--no-color\" SUPERVISOR_ARGS=\"-n -t\" make docker-compose\n```\n\nThis can be useful if this is ran in CI in any context.\n\n## Starting the Development Environment\n\n### Build the Image\n\nThe AWX base container image (defined in the Dockerfile templated from [Dockerfile.j2](./../ansible/roles/dockerfile/templates/Dockerfile.j2)) contains basic OS dependencies and symbolic links into the development environment that make running the services easy.\n\nRun the following to build the image:\n\n```bash\n$ make docker-compose-build\n```\n\n> The image will need to be rebuilt if there are any changes to Dockerfile.j2 or any of the files used by the templated Dockerfile.\n\nOnce the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows:\n\n```bash\n(host)$ docker images\n\nREPOSITORY TAG IMAGE ID CREATED SIZE\nansible/awx_devel latest ba9ec3e8df74 26 minutes ago 1.42GB\n```\n\n> By default, this image will be tagged with your branch name. You can specify a custom tag by setting an environment variable, for example: `DEVEL_IMAGE_NAME=quay.io/your_user/awx_devel:17.0.1`\n\n#### Customizing the Receptor Image\n\nBy default, the development environment will use the `devel` image from receptor.\nThis is used directly in `docker-compose.yml` for the hop nodes.\nThe receptor binary is also copied over to the main awx_devel image, used in all other AWX nodes.\n\nBecause of this, the `RECEPTOR_IMAGE` environment variable must be set when running\nboth docker-compose-build and docker-compose in order to use the correct receptor in all containers.\n\nIf you need to create a new receptor image, you can check out receptor and build it like this:\n\n```bash\nCONTAINERCMD=docker TAG=quay.io/ansible/receptor:release_1.1 make container\n```\n\nThen that can be used by AWX like this:\n\n```bash\nexport RECEPTOR_IMAGE=quay.io/ansible/receptor:release_1.1\nmake docker-compose-build\nmake docker-compose\n```\n\n### Run AWX\n\n##### Start the containers\n\nRun the awx, postgres and redis containers. This utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the awx container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process.\n\n```bash\n$ make docker-compose\n```\n\n> The make target assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG= `in your environment. For example, you might be working in a feature branch, but you want to run the containers using the devel image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose`\n\n> For running docker-compose detached mode, start the containers using the following command: `$ make docker-compose COMPOSE_UP_OPTS=-d`\n\n###### Solving database-related issues during initial startup\n\nIf you have encountered the infinitely-repeating `Waiting for postgres to be ready to accept connections` message during the execution, try to do the following:\n\n1. Stop and delete AWX-related docker containers.\n2. Delete all associated docker volumes.\n3. Delete all associated docker networks.\n4. Repeat the process from scratch.\n\nIf you have **only** AWX-related container entities in your system, you can simply stop and delete everything using the following commands:\n\n```bash\ndocker stop $(docker ps -a -q)\ndocker system prune -a\ndocker volume prune\ndocker network prune\n```\n\n##### _(alternative method)_ Spin up a development environment with customized mesh node cluster\n\nA cluster (of containers) with execution nodes and a hop node can be created by the docker-compose Makefile target.\nBy default, it will create 1 hybrid node.\nYou can switch the type of AWX nodes between hybrid and control with `MAIN_NODE_TYPE`.\n\n```\nMAIN_NODE_TYPE=control EXECUTION_NODE_COUNT=2 COMPOSE_TAG=devel make docker-compose\n```\n\nRunning the above command will create a cluster of 1 control node, 1 hop node, and 2 execution nodes.\nA hop node is automatically created whenever there are 1 or more execution nodes.\n\nThe number of nodes can be changed:\n\n```\nCONTROL_PLANE_NODE_COUNT=2 EXECUTION_NODE_COUNT=3 COMPOSE_TAG=devel make docker-compose\n```\n\nThis will spin up a topology represented below.\n(names are the receptor node names, which differ from the AWX Instance names and network address in some cases)\n\n```\n ┌──────────────┐\n │ │\n┌──────────────┐ ┌──────────┤ receptor-1 │\n│ │ │ │ │\n│ awx_1 │◄──────────┐ │ └──────────────┘\n│ │ │ ▼\n└──────┬───────┘ ┌──────┴───────┐ ┌──────────────┐\n │ │ │ │ │\n │ │ receptor-hop │◄───────┤ receptor-2 │\n ▼ │ │ │ │\n┌──────────────┐ └──────────────┘ └──────────────┘\n│ │ ▲\n│ awx_2 │ │ ┌──────────────┐\n│ │ │ │ │\n└──────────────┘ └──────────┤ receptor-3 │\n │ │\n └──────────────┘\n```\n\nAll execution (`receptor-*`) nodes connect to the hop node.\nOnly the `awx_1` node connects to the hop node out of the AWX cluster.\n`awx_1` connects to `awx_2`, fulfilling the requirement that the AWX cluster is fully connected.\n\nFor example, if a job is launched with `awx_2` as the `controller_node` and `receptor-3` as the `execution_node`,\nthen `awx_2` communicates to `receptor-3` via `awx_1` and then `receptor-hop`.\n\n\n##### Wait for migrations to complete\n\nThe first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following:\n\n```bash\nawx_1 | Operations to perform:\nawx_1 | Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic\nawx_1 | Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, contenttypes, auth, conf, main\nawx_1 | Synchronizing apps without migrations:\nawx_1 | Creating tables...\nawx_1 | Running deferred SQL...\nawx_1 | Installing custom SQL...\nawx_1 | Running migrations:\nawx_1 | Rendering model states... DONE\nawx_1 | Applying contenttypes.0001_initial... OK\nawx_1 | Applying contenttypes.0002_remove_content_type_name... OK\nawx_1 | Applying auth.0001_initial... OK\n...\n```\n\n##### Clean and build the UI\n\nPrerequisites (on your local machine)\n- npm\n- nodejs\n\nRequired versions listed here https://github.com/ansible/ansible-ui/blob/main/README.md\n\nOn your local machine (not in awx container)\n\n```bash\nmake clean/ui ui\n```\n\nThis will clone the ansible-ui into the `awx/ui/src` directory and build the static files. Then when the containers come up, awx-manage collectstatic will copy those files into the proper place.\n\nYou can also use `UI_LOCAL` to build from a locally cloned ansible-ui repo.\n\nSee [the ui development documentation](https://github.com/ansible/ansible-ui/blob/main/CONTRIBUTING.md) for more information on using the frontend development, build, and test tooling.\n\nOnce migrations are completed and the UI is built, you can begin using AWX. The UI can be reached in your browser at `https://localhost:8043/`, and the API can be found at `https://localhost:8043/api/v2`.\n\n##### Create an admin user\n\nBefore you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command:\n\n```bash\n$ docker exec -ti tools_awx_1 awx-manage createsuperuser\n```\n\n> Remember the username and password, as you will use them to log into the web interface for the first time.\n\n##### Load demo data\n\nOptionally, you may also want to load some demo data. This will create a demo project, inventory, and job template.\n\n```bash\n$ docker exec tools_awx_1 awx-manage create_preload_data\n```\n\n> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate\n> this container, and thus the database, if the database schema changes in an upstream commit.\n\n## Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior), you can\nmigrate your data to the development environment via the migrate.yml playbook using the steps described [here](./docs/data_migration.md).\n\n## Upgrading the Development Environment\n\nUpgrading AWX involves checking out the new source code and re-running the make target. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables.\n\nAfter updating the inventory file with any custom values, run the make target from the root of your AWX clone.\n\n```bash\n$ make docker-compose\n```\n\n## Extras\n\n- [Start a shell](#start-a-shell)\n- [Start AWX from the container shell](#start-awx-from-the-container-shell)\n- [Using Logstash](./docs/logstash.md)\n- [Start a Cluster](#start-a-cluster)\n- [Start with Minikube](#start-with-minikube)\n- [Splunk Integration](#splunk-integration)\n\n### Start a Shell\n\nTo run `awx-manage` commands and modify things inside the container, you will want to start a shell session on the _awx_ container. In a new terminal session, use the `docker exec` command to start the shell session:\n\n```bash\n(host)$ docker exec -it tools_awx_1 bash\n```\n\nThis creates a session in the _awx_ containers, just as if you were using `ssh`, and allows you execute commands within the running container.\n\n### Start AWX from the container shell\n\nOften times you'll want to start the development environment without immediately starting all of the services in the _awx_ container, and instead be taken directly to a shell. You can do this with the following:\n\n```bash\n(host)$ make docker-compose-test\n```\n\nUsing `docker exec`, this will create a session in the running _awx_ container, and place you at a command prompt, where you can run shell commands inside the container.\n\nIf you want to start and use the development environment, you'll first need to bootstrap it by running the following command:\n\n```bash\n(container)# /usr/bin/bootstrap_development.sh\n```\n\nThe above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it\nwill drop you back to the shell.\n\nIn order to launch all developer services:\n\n```bash\n(container)# /usr/bin/launch_awx.sh\n```\n\n`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't\nneed to call `bootstrap_development.sh` first.\n\n### Start a Cluster\n\nCertain features or bugs are only applicable when running a cluster of AWX nodes. To bring up a 3 node cluster development environment simply run the below command.\n\n```bash\n(host)$ CONTROL_PLANE_NODE_COUNT=3 make docker-compose\n```\n\n`CONTROL_PLANE_NODE_COUNT` is configurable and defaults to 1, effectively a non-clustered AWX.\n\nNote that you may see multiple messages of the form `2021-03-04 20:11:47,666 WARNING [-] awx.main.wsrelay Connection from awx_2 to awx_5 failed: 'Cannot connect to host awx_5:8013 ssl:False [Name or service not known]'.`. This can happen when you bring up a cluster of many nodes, say 10, then you bring up a cluster of less nodes, say 3. In this example, there will be 7 `Instance` records in the database that represent AWX instances. The AWX development environment mimics the VM deployment (vs. kubernetes) and expects the missing nodes to be brought back to healthy by the admin. The warning message you are seeing is all of the AWX nodes trying to connect the websocket backplane. You can manually delete the `Instance` records from the database i.e. `Instance.objects.get(hostname='awx_9').delete()` to stop the warnings.\n\n### Start with Minikube\n\nTo bring up a 1 node AWX + minikube that is accessible from AWX run the following.\n\nStart minikube\n\n```bash\n(host)$minikube start --cpus=4 --memory=8g --addons=ingress\n```\n\nStart AWX\n\n```bash\n(host)$ make docker-compose-container-group\n```\n\nAlternatively, you can set the env var `MINIKUBE_CONTAINER_GROUP=true` to use the default dev env bring up. his way you can use other env flags like the cluster node count. Set `MINIKUBE_SETUP=true` to make the roles download, install and run minikube for you, but if you run into issues with this just start minikube yourself.\n\n```bash\n(host)$ MINIKUBE_CONTAINER_GROUP=true make docker-compose\n```\n\nIf you want to clean all things once your are done, you can do:\n\n```bash\n(host)$ make docker-compose-container-group-clean\n```\n\n### Splunk Integration\n\nSplunk is a log aggregation tool that can be used to test AWX with external logging integration. This section describes how to build a reference Splunk instance and plumb it with your AWX for testing purposes.\n\nFirst, be sure that you have the awx.awx collection installed by running `make install_collection`.\n\nNext, install the splunk.es collection by running `ansible-galaxy collection install splunk.es`.\n\nAnytime you want to run a Splunk instance alongside AWX we can start docker-compose with the SPLUNK option to get a Splunk instance with the command:\n```bash\nSPLUNK=true make docker-compose\n```\n\nOnce the containers come up three new ports (8000, 8089 and 9199) should be exposed and the Splunk server should be running on some of those ports (the 9199 will be created later by the plumbing playbook). The first port (8000) is the non-SSL admin port and you can log into splunk with the credentials admin/splunk_admin. The url will be like http://:8000/ this will be referenced below. The 8089 is the API port that the ansible modules will use to connect to and configure splunk. The 9199 port will be used to construct a TCP listener in Splunk that AWX will forward messages to.\n\nOnce the containers are up we are ready to configure and plumb Splunk with AWX. To do this we have provided a playbook which will:\n* Backup and configure the External Logging adapter in AWX. NOTE: this will back up your existing settings but the password fields can not be backed up through the API, you need a DB backup to recover this.\n* Create a TCP port in Splunk for log forwarding\n\nFor routing traffic between AWX and Splunk we will use the internal docker compose network. The `Logging Aggregator` will be configured using the internal network machine name of `splunk`.\n\nOnce you have have the collections installed (from above) you can run the playbook like:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_splunk.yml\n```\n\nOnce the playbook is done running Splunk should now be setup in your development environment. You can log into the admin console (see above for username/password) and click on \"Searching and Reporting\" in the left hand navigation. In the search box enter `source=\"http:tower_logging_collections\"` and click search.\n\n### HashiVault Integration\n\nRun a HashiVault container alongside of AWX.\n\n```bash\nVAULT=true make docker-compose\n```\n\nYou can find the initialization data at `tools/docker-compose/_sources/secrets/vault_init.yml`,\nThis includes the unseal keys and a root token.\n\nYou will need to unseal the HashiVault each time the container is started.\nThe easiest way to do that is to run:\n```bash\nansible-playbook tools/docker-compose/ansible/unseal_vault.yml\n```\nThis will perform the unseal and also display the root token for login.\n\nFor demo purposes, Vault will be auto-configured to include a Key Value (KV) vault called `my_engine` along with a secret called `my_key` in `/my_engine/my_root/my_folder`.\nThe secret value is `this_is_the_secret_value`.\n\nTo create a secret connected to this vault in AWX you can run the following playbook:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_vault.yml\n```\n\nThis will create the following items in your AWX instance:\n* A credential called `Vault Lookup Cred` tied to the vault instance.\n* A credential called `Vault UserPass Lookup Cred` tied to the vault instance.\n* A custom credential type called `Vault Custom Cred Type`.\n* A credential called `Credential From HashiCorp Vault via Token Auth` which is of the created type using the `Vault Lookup Cred` to get the secret.\n* A credential called `Credential From HashiCorp Vault via UserPass Auth` which is of the created type using the `Vault Userpass Lookup Cred` to get the secret.\n\nThe custom credential type adds a variable when used in a playbook called `the_secret_from_vault`.\nIf you have a playbook like:\n```\n---\n- name: Show a vault secret\n hosts: localhost\n connection: local\n gather_facts: False\n tasks:\n - debug:\n var: the_secret_from_vault\n```\n\nAnd run it through AWX with the credential `Credential From Vault via Token Auth` tied to it, the debug should result in `this_is_the_secret_value`. If you run it through AWX with the credential `Credential From Vault via Userpass Auth`, the debug should result in `this_is_the_userpass_secret_value`.\n\n### Prometheus and Grafana integration\n\nSee docs at https://github.com/ansible/awx/blob/devel/tools/grafana/README.md\n\n### OpenTelemetry Integration\n\n```bash\nOTEL=true GRAFANA=true LOKI=true PROMETHEUS=true make docker-compose\n```\n\nThis will start the sidecar container `tools_otel_1` and configure AWX logging to send to it. The OpenTelemetry Collector is configured to export logs to Loki. Grafana is configured with Loki as a datasource. AWX logs can be viewed in Grafana.\n\n`http://localhost:3001` grafana\n","tools/docker-compose/docs/data_migration.md":"# Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior) to AWX 18.0 or higher, you can\nmigrate your data to the development environment via the migrate.yml playbook.\n\n> Note: This will also convert your postgresql bind-mount into a docker volume.\n\n### Migrate data with migrate.yml\n\nFirst, in the [`inventory` file](../inventory), set your `pg_password`, `broadcast_websocket_secret`, `secret_key`, and any other settings you need for your deployment. **Make sure you use the same values from the `inventory` file in your existing pre-18.0.0 Docker deployment.**\n\nIf you used a custom pgdocker or awxcompose location, you will need to set the `postgres_data_dir` and `old_docker_compose_dir` variables.\n\n1. Run the [migrate playbook](../ansible/migrate.yml) to migrate your data to the new postgresql container and convert the data directory to a volume mount.\n```bash\n$ ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/migrate.yml -e \"migrate_local_docker=true\" -e \"postgres_data_dir=~/.awx/pgdocker\" -e \"old_docker_compose_dir=~/.awx/awxcompose\"\n```\n\n2. Change directory to the top of your awx checkout, and follow the [development environment installation instructions](../README.md) to start your containers:\n```bash\n$ make docker-compose-build\n$ make docker-compose\n$ make ui\n```\n\n3. After ensuring your data has been successfully migrated, you may delete your old data directory (typically stored at `~/.awx/pgdocker`).\n\n4. Note that after migration, the development environment will be available at https://localhost:8043/ (not http://localhost).\n","tools/docker-compose/docs/logstash.md":"# How to use the logstash container\n\n#### Modify the docker-compose.yml\n\nYou can run the logstash container by adding another compose file to the docker-compose target.\n\n```\nCOMPOSE_OPTS=\"-f tools/docker-compose/logstash-override.yaml\" COMPOSE_TAG=devel make docker-compose\n```\n\nPOST the following content to `/api/v2/settings/logging/` (this uses\nauthentication set up inside of the logstash configuration file).\n\n```\n{\n \"LOG_AGGREGATOR_HOST\": \"http://logstash\",\n \"LOG_AGGREGATOR_PORT\": 8085,\n \"LOG_AGGREGATOR_TYPE\": \"logstash\",\n \"LOG_AGGREGATOR_USERNAME\": \"awx_logger\",\n \"LOG_AGGREGATOR_PASSWORD\": \"workflows\",\n \"LOG_AGGREGATOR_LOGGERS\": [\n \"awx\",\n \"activity_stream\",\n \"job_events\",\n \"system_tracking\",\n \"job_lifecycle\"\n ],\n \"LOG_AGGREGATOR_INDIVIDUAL_FACTS\": false,\n \"LOG_AGGREGATOR_TOWER_UUID\": \"991ac7e9-6d68-48c8-bbde-7ca1096653c6\",\n \"LOG_AGGREGATOR_ENABLED\": true\n}\n```\n\n> Note: HTTP must be specified in the `LOG_AGGREGATOR_HOST` if you are using the docker development environment. \n\nAn example of how to view the most recent logs from the container:\n\n```\ndocker exec -i -t $(docker ps -aqf \"name=tools_logstash_1\") tail -n 50 /logstash.log\n```\n\n#### How to add logstash plugins\n\nAdd any plugins you need in `tools/elastic/logstash/Dockerfile` before running the container. \n","tools/docker-compose/editable_dependencies/README.md":"# Editable dependencies in AWX Docker Compose Development Environment\n\nThis folder contains the symlink to editable dependencies for AWX\n\nDuring the bootstrap of awx development environment we will try to crawl through the symlinks and mount (the source of the symlink) to `tools_awx_` containers and `init_awx` containers then install all the dependencies in editable mode\n\n## How to enable/disable editable dependnecies\n\n### Enable\n\nSet `EDITABLE_DEPENDENCIES=true` either as an Environment Variable before invoking `make docker-compose`\n\n```bash\nexport EDITABLE_DEPENDENCIES=true\n```\n\nor during invocation of `make docker-compose`\n\n```bash\n\nEDITABLE_DEPENDENCIES=true make docker-compose\n```\n\nThis will cause the `make docker-compose-source` to template out docker-compose file with editable dependencies.\n\n### Disable\n\nTo disable editable dependency simply `unset EDITABLE_DEPENDENCIES`\n\n## How to add editable dependencies\n\nAdding symlink to the directory that contains the source of the editable dependencies will cause the dependency to be mounted and installed in the docker-compose development environment.\n\nBoth relative path or absolute path will work.\n\n### Examples\n\nI have `awx` checked out at `~/projects/src/github.com/TheRealHaoLiu/awx`\nI have `django-ansible-base` checked out at `~/projects/src/github.com/TheRealHaoLiu/django-ansible-base`\n\nFrom root of AWX project `~/projects/src/github.com/TheRealHaoLiu/awx`\n\nI can either do\n\n```bash\nln -s ~/projects/src/github.com/TheRealHaoLiu/django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\nor\n\n```bash\nln -s ../django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\n## How to remove indivisual editable dependencies\n\nSimply removing the symlink from `tools/docker-compose/editable_dependencies` **will cause problem**!\n\nand the volume `tools_var_lib_awx` needs to be deleted as well with\n\n```bash\nmake docker-compose-down\ndocker volume rm tools_var_lib_awx\n```\n\n","tools/sosreport/TESTING.md":"Create a RHEL box and then do the following.\n```bash\nsudo mkdir -p /usr/share/sosreport/sos/plugins\nsudo yum install sos\ncp controller.py /usr/share/sosreport/sos/plugins\nsudo chmod 644 /usr/share/sosreport/sos/plugins/controller.py\nln -s /usr/share/sosreport/sos/plugins/controller.py `find `find /usr/lib -name sos` -name plugins`\nsosreport -l | grep controller\n```\n\nThe results should be:\n```bash\n# sosreport -l | grep controller\n controller Ansible Automation Platform controller information\n```\n\nTo run only the controller plugin run: `sosreport --only-plugins controller`\n\n","tools/grafana/dashboards/demo_dashboard.json":"{\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": {\n \"type\": \"grafana\",\n \"uid\": \"-- Grafana --\"\n },\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"fiscalYearStartMonth\": 0,\n \"graphTooltip\": 0,\n \"id\": 2,\n \"links\": [],\n \"liveNow\": false,\n \"panels\": [\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 38,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"Fraction of time dispatcher is listening for new messages\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 39,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_availability\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Availability\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 1\n },\n \"id\": 40,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_max_worker_count\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_active_task_count\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Dispatcher Workers\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 9\n },\n \"id\": 41,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_scale_up_events\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Pool Scale-Up Events\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Dispatcher\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 37,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 14,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_database_connections_total\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Database\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {},\n \"fieldConfig\": {\n \"defaults\": {\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"short\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 25,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"/^tower_version$/\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"000000021\"\n },\n \"editorMode\": \"code\",\n \"exemplar\": false,\n \"expr\": \"awx_system_info\",\n \"format\": \"table\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Version\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"displayName\": \"Instances\",\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 30\n },\n \"id\": 13,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"center\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"count(awx_instance_info)\",\n \"interval\": \"\",\n \"legendFormat\": \" \",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Node Count\",\n \"type\": \"stat\"\n }\n ],\n \"title\": \"System\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 2\n },\n \"id\": 35,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"running\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"canceled\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"super-light-yellow\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 8,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"awx_status_total\",\n \"legendFormat\": \"{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"job status\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-error\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-successful\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-purple\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_3-failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-orange\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 29,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_status_total\",\n \"legendFormat\": \"{{node}}-{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Job Status per Instance\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 34\n },\n \"id\": 16,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_generate_dependencies_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"D\"\n }\n ],\n \"title\": \"Dependency Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 34\n },\n \"id\": 18,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"workflow_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Workflow Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 6,\n \"w\": 12,\n \"x\": 0,\n \"y\": 42\n },\n \"id\": 27,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_remaining_capacity\",\n \"legendFormat\": \"remaining_capacity_{{hostname}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Remaining Instance Capacity\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 42\n },\n \"id\": 20,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_consumed_capacity\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Consumed Instance Capacity\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Jobs and Capacity\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 3\n },\n \"id\": 33,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"__systemRef\": \"hideSeriesFrom\",\n \"matcher\": {\n \"id\": \"byNames\",\n \"options\": {\n \"mode\": \"exclude\",\n \"names\": [\n \"task_manager_pending_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_running_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_tasks_started{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\"\n ],\n \"prefix\": \"All except:\",\n \"readOnly\": true\n }\n },\n \"properties\": [\n {\n \"id\": \"custom.hideFrom\",\n \"value\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": true\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 27\n },\n \"id\": 12,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_running_processed\",\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_pending_processed\",\n \"hide\": false,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_blocked\",\n \"hide\": false,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_started\",\n \"hide\": false,\n \"refId\": \"C\"\n }\n ],\n \"title\": \"Task manager workload\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 27\n },\n \"id\": 10,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_pending_tasks_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_running_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_commit_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager__schedule_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"E\"\n }\n ],\n \"title\": \"Task manager timings\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Task Manager\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 4\n },\n \"id\": 31,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 18,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"smooth\",\n \"lineStyle\": {\n \"fill\": \"solid\"\n },\n \"lineWidth\": 1,\n \"pointSize\": 6,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 28\n },\n \"id\": 26,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_event_processing_avg_seconds\",\n \"legendFormat\": \"{{node}}-processing-avg-seconds\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Avg Job Event Processing Lag Time\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 31,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 1,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_2-job-event-processing-rate\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-redis-queue\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 28\n },\n \"id\": 24,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_events_queue_size_redis\",\n \"legendFormat\": \"{{node}}-redis-queue\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"irate(callback_receiver_events_insert_db[1m])*60\",\n \"hide\": ","tools/grafana/dashboards/services_dashboard.json":"# Dashboards Skill\n\nPath: `tools/grafana/dashboards/services_dashboard.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/grafana/dashboards/services_dashboard.json)","tools/scripts/ig-hotfix/README.md":"# ig-hotfix Documentation\n\nPath: `tools/scripts/ig-hotfix/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/ig-hotfix/README.md)","tools/scripts/webhook_examples/github_push.json":"# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/github_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/github_push.json)","tools/scripts/webhook_examples/gitlab_push.json":"# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/gitlab_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/gitlab_push.json)","awx/main/tests/data/inventory/plugins/vmware/env.json":"# Vmware Skill\n\nPath: `awx/main/tests/data/inventory/plugins/vmware/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/vmware/env.json)","awx/main/tests/data/inventory/plugins/terraform/env.json":"# Terraform Skill\n\nPath: `awx/main/tests/data/inventory/plugins/terraform/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/terraform/env.json)","awx/main/tests/data/inventory/plugins/satellite6/env.json":"# Satellite6 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/satellite6/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/satellite6/env.json)","awx/main/tests/data/inventory/plugins/rhv/env.json":"# Rhv Skill\n\nPath: `awx/main/tests/data/inventory/plugins/rhv/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/rhv/env.json)","awx/main/tests/data/inventory/plugins/openstack/env.json":"# Openstack Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openstack/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openstack/env.json)","awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json":"# Openshift_virtualization Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json)","awx/main/tests/data/inventory/plugins/insights/env.json":"# Insights Skill\n\nPath: `awx/main/tests/data/inventory/plugins/insights/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/insights/env.json)","awx/main/tests/data/inventory/plugins/gce/env.json":"# Gce Skill\n\nPath: `awx/main/tests/data/inventory/plugins/gce/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/gce/env.json)","awx/main/tests/data/inventory/plugins/ec2/env.json":"# Ec2 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/ec2/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/ec2/env.json)","awx/main/tests/data/inventory/plugins/controller/env.json":"# Controller Skill\n\nPath: `awx/main/tests/data/inventory/plugins/controller/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/controller/env.json)","awx/main/tests/data/inventory/plugins/azure_rm/env.json":"# Azure_rm Skill\n\nPath: `awx/main/tests/data/inventory/plugins/azure_rm/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/azure_rm/env.json)"},"items":[{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/azure_rm/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/azure_rm/env.json","title":"Azure_rm Skill","category":"anthropic-skill","format":"json","content":"# Azure_rm Skill\n\nPath: `awx/main/tests/data/inventory/plugins/azure_rm/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/azure_rm/env.json)","tokens":300,"sizeBytes":219},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/controller/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/controller/env.json","title":"Controller Skill","category":"anthropic-skill","format":"json","content":"# Controller Skill\n\nPath: `awx/main/tests/data/inventory/plugins/controller/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/controller/env.json)","tokens":300,"sizeBytes":225},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/ec2/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/ec2/env.json","title":"Ec2 Skill","category":"anthropic-skill","format":"json","content":"# Ec2 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/ec2/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/ec2/env.json)","tokens":300,"sizeBytes":204},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/gce/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/gce/env.json","title":"Gce Skill","category":"anthropic-skill","format":"json","content":"# Gce Skill\n\nPath: `awx/main/tests/data/inventory/plugins/gce/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/gce/env.json)","tokens":300,"sizeBytes":204},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/insights/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/insights/env.json","title":"Insights Skill","category":"anthropic-skill","format":"json","content":"# Insights Skill\n\nPath: `awx/main/tests/data/inventory/plugins/insights/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/insights/env.json)","tokens":300,"sizeBytes":219},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json","title":"Openshift_virtualization Skill","category":"anthropic-skill","format":"json","content":"# Openshift_virtualization Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json)","tokens":300,"sizeBytes":267},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/openstack/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openstack/env.json","title":"Openstack Skill","category":"anthropic-skill","format":"json","content":"# Openstack Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openstack/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openstack/env.json)","tokens":300,"sizeBytes":222},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/rhv/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/rhv/env.json","title":"Rhv Skill","category":"anthropic-skill","format":"json","content":"# Rhv Skill\n\nPath: `awx/main/tests/data/inventory/plugins/rhv/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/rhv/env.json)","tokens":300,"sizeBytes":204},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/satellite6/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/satellite6/env.json","title":"Satellite6 Skill","category":"anthropic-skill","format":"json","content":"# Satellite6 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/satellite6/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/satellite6/env.json)","tokens":300,"sizeBytes":225},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/terraform/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/terraform/env.json","title":"Terraform Skill","category":"anthropic-skill","format":"json","content":"# Terraform Skill\n\nPath: `awx/main/tests/data/inventory/plugins/terraform/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/terraform/env.json)","tokens":300,"sizeBytes":222},{"name":"env.json","path":"awx/main/tests/data/inventory/plugins/vmware/env.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/vmware/env.json","title":"Vmware Skill","category":"anthropic-skill","format":"json","content":"# Vmware Skill\n\nPath: `awx/main/tests/data/inventory/plugins/vmware/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/vmware/env.json)","tokens":300,"sizeBytes":213},{"name":"data_migration.md","path":"tools/docker-compose/docs/data_migration.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/docker-compose/docs/data_migration.md","title":"Docs Skill","category":"anthropic-skill","format":"markdown","content":"# Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior) to AWX 18.0 or higher, you can\nmigrate your data to the development environment via the migrate.yml playbook.\n\n> Note: This will also convert your postgresql bind-mount into a docker volume.\n\n### Migrate data with migrate.yml\n\nFirst, in the [`inventory` file](../inventory), set your `pg_password`, `broadcast_websocket_secret`, `secret_key`, and any other settings you need for your deployment. **Make sure you use the same values from the `inventory` file in your existing pre-18.0.0 Docker deployment.**\n\nIf you used a custom pgdocker or awxcompose location, you will need to set the `postgres_data_dir` and `old_docker_compose_dir` variables.\n\n1. Run the [migrate playbook](../ansible/migrate.yml) to migrate your data to the new postgresql container and convert the data directory to a volume mount.\n```bash\n$ ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/migrate.yml -e \"migrate_local_docker=true\" -e \"postgres_data_dir=~/.awx/pgdocker\" -e \"old_docker_compose_dir=~/.awx/awxcompose\"\n```\n\n2. Change directory to the top of your awx checkout, and follow the [development environment installation instructions](../README.md) to start your containers:\n```bash\n$ make docker-compose-build\n$ make docker-compose\n$ make ui\n```\n\n3. After ensuring your data has been successfully migrated, you may delete your old data directory (typically stored at `~/.awx/pgdocker`).\n\n4. Note that after migration, the development environment will be available at https://localhost:8043/ (not http://localhost).\n","isInternal":false,"tokens":394,"sizeBytes":1647},{"name":"logstash.md","path":"tools/docker-compose/docs/logstash.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/docker-compose/docs/logstash.md","title":"Docs Skill","category":"anthropic-skill","format":"markdown","content":"# How to use the logstash container\n\n#### Modify the docker-compose.yml\n\nYou can run the logstash container by adding another compose file to the docker-compose target.\n\n```\nCOMPOSE_OPTS=\"-f tools/docker-compose/logstash-override.yaml\" COMPOSE_TAG=devel make docker-compose\n```\n\nPOST the following content to `/api/v2/settings/logging/` (this uses\nauthentication set up inside of the logstash configuration file).\n\n```\n{\n \"LOG_AGGREGATOR_HOST\": \"http://logstash\",\n \"LOG_AGGREGATOR_PORT\": 8085,\n \"LOG_AGGREGATOR_TYPE\": \"logstash\",\n \"LOG_AGGREGATOR_USERNAME\": \"awx_logger\",\n \"LOG_AGGREGATOR_PASSWORD\": \"workflows\",\n \"LOG_AGGREGATOR_LOGGERS\": [\n \"awx\",\n \"activity_stream\",\n \"job_events\",\n \"system_tracking\",\n \"job_lifecycle\"\n ],\n \"LOG_AGGREGATOR_INDIVIDUAL_FACTS\": false,\n \"LOG_AGGREGATOR_TOWER_UUID\": \"991ac7e9-6d68-48c8-bbde-7ca1096653c6\",\n \"LOG_AGGREGATOR_ENABLED\": true\n}\n```\n\n> Note: HTTP must be specified in the `LOG_AGGREGATOR_HOST` if you are using the docker development environment. \n\nAn example of how to view the most recent logs from the container:\n\n```\ndocker exec -i -t $(docker ps -aqf \"name=tools_logstash_1\") tail -n 50 /logstash.log\n```\n\n#### How to add logstash plugins\n\nAdd any plugins you need in `tools/elastic/logstash/Dockerfile` before running the container. \n","isInternal":false,"tokens":374,"sizeBytes":1351},{"name":"logstash-override.yaml","path":"tools/docker-compose/logstash-override.yaml","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/docker-compose/logstash-override.yaml","title":"Docker-compose Skill","category":"anthropic-skill","format":"yaml","content":"---\nservices:\n # A useful container that simply passes through log messages to the console\n # helpful for testing awx/tower logging\n logstash:\n build:\n context: ../\n dockerfile: Dockerfile-logstash\n container_name: tools_logstash_1\n hostname: logstash\n networks:\n - awx\n","isInternal":false,"tokens":74,"sizeBytes":300},{"name":"demo_dashboard.json","path":"tools/grafana/dashboards/demo_dashboard.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/grafana/dashboards/demo_dashboard.json","title":"Dashboards Skill","category":"anthropic-skill","format":"json","content":"{\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": {\n \"type\": \"grafana\",\n \"uid\": \"-- Grafana --\"\n },\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"fiscalYearStartMonth\": 0,\n \"graphTooltip\": 0,\n \"id\": 2,\n \"links\": [],\n \"liveNow\": false,\n \"panels\": [\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 38,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"Fraction of time dispatcher is listening for new messages\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 39,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_availability\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Availability\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 1\n },\n \"id\": 40,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_max_worker_count\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_active_task_count\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Dispatcher Workers\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 9\n },\n \"id\": 41,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_scale_up_events\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Pool Scale-Up Events\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Dispatcher\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 37,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 14,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_database_connections_total\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Database\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {},\n \"fieldConfig\": {\n \"defaults\": {\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"short\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 25,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"/^tower_version$/\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"000000021\"\n },\n \"editorMode\": \"code\",\n \"exemplar\": false,\n \"expr\": \"awx_system_info\",\n \"format\": \"table\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Version\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"displayName\": \"Instances\",\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 30\n },\n \"id\": 13,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"center\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"count(awx_instance_info)\",\n \"interval\": \"\",\n \"legendFormat\": \" \",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Node Count\",\n \"type\": \"stat\"\n }\n ],\n \"title\": \"System\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 2\n },\n \"id\": 35,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"running\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"canceled\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"super-light-yellow\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 8,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"awx_status_total\",\n \"legendFormat\": \"{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"job status\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-error\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-successful\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-purple\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_3-failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-orange\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 29,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_status_total\",\n \"legendFormat\": \"{{node}}-{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Job Status per Instance\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 34\n },\n \"id\": 16,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_generate_dependencies_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"D\"\n }\n ],\n \"title\": \"Dependency Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 34\n },\n \"id\": 18,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"workflow_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Workflow Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 6,\n \"w\": 12,\n \"x\": 0,\n \"y\": 42\n },\n \"id\": 27,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_remaining_capacity\",\n \"legendFormat\": \"remaining_capacity_{{hostname}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Remaining Instance Capacity\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 42\n },\n \"id\": 20,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_consumed_capacity\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Consumed Instance Capacity\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Jobs and Capacity\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 3\n },\n \"id\": 33,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"__systemRef\": \"hideSeriesFrom\",\n \"matcher\": {\n \"id\": \"byNames\",\n \"options\": {\n \"mode\": \"exclude\",\n \"names\": [\n \"task_manager_pending_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_running_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_tasks_started{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\"\n ],\n \"prefix\": \"All except:\",\n \"readOnly\": true\n }\n },\n \"properties\": [\n {\n \"id\": \"custom.hideFrom\",\n \"value\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": true\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 27\n },\n \"id\": 12,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_running_processed\",\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_pending_processed\",\n \"hide\": false,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_blocked\",\n \"hide\": false,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_started\",\n \"hide\": false,\n \"refId\": \"C\"\n }\n ],\n \"title\": \"Task manager workload\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 27\n },\n \"id\": 10,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_pending_tasks_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_running_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_commit_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager__schedule_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"E\"\n }\n ],\n \"title\": \"Task manager timings\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Task Manager\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 4\n },\n \"id\": 31,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 18,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"smooth\",\n \"lineStyle\": {\n \"fill\": \"solid\"\n },\n \"lineWidth\": 1,\n \"pointSize\": 6,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 28\n },\n \"id\": 26,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_event_processing_avg_seconds\",\n \"legendFormat\": \"{{node}}-processing-avg-seconds\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Avg Job Event Processing Lag Time\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 31,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 1,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_2-job-event-processing-rate\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-redis-queue\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 28\n },\n \"id\": 24,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_events_queue_size_redis\",\n \"legendFormat\": \"{{node}}-redis-queue\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"irate(callback_receiver_events_insert_db[1m])*60\",\n \"hide\": ","isInternal":false,"tokens":10202,"sizeBytes":50000},{"name":"services_dashboard.json","path":"tools/grafana/dashboards/services_dashboard.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/grafana/dashboards/services_dashboard.json","title":"Dashboards Skill","category":"anthropic-skill","format":"json","content":"# Dashboards Skill\n\nPath: `tools/grafana/dashboards/services_dashboard.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/grafana/dashboards/services_dashboard.json)","tokens":300,"sizeBytes":207},{"name":"local-config.yaml","path":"tools/loki/local-config.yaml","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/loki/local-config.yaml","title":"Loki Skill","category":"anthropic-skill","format":"yaml","content":"auth_enabled: false\n\nserver:\n http_listen_port: 3100\n grpc_server_max_recv_msg_size: 524288000 # 500 MB\n grpc_server_max_send_msg_size: 524288000 # 500 MB, might be too much, be careful\n\nfrontend_worker:\n match_max_concurrent: true\n grpc_client_config:\n max_send_msg_size: 524288000 # 500 MB\n\n\ningester:\n max_chunk_age: 8766h\n\ncommon:\n path_prefix: /loki\n storage:\n filesystem:\n chunks_directory: /loki/chunks\n rules_directory: /loki/rules\n replication_factor: 1\n ring:\n kvstore:\n store: inmemory\n\n# compactor:\n# retention_enabled: true\n# # cmeyers: YOLO. 1s seems wrong but it works so right\n# compaction_interval: 1s # default 10m\n\nschema_config:\n configs:\n - from: 2020-10-24\n store: boltdb-shipper\n object_store: filesystem\n schema: v11\n index:\n prefix: index_\n period: 24h\n\nstorage_config:\n boltdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/boltdb-cache\n\nruler:\n alertmanager_url: http://localhost:9093\n\nlimits_config:\n retention_period: 3y\n # cmeyers: The default of 30m triggers a loop of queries that take a long time\n # to complete and the UI times out\n split_queries_by_interval: 1d\n # cmeyers: Default of 30d1h limits grafana time queries. Can't, for example,\n # query last 90 days\n max_query_length: 3y\n # cmeyers: Made the batch post request succeed.\n reject_old_samples: false\n reject_old_samples_max_age: 365d\n\n ingestion_rate_mb: 32\n ingestion_burst_size_mb: 32\n per_stream_rate_limit: 32M\n per_stream_rate_limit_burst: 32M\n ingestion_rate_strategy: local # Default: global\n max_global_streams_per_user: 100000000\n max_entries_limit_per_query: 100000000\n max_query_series: 1000000\n max_query_parallelism: 32 # Old Default: 14\n max_streams_per_user: 100000000 # Old Default: 10000\n\n# Taken from aap-log-visualizer\nfrontend:\n max_outstanding_per_tenant: 2048\n\nquery_scheduler:\n max_outstanding_requests_per_tenant: 2048\n\nquery_range:\n parallelise_shardable_queries: false\n split_queries_by_interval: 0\n\n# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration\n# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/\n#\n# Statistics help us better understand how Loki is used, and they show us performance\n# levels for most users. This helps us prioritize features and documentation.\n# For more information on what's sent, look at\n# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go\n# Refer to the buildReport method to see what goes into a report.\n#\n# If you would like to disable reporting, uncomment the following lines:\n#analytics:\n# reporting_enabled: false\n","isInternal":false,"tokens":757,"sizeBytes":2690},{"name":"otel-collector-config.yaml","path":"tools/otel/otel-collector-config.yaml","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/otel/otel-collector-config.yaml","title":"Otel Skill","category":"anthropic-skill","format":"yaml","content":"receivers:\n otlp:\n protocols:\n grpc:\n http:\n\nexporters:\n debug:\n verbosity: detailed\n\n file:\n path: /awx-logs/awx-logs.json.zstd\n rotation:\n max_days: 14\n localtime: false\n max_megabytes: 300\n max_backups: 200\n format: json\n compression: zstd\n\n loki:\n endpoint: http://loki:3100/loki/api/v1/push\n tls:\n insecure: true\n headers:\n \"X-Scope-OrgID\": \"1\"\n default_labels_enabled:\n exporter: true\n job: true\n instance: true\n level: true\n\nprocessors:\n batch:\n\nextensions:\n health_check:\n zpages:\n endpoint: \":55679\"\n\nservice:\n pipelines:\n logs:\n receivers: [otlp]\n processors: [batch]\n exporters: [file, loki]\n\n extensions:\n - health_check\n - zpages\n","isInternal":false,"tokens":227,"sizeBytes":769},{"name":"github_push.json","path":"tools/scripts/webhook_examples/github_push.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/github_push.json","title":"Webhook_examples Skill","category":"anthropic-skill","format":"json","content":"# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/github_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/github_push.json)","tokens":300,"sizeBytes":211},{"name":"gitlab_push.json","path":"tools/scripts/webhook_examples/gitlab_push.json","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/gitlab_push.json","title":"Webhook_examples Skill","category":"anthropic-skill","format":"json","content":"# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/gitlab_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/gitlab_push.json)","tokens":300,"sizeBytes":211},{"name":"TESTING.md","path":"tools/sosreport/TESTING.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/sosreport/TESTING.md","title":"Sosreport Skill","category":"anthropic-skill","format":"markdown","content":"Create a RHEL box and then do the following.\n```bash\nsudo mkdir -p /usr/share/sosreport/sos/plugins\nsudo yum install sos\ncp controller.py /usr/share/sosreport/sos/plugins\nsudo chmod 644 /usr/share/sosreport/sos/plugins/controller.py\nln -s /usr/share/sosreport/sos/plugins/controller.py `find `find /usr/lib -name sos` -name plugins`\nsosreport -l | grep controller\n```\n\nThe results should be:\n```bash\n# sosreport -l | grep controller\n controller Ansible Automation Platform controller information\n```\n\nTo run only the controller plugin run: `sosreport --only-plugins controller`\n\n","isInternal":false,"tokens":147,"sizeBytes":589},{"name":"README.md","path":"awx_collection/tools/README.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/awx_collection/tools/README.md","title":"tools Documentation","category":"plugin-manifest","format":"markdown","content":"## Collection tools\n\nTools used for building, maintaining, and testing the collection.\n\n### Template Galaxy\n\nThe `template_galaxy.yml` playbook ran as a pre-requisite for building the collection.\n\n```\nmake awx_collection_build\n```\n\n### Generate\n\nThis will template resource modules (like `group`, for groups in inventory) from a boilerplate template.\nIt is intended as a tool for writing new modules or enforcing consistency.\n\n### Integration Testing\n\nThese instructions assume you have ansible-core and the collection installed.\nTo install the collection in-place (to pick up any local changes to source)\nthe `make symlink_collection` will symlink the `awx_collection/` folder to\nthe appropriate place under `~/.ansible/collections`.\n\nThis is a shortcut for quick validation of tests that bypasses `ansible-test`.\nTo use this, you need the `~/.tower_cli.cfg` config file populated,\nwhich can be done via the deprecated `tower-cli login ` or manually\nwriting it, where the format looks like:\n\n```\n[general]\nhost = https://localhost:8043/\nverify_ssl = false\nusername = admin\npassword = password\n```\n\nTODO: adjust playbook to allow using environment variables as well.\n\nTo run some sample modules:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml\n```\n\nTo run just one module (the most common use case), use the `-e test=`.\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=host\n```\n\nIf you want to run _all_ the tests, then you need to pass in the whole list.\nThis will take significant time and is not ideal from an error-handling perspective,\nbut this is a way to do it:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=$(ls -1Am awx_collection/tests/integration/targets/ | tr -d '[:space:]')\n```\n\nDepending on the module, you may need special dependencies.\nFor instance, the rrule lookup plugins need `pytz`.\nThese will be satisfied if you install requirements in `awx_collection/requirements.txt`.\n","isInternal":false,"tokens":454,"sizeBytes":2027},{"name":"README.md","path":"tools/community-bugscrub/README.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/community-bugscrub/README.md","title":"community-bugscrub Documentation","category":"plugin-manifest","format":"markdown","content":"# Community BugScrub tooling\n\nSmall python script that automatically distributes PRs and Issues given a list of `people` and dumps the contents in a Spreadsheet.\n\nTo be used when distributing the work of reviewing community contributions.\n\n## Usage\n\nInstall requirements.\n\n```\npip install -r requirements.txt\n```\n\nGet the usage.\n\n```\npython generate-sheet.py -h\n```\n\n## Adding a github Personal Access Token\nThe scripts looks first for a github personal access token to use to avoid having the scripts calls rate limited, you can create one or use an existing one if you have. The script looks for the PAT under the environment var `GITHUB_ACCESS_TOKEN`. \n\n\n# For internal spreadsheet usage\nAWX engineers will need to import the data generated from the script into a spreadshet manager. Please make sure that you do not replace the existing sheets but make a new one or create a new sheet inside the existing spreadsheet upon import. ","isInternal":false,"tokens":191,"sizeBytes":934},{"name":"README.md","path":"tools/docker-compose/editable_dependencies/README.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/docker-compose/editable_dependencies/README.md","title":"editable_dependencies Documentation","category":"plugin-manifest","format":"markdown","content":"# Editable dependencies in AWX Docker Compose Development Environment\n\nThis folder contains the symlink to editable dependencies for AWX\n\nDuring the bootstrap of awx development environment we will try to crawl through the symlinks and mount (the source of the symlink) to `tools_awx_` containers and `init_awx` containers then install all the dependencies in editable mode\n\n## How to enable/disable editable dependnecies\n\n### Enable\n\nSet `EDITABLE_DEPENDENCIES=true` either as an Environment Variable before invoking `make docker-compose`\n\n```bash\nexport EDITABLE_DEPENDENCIES=true\n```\n\nor during invocation of `make docker-compose`\n\n```bash\n\nEDITABLE_DEPENDENCIES=true make docker-compose\n```\n\nThis will cause the `make docker-compose-source` to template out docker-compose file with editable dependencies.\n\n### Disable\n\nTo disable editable dependency simply `unset EDITABLE_DEPENDENCIES`\n\n## How to add editable dependencies\n\nAdding symlink to the directory that contains the source of the editable dependencies will cause the dependency to be mounted and installed in the docker-compose development environment.\n\nBoth relative path or absolute path will work.\n\n### Examples\n\nI have `awx` checked out at `~/projects/src/github.com/TheRealHaoLiu/awx`\nI have `django-ansible-base` checked out at `~/projects/src/github.com/TheRealHaoLiu/django-ansible-base`\n\nFrom root of AWX project `~/projects/src/github.com/TheRealHaoLiu/awx`\n\nI can either do\n\n```bash\nln -s ~/projects/src/github.com/TheRealHaoLiu/django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\nor\n\n```bash\nln -s ../django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\n## How to remove indivisual editable dependencies\n\nSimply removing the symlink from `tools/docker-compose/editable_dependencies` **will cause problem**!\n\nand the volume `tools_var_lib_awx` needs to be deleted as well with\n\n```bash\nmake docker-compose-down\ndocker volume rm tools_var_lib_awx\n```\n\n","isInternal":false,"tokens":429,"sizeBytes":1959},{"name":"README.md","path":"tools/docker-compose/README.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/docker-compose/README.md","title":"docker-compose Documentation","category":"plugin-manifest","format":"markdown","content":"# Docker Compose for Development\n\n## Getting started\n\n### Clone the repo\n\nIf you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). We generally recommend that you view the releases page:\n\nhttps://github.com/ansible/awx/releases/latest\n\n...and clone the latest stable tag, e.g.,\n\n`git clone -b x.y.z https://github.com/ansible/awx.git`\n\nPlease note that deploying from `HEAD` (or the latest commit) is **not** stable, and that if you want to do this, you should proceed at your own risk.\n\nFor more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).\n\nOnce you have a local copy, run the commands in the following sections from the root of the project tree.\n\n## Overview\n\nHere are the main `make` targets:\n\n- `docker-compose-build` - used for building the development image, which is used by the `docker-compose` target. You can skip this target if you want to use the latest [ghcr.io/ansible/awx_devel:devel](https://github.com/ansible/awx/pkgs/container/awx_devel) image rather than build a new one.\n- `docker-compose` - make target for development, passes awx_devel image and tag\n\nNotable files:\n\n- `tools/docker-compose/inventory` file - used to configure the AWX development environment.\n- `tools/docker-compose/ansible/migrate.yml` - playbook for migrating data from Local Docker to the Development Environment\n\n### Prerequisites\n\n- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the `docker` group, refer to the documentation for details)\n- [Docker Compose](https://docs.docker.com/compose/install/).\n- [Ansible](https://docs.ansible.com/projects/ansible/latest/installation_guide/intro_installation.html) will need to be installed as we use it to template files needed for the docker-compose.\n- OpenSSL.\n\n### Tested Operating Systems\n\nThe docker-compose development environment is regularly used and should work on x86_64 systems running:\n\n- Fedora (maintained versions)\n- Ubuntu LTS (18, 20)\n- Red Hat Enterprise Linux 8, CentOS Stream 8\n- macOS 11\n\nUse on other platforms is untested, and may require local changes.\n\n### How to Disable Color?\n\nThere are several layers that might apply color:\n - docker compose coloring based on what container log comes from\n - supervisord coloring based on what server it comes from\n - general coloration from Django management commands\n\nThese can have color turned off by adding things to the\nenvironment variable to modify the call options.\n\n```\nDJANGO_COLORS=nocolor COMPOSE_UP_OPTS=\"--no-color\" SUPERVISOR_ARGS=\"-n -t\" make docker-compose\n```\n\nThis can be useful if this is ran in CI in any context.\n\n## Starting the Development Environment\n\n### Build the Image\n\nThe AWX base container image (defined in the Dockerfile templated from [Dockerfile.j2](./../ansible/roles/dockerfile/templates/Dockerfile.j2)) contains basic OS dependencies and symbolic links into the development environment that make running the services easy.\n\nRun the following to build the image:\n\n```bash\n$ make docker-compose-build\n```\n\n> The image will need to be rebuilt if there are any changes to Dockerfile.j2 or any of the files used by the templated Dockerfile.\n\nOnce the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows:\n\n```bash\n(host)$ docker images\n\nREPOSITORY TAG IMAGE ID CREATED SIZE\nansible/awx_devel latest ba9ec3e8df74 26 minutes ago 1.42GB\n```\n\n> By default, this image will be tagged with your branch name. You can specify a custom tag by setting an environment variable, for example: `DEVEL_IMAGE_NAME=quay.io/your_user/awx_devel:17.0.1`\n\n#### Customizing the Receptor Image\n\nBy default, the development environment will use the `devel` image from receptor.\nThis is used directly in `docker-compose.yml` for the hop nodes.\nThe receptor binary is also copied over to the main awx_devel image, used in all other AWX nodes.\n\nBecause of this, the `RECEPTOR_IMAGE` environment variable must be set when running\nboth docker-compose-build and docker-compose in order to use the correct receptor in all containers.\n\nIf you need to create a new receptor image, you can check out receptor and build it like this:\n\n```bash\nCONTAINERCMD=docker TAG=quay.io/ansible/receptor:release_1.1 make container\n```\n\nThen that can be used by AWX like this:\n\n```bash\nexport RECEPTOR_IMAGE=quay.io/ansible/receptor:release_1.1\nmake docker-compose-build\nmake docker-compose\n```\n\n### Run AWX\n\n##### Start the containers\n\nRun the awx, postgres and redis containers. This utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the awx container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process.\n\n```bash\n$ make docker-compose\n```\n\n> The make target assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG= `in your environment. For example, you might be working in a feature branch, but you want to run the containers using the devel image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose`\n\n> For running docker-compose detached mode, start the containers using the following command: `$ make docker-compose COMPOSE_UP_OPTS=-d`\n\n###### Solving database-related issues during initial startup\n\nIf you have encountered the infinitely-repeating `Waiting for postgres to be ready to accept connections` message during the execution, try to do the following:\n\n1. Stop and delete AWX-related docker containers.\n2. Delete all associated docker volumes.\n3. Delete all associated docker networks.\n4. Repeat the process from scratch.\n\nIf you have **only** AWX-related container entities in your system, you can simply stop and delete everything using the following commands:\n\n```bash\ndocker stop $(docker ps -a -q)\ndocker system prune -a\ndocker volume prune\ndocker network prune\n```\n\n##### _(alternative method)_ Spin up a development environment with customized mesh node cluster\n\nA cluster (of containers) with execution nodes and a hop node can be created by the docker-compose Makefile target.\nBy default, it will create 1 hybrid node.\nYou can switch the type of AWX nodes between hybrid and control with `MAIN_NODE_TYPE`.\n\n```\nMAIN_NODE_TYPE=control EXECUTION_NODE_COUNT=2 COMPOSE_TAG=devel make docker-compose\n```\n\nRunning the above command will create a cluster of 1 control node, 1 hop node, and 2 execution nodes.\nA hop node is automatically created whenever there are 1 or more execution nodes.\n\nThe number of nodes can be changed:\n\n```\nCONTROL_PLANE_NODE_COUNT=2 EXECUTION_NODE_COUNT=3 COMPOSE_TAG=devel make docker-compose\n```\n\nThis will spin up a topology represented below.\n(names are the receptor node names, which differ from the AWX Instance names and network address in some cases)\n\n```\n ┌──────────────┐\n │ │\n┌──────────────┐ ┌──────────┤ receptor-1 │\n│ │ │ │ │\n│ awx_1 │◄──────────┐ │ └──────────────┘\n│ │ │ ▼\n└──────┬───────┘ ┌──────┴───────┐ ┌──────────────┐\n │ │ │ │ │\n │ │ receptor-hop │◄───────┤ receptor-2 │\n ▼ │ │ │ │\n┌──────────────┐ └──────────────┘ └──────────────┘\n│ │ ▲\n│ awx_2 │ │ ┌──────────────┐\n│ │ │ │ │\n└──────────────┘ └──────────┤ receptor-3 │\n │ │\n └──────────────┘\n```\n\nAll execution (`receptor-*`) nodes connect to the hop node.\nOnly the `awx_1` node connects to the hop node out of the AWX cluster.\n`awx_1` connects to `awx_2`, fulfilling the requirement that the AWX cluster is fully connected.\n\nFor example, if a job is launched with `awx_2` as the `controller_node` and `receptor-3` as the `execution_node`,\nthen `awx_2` communicates to `receptor-3` via `awx_1` and then `receptor-hop`.\n\n\n##### Wait for migrations to complete\n\nThe first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following:\n\n```bash\nawx_1 | Operations to perform:\nawx_1 | Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic\nawx_1 | Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, contenttypes, auth, conf, main\nawx_1 | Synchronizing apps without migrations:\nawx_1 | Creating tables...\nawx_1 | Running deferred SQL...\nawx_1 | Installing custom SQL...\nawx_1 | Running migrations:\nawx_1 | Rendering model states... DONE\nawx_1 | Applying contenttypes.0001_initial... OK\nawx_1 | Applying contenttypes.0002_remove_content_type_name... OK\nawx_1 | Applying auth.0001_initial... OK\n...\n```\n\n##### Clean and build the UI\n\nPrerequisites (on your local machine)\n- npm\n- nodejs\n\nRequired versions listed here https://github.com/ansible/ansible-ui/blob/main/README.md\n\nOn your local machine (not in awx container)\n\n```bash\nmake clean/ui ui\n```\n\nThis will clone the ansible-ui into the `awx/ui/src` directory and build the static files. Then when the containers come up, awx-manage collectstatic will copy those files into the proper place.\n\nYou can also use `UI_LOCAL` to build from a locally cloned ansible-ui repo.\n\nSee [the ui development documentation](https://github.com/ansible/ansible-ui/blob/main/CONTRIBUTING.md) for more information on using the frontend development, build, and test tooling.\n\nOnce migrations are completed and the UI is built, you can begin using AWX. The UI can be reached in your browser at `https://localhost:8043/`, and the API can be found at `https://localhost:8043/api/v2`.\n\n##### Create an admin user\n\nBefore you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command:\n\n```bash\n$ docker exec -ti tools_awx_1 awx-manage createsuperuser\n```\n\n> Remember the username and password, as you will use them to log into the web interface for the first time.\n\n##### Load demo data\n\nOptionally, you may also want to load some demo data. This will create a demo project, inventory, and job template.\n\n```bash\n$ docker exec tools_awx_1 awx-manage create_preload_data\n```\n\n> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate\n> this container, and thus the database, if the database schema changes in an upstream commit.\n\n## Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior), you can\nmigrate your data to the development environment via the migrate.yml playbook using the steps described [here](./docs/data_migration.md).\n\n## Upgrading the Development Environment\n\nUpgrading AWX involves checking out the new source code and re-running the make target. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables.\n\nAfter updating the inventory file with any custom values, run the make target from the root of your AWX clone.\n\n```bash\n$ make docker-compose\n```\n\n## Extras\n\n- [Start a shell](#start-a-shell)\n- [Start AWX from the container shell](#start-awx-from-the-container-shell)\n- [Using Logstash](./docs/logstash.md)\n- [Start a Cluster](#start-a-cluster)\n- [Start with Minikube](#start-with-minikube)\n- [Splunk Integration](#splunk-integration)\n\n### Start a Shell\n\nTo run `awx-manage` commands and modify things inside the container, you will want to start a shell session on the _awx_ container. In a new terminal session, use the `docker exec` command to start the shell session:\n\n```bash\n(host)$ docker exec -it tools_awx_1 bash\n```\n\nThis creates a session in the _awx_ containers, just as if you were using `ssh`, and allows you execute commands within the running container.\n\n### Start AWX from the container shell\n\nOften times you'll want to start the development environment without immediately starting all of the services in the _awx_ container, and instead be taken directly to a shell. You can do this with the following:\n\n```bash\n(host)$ make docker-compose-test\n```\n\nUsing `docker exec`, this will create a session in the running _awx_ container, and place you at a command prompt, where you can run shell commands inside the container.\n\nIf you want to start and use the development environment, you'll first need to bootstrap it by running the following command:\n\n```bash\n(container)# /usr/bin/bootstrap_development.sh\n```\n\nThe above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it\nwill drop you back to the shell.\n\nIn order to launch all developer services:\n\n```bash\n(container)# /usr/bin/launch_awx.sh\n```\n\n`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't\nneed to call `bootstrap_development.sh` first.\n\n### Start a Cluster\n\nCertain features or bugs are only applicable when running a cluster of AWX nodes. To bring up a 3 node cluster development environment simply run the below command.\n\n```bash\n(host)$ CONTROL_PLANE_NODE_COUNT=3 make docker-compose\n```\n\n`CONTROL_PLANE_NODE_COUNT` is configurable and defaults to 1, effectively a non-clustered AWX.\n\nNote that you may see multiple messages of the form `2021-03-04 20:11:47,666 WARNING [-] awx.main.wsrelay Connection from awx_2 to awx_5 failed: 'Cannot connect to host awx_5:8013 ssl:False [Name or service not known]'.`. This can happen when you bring up a cluster of many nodes, say 10, then you bring up a cluster of less nodes, say 3. In this example, there will be 7 `Instance` records in the database that represent AWX instances. The AWX development environment mimics the VM deployment (vs. kubernetes) and expects the missing nodes to be brought back to healthy by the admin. The warning message you are seeing is all of the AWX nodes trying to connect the websocket backplane. You can manually delete the `Instance` records from the database i.e. `Instance.objects.get(hostname='awx_9').delete()` to stop the warnings.\n\n### Start with Minikube\n\nTo bring up a 1 node AWX + minikube that is accessible from AWX run the following.\n\nStart minikube\n\n```bash\n(host)$minikube start --cpus=4 --memory=8g --addons=ingress\n```\n\nStart AWX\n\n```bash\n(host)$ make docker-compose-container-group\n```\n\nAlternatively, you can set the env var `MINIKUBE_CONTAINER_GROUP=true` to use the default dev env bring up. his way you can use other env flags like the cluster node count. Set `MINIKUBE_SETUP=true` to make the roles download, install and run minikube for you, but if you run into issues with this just start minikube yourself.\n\n```bash\n(host)$ MINIKUBE_CONTAINER_GROUP=true make docker-compose\n```\n\nIf you want to clean all things once your are done, you can do:\n\n```bash\n(host)$ make docker-compose-container-group-clean\n```\n\n### Splunk Integration\n\nSplunk is a log aggregation tool that can be used to test AWX with external logging integration. This section describes how to build a reference Splunk instance and plumb it with your AWX for testing purposes.\n\nFirst, be sure that you have the awx.awx collection installed by running `make install_collection`.\n\nNext, install the splunk.es collection by running `ansible-galaxy collection install splunk.es`.\n\nAnytime you want to run a Splunk instance alongside AWX we can start docker-compose with the SPLUNK option to get a Splunk instance with the command:\n```bash\nSPLUNK=true make docker-compose\n```\n\nOnce the containers come up three new ports (8000, 8089 and 9199) should be exposed and the Splunk server should be running on some of those ports (the 9199 will be created later by the plumbing playbook). The first port (8000) is the non-SSL admin port and you can log into splunk with the credentials admin/splunk_admin. The url will be like http://:8000/ this will be referenced below. The 8089 is the API port that the ansible modules will use to connect to and configure splunk. The 9199 port will be used to construct a TCP listener in Splunk that AWX will forward messages to.\n\nOnce the containers are up we are ready to configure and plumb Splunk with AWX. To do this we have provided a playbook which will:\n* Backup and configure the External Logging adapter in AWX. NOTE: this will back up your existing settings but the password fields can not be backed up through the API, you need a DB backup to recover this.\n* Create a TCP port in Splunk for log forwarding\n\nFor routing traffic between AWX and Splunk we will use the internal docker compose network. The `Logging Aggregator` will be configured using the internal network machine name of `splunk`.\n\nOnce you have have the collections installed (from above) you can run the playbook like:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_splunk.yml\n```\n\nOnce the playbook is done running Splunk should now be setup in your development environment. You can log into the admin console (see above for username/password) and click on \"Searching and Reporting\" in the left hand navigation. In the search box enter `source=\"http:tower_logging_collections\"` and click search.\n\n### HashiVault Integration\n\nRun a HashiVault container alongside of AWX.\n\n```bash\nVAULT=true make docker-compose\n```\n\nYou can find the initialization data at `tools/docker-compose/_sources/secrets/vault_init.yml`,\nThis includes the unseal keys and a root token.\n\nYou will need to unseal the HashiVault each time the container is started.\nThe easiest way to do that is to run:\n```bash\nansible-playbook tools/docker-compose/ansible/unseal_vault.yml\n```\nThis will perform the unseal and also display the root token for login.\n\nFor demo purposes, Vault will be auto-configured to include a Key Value (KV) vault called `my_engine` along with a secret called `my_key` in `/my_engine/my_root/my_folder`.\nThe secret value is `this_is_the_secret_value`.\n\nTo create a secret connected to this vault in AWX you can run the following playbook:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_vault.yml\n```\n\nThis will create the following items in your AWX instance:\n* A credential called `Vault Lookup Cred` tied to the vault instance.\n* A credential called `Vault UserPass Lookup Cred` tied to the vault instance.\n* A custom credential type called `Vault Custom Cred Type`.\n* A credential called `Credential From HashiCorp Vault via Token Auth` which is of the created type using the `Vault Lookup Cred` to get the secret.\n* A credential called `Credential From HashiCorp Vault via UserPass Auth` which is of the created type using the `Vault Userpass Lookup Cred` to get the secret.\n\nThe custom credential type adds a variable when used in a playbook called `the_secret_from_vault`.\nIf you have a playbook like:\n```\n---\n- name: Show a vault secret\n hosts: localhost\n connection: local\n gather_facts: False\n tasks:\n - debug:\n var: the_secret_from_vault\n```\n\nAnd run it through AWX with the credential `Credential From Vault via Token Auth` tied to it, the debug should result in `this_is_the_secret_value`. If you run it through AWX with the credential `Credential From Vault via Userpass Auth`, the debug should result in `this_is_the_userpass_secret_value`.\n\n### Prometheus and Grafana integration\n\nSee docs at https://github.com/ansible/awx/blob/devel/tools/grafana/README.md\n\n### OpenTelemetry Integration\n\n```bash\nOTEL=true GRAFANA=true LOKI=true PROMETHEUS=true make docker-compose\n```\n\nThis will start the sidecar container `tools_otel_1` and configure AWX logging to send to it. The OpenTelemetry Collector is configured to export logs to Loki. Grafana is configured with Loki as a datasource. AWX logs can be viewed in Grafana.\n\n`http://localhost:3001` grafana\n","isInternal":false,"tokens":4964,"sizeBytes":21868},{"name":"README.md","path":"tools/grafana/README.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/grafana/README.md","title":"grafana Documentation","category":"plugin-manifest","format":"markdown","content":"### Prometheus and Grafana\nPrometheus is a metrics collecting tool, and we support prometheus formatted data at the`api/v2/metrics` endpoint.\n\nPrometheus is configured to poll targets on some interval and can use various authentication methods.\nWe configure prometheus to use basic auth and poll AWX every 5 seconds by default. This configuration is created using the `docker-compose/ansible/roles/sources/templates/prometheus.yml.j2` template. To change the values, modify the template or set the ansible variables used in that template (see use of `EXTRA_SOURCES_ANSIBLE_OPTS` in the next section).\n\nGrafana provides dashboards to visualize this data as well as alerting features.\n\nWe use the \"provisioning files\" functionality in Grafana to be able to deploy alert and dashboard configs from file.\n\nThese config files are located in directories that are mounted into the container in `tools/grafana`.\n\n### How to run docker-compose with Grafana and Prometheus\n\nTo run the development environment (see [docs](https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md)) with Prometheus and Grafana enabled, set the following variables:\n\n```\n$ PROMETHEUS=yes GRAFANA=yes make docker-compose\n```\n\nIf we want to pass additional ansible vars used in the deployment process, we can pass `EXTRA_SOURCES_ANSIBLE_OPTS`:\n\n```\nGRAFANA=true PROMETHEUS=true EXTRA_SOURCES_ANSIBLE_OPTS=\"-e scrape_interval=1 admin_password=foobar\" make docker-compose\n```\n\n### Where to view grafana and prometheus\n\n1. navigate to Prometheus at `http://localhost:9090/targets` and check that the metrics endpoint State is Up.\n2. Click the Graph tab, start typing a metric name, or use the Open metrics explorer button to find a metric to display (next to `Execute` button)\n3. Navigate to Grafana at `http://localhost:3001`. Sign in, using `admin` for both username and password.\n4. In the left navigation menu go to Dashboards->Browse, find the \"awx-demo\" and click. These should have graphs.\n5. Now you can modify these and add panels for whichever metrics you like.\n\n### Alerts in Grafana\n\nWe are configuring alerts in grafana using the provisioning files method. This feature is new in Grafana as of August, 2022. Documentation can be found: https://grafana.com/docs/grafana/latest/administration/provisioning/#alerting however it does not fully show all parameters to the config.\n\nOne way to understand how to build rules is to build them in the UI and use chrometools to inspect the payload as you save the rules. It appears that the \"data\" portion of the payload for each rule is the same syntax as needed in the provisioning file config. To reload the alerts without restarting the container, from your terminal you can send a POST with `curl -X POST http://admin:admin@localhost:3001/api/admin/provisioning/alerting/reload`.\n\nAnother way to export rules is explore the api.\n1. Get all the folders: `GET` to `/api/folders`\n2. Get the rules `GET` to `/api/ruler/grafana/api/v1/rules/{{ Folder }}`\n\nYou can do this via curl or in the web browser.\n\n### Included Alerts\n\n#### Alert if remaining capacity low and pending jobs exist\n\nWe want to know if jobs are in pending but we lack capacity in the cluster to run them. Our approach is to sum all remaining capacity in the cluster and compare it to the total capacity of the cluster. If less than 10% of our capacity is remaining and we have pending jobs, and this is true for more than 180s, we will fire the alert.\n\nThis alert is named \"capacity_below_10_percent\" and can be found in this directory in https://github.com/ansible/awx/blob/devel/tools/grafana/alerting/alerts.yml\n","isInternal":false,"tokens":826,"sizeBytes":3619},{"name":"README.md","path":"tools/scripts/ig-hotfix/README.md","rawUrl":"https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/ig-hotfix/README.md","title":"ig-hotfix Documentation","category":"plugin-manifest","format":"markdown","content":"# ig-hotfix Documentation\n\nPath: `tools/scripts/ig-hotfix/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/ig-hotfix/README.md)","tokens":300,"sizeBytes":184}],"systemPromptSnippet":"\n\n# Azure_rm Skill\n\nPath: `awx/main/tests/data/inventory/plugins/azure_rm/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/azure_rm/env.json)\n\n\n# Controller Skill\n\nPath: `awx/main/tests/data/inventory/plugins/controller/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/controller/env.json)\n\n\n# Ec2 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/ec2/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/ec2/env.json)\n\n\n# Gce Skill\n\nPath: `awx/main/tests/data/inventory/plugins/gce/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/gce/env.json)\n\n\n# Insights Skill\n\nPath: `awx/main/tests/data/inventory/plugins/insights/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/insights/env.json)\n\n\n# Openshift_virtualization Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openshift_virtualization/env.json)\n\n\n# Openstack Skill\n\nPath: `awx/main/tests/data/inventory/plugins/openstack/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/openstack/env.json)\n\n\n# Rhv Skill\n\nPath: `awx/main/tests/data/inventory/plugins/rhv/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/rhv/env.json)\n\n\n# Satellite6 Skill\n\nPath: `awx/main/tests/data/inventory/plugins/satellite6/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/satellite6/env.json)\n\n\n# Terraform Skill\n\nPath: `awx/main/tests/data/inventory/plugins/terraform/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/terraform/env.json)\n\n\n# Vmware Skill\n\nPath: `awx/main/tests/data/inventory/plugins/vmware/env.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/awx/main/tests/data/inventory/plugins/vmware/env.json)\n\n\n# Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior) to AWX 18.0 or higher, you can\nmigrate your data to the development environment via the migrate.yml playbook.\n\n> Note: This will also convert your postgresql bind-mount into a docker volume.\n\n### Migrate data with migrate.yml\n\nFirst, in the [`inventory` file](../inventory), set your `pg_password`, `broadcast_websocket_secret`, `secret_key`, and any other settings you need for your deployment. **Make sure you use the same values from the `inventory` file in your existing pre-18.0.0 Docker deployment.**\n\nIf you used a custom pgdocker or awxcompose location, you will need to set the `postgres_data_dir` and `old_docker_compose_dir` variables.\n\n1. Run the [migrate playbook](../ansible/migrate.yml) to migrate your data to the new postgresql container and convert the data directory to a volume mount.\n```bash\n$ ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/migrate.yml -e \"migrate_local_docker=true\" -e \"postgres_data_dir=~/.awx/pgdocker\" -e \"old_docker_compose_dir=~/.awx/awxcompose\"\n```\n\n2. Change directory to the top of your awx checkout, and follow the [development environment installation instructions](../README.md) to start your containers:\n```bash\n$ make docker-compose-build\n$ make docker-compose\n$ make ui\n```\n\n3. After ensuring your data has been successfully migrated, you may delete your old data directory (typically stored at `~/.awx/pgdocker`).\n\n4. Note that after migration, the development environment will be available at https://localhost:8043/ (not http://localhost).\n\n\n# How to use the logstash container\n\n#### Modify the docker-compose.yml\n\nYou can run the logstash container by adding another compose file to the docker-compose target.\n\n```\nCOMPOSE_OPTS=\"-f tools/docker-compose/logstash-override.yaml\" COMPOSE_TAG=devel make docker-compose\n```\n\nPOST the following content to `/api/v2/settings/logging/` (this uses\nauthentication set up inside of the logstash configuration file).\n\n```\n{\n \"LOG_AGGREGATOR_HOST\": \"http://logstash\",\n \"LOG_AGGREGATOR_PORT\": 8085,\n \"LOG_AGGREGATOR_TYPE\": \"logstash\",\n \"LOG_AGGREGATOR_USERNAME\": \"awx_logger\",\n \"LOG_AGGREGATOR_PASSWORD\": \"workflows\",\n \"LOG_AGGREGATOR_LOGGERS\": [\n \"awx\",\n \"activity_stream\",\n \"job_events\",\n \"system_tracking\",\n \"job_lifecycle\"\n ],\n \"LOG_AGGREGATOR_INDIVIDUAL_FACTS\": false,\n \"LOG_AGGREGATOR_TOWER_UUID\": \"991ac7e9-6d68-48c8-bbde-7ca1096653c6\",\n \"LOG_AGGREGATOR_ENABLED\": true\n}\n```\n\n> Note: HTTP must be specified in the `LOG_AGGREGATOR_HOST` if you are using the docker development environment. \n\nAn example of how to view the most recent logs from the container:\n\n```\ndocker exec -i -t $(docker ps -aqf \"name=tools_logstash_1\") tail -n 50 /logstash.log\n```\n\n#### How to add logstash plugins\n\nAdd any plugins you need in `tools/elastic/logstash/Dockerfile` before running the container.\n\n\n---\nservices:\n # A useful container that simply passes through log messages to the console\n # helpful for testing awx/tower logging\n logstash:\n build:\n context: ../\n dockerfile: Dockerfile-logstash\n container_name: tools_logstash_1\n hostname: logstash\n networks:\n - awx\n\n\n{\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": {\n \"type\": \"grafana\",\n \"uid\": \"-- Grafana --\"\n },\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"fiscalYearStartMonth\": 0,\n \"graphTooltip\": 0,\n \"id\": 2,\n \"links\": [],\n \"liveNow\": false,\n \"panels\": [\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 38,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"Fraction of time dispatcher is listening for new messages\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 39,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_availability\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Availability\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 1\n },\n \"id\": 40,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_max_worker_count\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_active_task_count\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Dispatcher Workers\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 9\n },\n \"id\": 41,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dispatcher_pool_scale_up_events\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Dispatcher Pool Scale-Up Events\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Dispatcher\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 37,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 14,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_database_connections_total\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Database\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {},\n \"fieldConfig\": {\n \"defaults\": {\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"short\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 25,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"/^tower_version$/\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"000000021\"\n },\n \"editorMode\": \"code\",\n \"exemplar\": false,\n \"expr\": \"awx_system_info\",\n \"format\": \"table\",\n \"instant\": true,\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Version\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"displayName\": \"Instances\",\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"light-blue\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 4,\n \"w\": 5,\n \"x\": 12,\n \"y\": 30\n },\n \"id\": 13,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"center\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"9.5.2\",\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"count(awx_instance_info)\",\n \"interval\": \"\",\n \"legendFormat\": \" \",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Controller Node Count\",\n \"type\": \"stat\"\n }\n ],\n \"title\": \"System\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 2\n },\n \"id\": 35,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"running\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"canceled\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"super-light-yellow\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 26\n },\n \"id\": 8,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"awx_status_total\",\n \"legendFormat\": \"{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"job status\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-error\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-successful\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-purple\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_3-failed\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-orange\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 26\n },\n \"id\": 29,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_status_total\",\n \"legendFormat\": \"{{node}}-{{status}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Job Status per Instance\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"description\": \"\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 34\n },\n \"id\": 16,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_generate_dependencies_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"dependency_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"D\"\n }\n ],\n \"title\": \"Dependency Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 34\n },\n \"id\": 18,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"workflow_manager__schedule_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"hide\": false,\n \"refId\": \"B\"\n }\n ],\n \"title\": \"Workflow Manager Timings\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 6,\n \"w\": 12,\n \"x\": 0,\n \"y\": 42\n },\n \"id\": 27,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_remaining_capacity\",\n \"legendFormat\": \"remaining_capacity_{{hostname}}\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Remaining Instance Capacity\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 42\n },\n \"id\": 20,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"awx_instance_consumed_capacity\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Consumed Instance Capacity\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Jobs and Capacity\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 3\n },\n \"id\": 33,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"__systemRef\": \"hideSeriesFrom\",\n \"matcher\": {\n \"id\": \"byNames\",\n \"options\": {\n \"mode\": \"exclude\",\n \"names\": [\n \"task_manager_pending_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_running_processed{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\",\n \"task_manager_tasks_started{instance=\\\"awx1:8013\\\", job=\\\"awx\\\", node=\\\"awx_1\\\"}\"\n ],\n \"prefix\": \"All except:\",\n \"readOnly\": true\n }\n },\n \"properties\": [\n {\n \"id\": \"custom.hideFrom\",\n \"value\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": true\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 27\n },\n \"id\": 12,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_running_processed\",\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_pending_processed\",\n \"hide\": false,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_blocked\",\n \"hide\": false,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"expr\": \"task_manager_tasks_started\",\n \"hide\": false,\n \"refId\": \"C\"\n }\n ],\n \"title\": \"Task manager workload\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 5,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 27\n },\n \"id\": 10,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_pending_tasks_seconds\",\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_process_running_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_get_tasks_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"D\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager_commit_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"task_manager__schedule_seconds\",\n \"hide\": false,\n \"legendFormat\": \"__auto\",\n \"range\": true,\n \"refId\": \"E\"\n }\n ],\n \"title\": \"Task manager timings\",\n \"type\": \"timeseries\"\n }\n ],\n \"title\": \"Task Manager\",\n \"type\": \"row\"\n },\n {\n \"collapsed\": true,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 4\n },\n \"id\": 31,\n \"panels\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 18,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"smooth\",\n \"lineStyle\": {\n \"fill\": \"solid\"\n },\n \"lineWidth\": 1,\n \"pointSize\": 6,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 28\n },\n \"id\": 26,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_event_processing_avg_seconds\",\n \"legendFormat\": \"{{node}}-processing-avg-seconds\",\n \"range\": true,\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Avg Job Event Processing Lag Time\",\n \"type\": \"timeseries\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"custom\": {\n \"axisCenteredZero\": false,\n \"axisColorMode\": \"text\",\n \"axisLabel\": \"\",\n \"axisPlacement\": \"auto\",\n \"barAlignment\": 0,\n \"drawStyle\": \"line\",\n \"fillOpacity\": 31,\n \"gradientMode\": \"none\",\n \"hideFrom\": {\n \"legend\": false,\n \"tooltip\": false,\n \"viz\": false\n },\n \"lineInterpolation\": \"linear\",\n \"lineWidth\": 1,\n \"pointSize\": 1,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"showPoints\": \"auto\",\n \"spanNulls\": false,\n \"stacking\": {\n \"group\": \"A\",\n \"mode\": \"none\"\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n }\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n }\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_2-job-event-processing-rate\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"light-green\",\n \"mode\": \"fixed\"\n }\n }\n ]\n },\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"awx_1-redis-queue\"\n },\n \"properties\": [\n {\n \"id\": \"color\",\n \"value\": {\n \"fixedColor\": \"semi-dark-red\",\n \"mode\": \"fixed\"\n }\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 28\n },\n \"id\": 24,\n \"options\": {\n \"legend\": {\n \"calcs\": [],\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"showLegend\": true\n },\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"builder\",\n \"expr\": \"callback_receiver_events_queue_size_redis\",\n \"legendFormat\": \"{{node}}-redis-queue\",\n \"range\": true,\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"prometheus\",\n \"uid\": \"awx_prometheus\"\n },\n \"editorMode\": \"code\",\n \"expr\": \"irate(callback_receiver_events_insert_db[1m])*60\",\n \"hide\":\n\n\n# Dashboards Skill\n\nPath: `tools/grafana/dashboards/services_dashboard.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/grafana/dashboards/services_dashboard.json)\n\n\nauth_enabled: false\n\nserver:\n http_listen_port: 3100\n grpc_server_max_recv_msg_size: 524288000 # 500 MB\n grpc_server_max_send_msg_size: 524288000 # 500 MB, might be too much, be careful\n\nfrontend_worker:\n match_max_concurrent: true\n grpc_client_config:\n max_send_msg_size: 524288000 # 500 MB\n\n\ningester:\n max_chunk_age: 8766h\n\ncommon:\n path_prefix: /loki\n storage:\n filesystem:\n chunks_directory: /loki/chunks\n rules_directory: /loki/rules\n replication_factor: 1\n ring:\n kvstore:\n store: inmemory\n\n# compactor:\n# retention_enabled: true\n# # cmeyers: YOLO. 1s seems wrong but it works so right\n# compaction_interval: 1s # default 10m\n\nschema_config:\n configs:\n - from: 2020-10-24\n store: boltdb-shipper\n object_store: filesystem\n schema: v11\n index:\n prefix: index_\n period: 24h\n\nstorage_config:\n boltdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/boltdb-cache\n\nruler:\n alertmanager_url: http://localhost:9093\n\nlimits_config:\n retention_period: 3y\n # cmeyers: The default of 30m triggers a loop of queries that take a long time\n # to complete and the UI times out\n split_queries_by_interval: 1d\n # cmeyers: Default of 30d1h limits grafana time queries. Can't, for example,\n # query last 90 days\n max_query_length: 3y\n # cmeyers: Made the batch post request succeed.\n reject_old_samples: false\n reject_old_samples_max_age: 365d\n\n ingestion_rate_mb: 32\n ingestion_burst_size_mb: 32\n per_stream_rate_limit: 32M\n per_stream_rate_limit_burst: 32M\n ingestion_rate_strategy: local # Default: global\n max_global_streams_per_user: 100000000\n max_entries_limit_per_query: 100000000\n max_query_series: 1000000\n max_query_parallelism: 32 # Old Default: 14\n max_streams_per_user: 100000000 # Old Default: 10000\n\n# Taken from aap-log-visualizer\nfrontend:\n max_outstanding_per_tenant: 2048\n\nquery_scheduler:\n max_outstanding_requests_per_tenant: 2048\n\nquery_range:\n parallelise_shardable_queries: false\n split_queries_by_interval: 0\n\n# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration\n# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/\n#\n# Statistics help us better understand how Loki is used, and they show us performance\n# levels for most users. This helps us prioritize features and documentation.\n# For more information on what's sent, look at\n# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go\n# Refer to the buildReport method to see what goes into a report.\n#\n# If you would like to disable reporting, uncomment the following lines:\n#analytics:\n# reporting_enabled: false\n\n\nreceivers:\n otlp:\n protocols:\n grpc:\n http:\n\nexporters:\n debug:\n verbosity: detailed\n\n file:\n path: /awx-logs/awx-logs.json.zstd\n rotation:\n max_days: 14\n localtime: false\n max_megabytes: 300\n max_backups: 200\n format: json\n compression: zstd\n\n loki:\n endpoint: http://loki:3100/loki/api/v1/push\n tls:\n insecure: true\n headers:\n \"X-Scope-OrgID\": \"1\"\n default_labels_enabled:\n exporter: true\n job: true\n instance: true\n level: true\n\nprocessors:\n batch:\n\nextensions:\n health_check:\n zpages:\n endpoint: \":55679\"\n\nservice:\n pipelines:\n logs:\n receivers: [otlp]\n processors: [batch]\n exporters: [file, loki]\n\n extensions:\n - health_check\n - zpages\n\n\n# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/github_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/github_push.json)\n\n\n# Webhook_examples Skill\n\nPath: `tools/scripts/webhook_examples/gitlab_push.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/webhook_examples/gitlab_push.json)\n\n\nCreate a RHEL box and then do the following.\n```bash\nsudo mkdir -p /usr/share/sosreport/sos/plugins\nsudo yum install sos\ncp controller.py /usr/share/sosreport/sos/plugins\nsudo chmod 644 /usr/share/sosreport/sos/plugins/controller.py\nln -s /usr/share/sosreport/sos/plugins/controller.py `find `find /usr/lib -name sos` -name plugins`\nsosreport -l | grep controller\n```\n\nThe results should be:\n```bash\n# sosreport -l | grep controller\n controller Ansible Automation Platform controller information\n```\n\nTo run only the controller plugin run: `sosreport --only-plugins controller`\n\n\n## Collection tools\n\nTools used for building, maintaining, and testing the collection.\n\n### Template Galaxy\n\nThe `template_galaxy.yml` playbook ran as a pre-requisite for building the collection.\n\n```\nmake awx_collection_build\n```\n\n### Generate\n\nThis will template resource modules (like `group`, for groups in inventory) from a boilerplate template.\nIt is intended as a tool for writing new modules or enforcing consistency.\n\n### Integration Testing\n\nThese instructions assume you have ansible-core and the collection installed.\nTo install the collection in-place (to pick up any local changes to source)\nthe `make symlink_collection` will symlink the `awx_collection/` folder to\nthe appropriate place under `~/.ansible/collections`.\n\nThis is a shortcut for quick validation of tests that bypasses `ansible-test`.\nTo use this, you need the `~/.tower_cli.cfg` config file populated,\nwhich can be done via the deprecated `tower-cli login ` or manually\nwriting it, where the format looks like:\n\n```\n[general]\nhost = https://localhost:8043/\nverify_ssl = false\nusername = admin\npassword = password\n```\n\nTODO: adjust playbook to allow using environment variables as well.\n\nTo run some sample modules:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml\n```\n\nTo run just one module (the most common use case), use the `-e test=`.\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=host\n```\n\nIf you want to run _all_ the tests, then you need to pass in the whole list.\nThis will take significant time and is not ideal from an error-handling perspective,\nbut this is a way to do it:\n\n```\nansible-playbook -i localhost, awx_collection/tools/integration_testing.yml -e test=$(ls -1Am awx_collection/tests/integration/targets/ | tr -d '[:space:]')\n```\n\nDepending on the module, you may need special dependencies.\nFor instance, the rrule lookup plugins need `pytz`.\nThese will be satisfied if you install requirements in `awx_collection/requirements.txt`.\n\n\n# Community BugScrub tooling\n\nSmall python script that automatically distributes PRs and Issues given a list of `people` and dumps the contents in a Spreadsheet.\n\nTo be used when distributing the work of reviewing community contributions.\n\n## Usage\n\nInstall requirements.\n\n```\npip install -r requirements.txt\n```\n\nGet the usage.\n\n```\npython generate-sheet.py -h\n```\n\n## Adding a github Personal Access Token\nThe scripts looks first for a github personal access token to use to avoid having the scripts calls rate limited, you can create one or use an existing one if you have. The script looks for the PAT under the environment var `GITHUB_ACCESS_TOKEN`. \n\n\n# For internal spreadsheet usage\nAWX engineers will need to import the data generated from the script into a spreadshet manager. Please make sure that you do not replace the existing sheets but make a new one or create a new sheet inside the existing spreadsheet upon import.\n\n\n# Editable dependencies in AWX Docker Compose Development Environment\n\nThis folder contains the symlink to editable dependencies for AWX\n\nDuring the bootstrap of awx development environment we will try to crawl through the symlinks and mount (the source of the symlink) to `tools_awx_` containers and `init_awx` containers then install all the dependencies in editable mode\n\n## How to enable/disable editable dependnecies\n\n### Enable\n\nSet `EDITABLE_DEPENDENCIES=true` either as an Environment Variable before invoking `make docker-compose`\n\n```bash\nexport EDITABLE_DEPENDENCIES=true\n```\n\nor during invocation of `make docker-compose`\n\n```bash\n\nEDITABLE_DEPENDENCIES=true make docker-compose\n```\n\nThis will cause the `make docker-compose-source` to template out docker-compose file with editable dependencies.\n\n### Disable\n\nTo disable editable dependency simply `unset EDITABLE_DEPENDENCIES`\n\n## How to add editable dependencies\n\nAdding symlink to the directory that contains the source of the editable dependencies will cause the dependency to be mounted and installed in the docker-compose development environment.\n\nBoth relative path or absolute path will work.\n\n### Examples\n\nI have `awx` checked out at `~/projects/src/github.com/TheRealHaoLiu/awx`\nI have `django-ansible-base` checked out at `~/projects/src/github.com/TheRealHaoLiu/django-ansible-base`\n\nFrom root of AWX project `~/projects/src/github.com/TheRealHaoLiu/awx`\n\nI can either do\n\n```bash\nln -s ~/projects/src/github.com/TheRealHaoLiu/django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\nor\n\n```bash\nln -s ../django-ansible-base tools/docker-compose/editable_dependencies/\n```\n\n## How to remove indivisual editable dependencies\n\nSimply removing the symlink from `tools/docker-compose/editable_dependencies` **will cause problem**!\n\nand the volume `tools_var_lib_awx` needs to be deleted as well with\n\n```bash\nmake docker-compose-down\ndocker volume rm tools_var_lib_awx\n```\n\n\n# Docker Compose for Development\n\n## Getting started\n\n### Clone the repo\n\nIf you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). We generally recommend that you view the releases page:\n\nhttps://github.com/ansible/awx/releases/latest\n\n...and clone the latest stable tag, e.g.,\n\n`git clone -b x.y.z https://github.com/ansible/awx.git`\n\nPlease note that deploying from `HEAD` (or the latest commit) is **not** stable, and that if you want to do this, you should proceed at your own risk.\n\nFor more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).\n\nOnce you have a local copy, run the commands in the following sections from the root of the project tree.\n\n## Overview\n\nHere are the main `make` targets:\n\n- `docker-compose-build` - used for building the development image, which is used by the `docker-compose` target. You can skip this target if you want to use the latest [ghcr.io/ansible/awx_devel:devel](https://github.com/ansible/awx/pkgs/container/awx_devel) image rather than build a new one.\n- `docker-compose` - make target for development, passes awx_devel image and tag\n\nNotable files:\n\n- `tools/docker-compose/inventory` file - used to configure the AWX development environment.\n- `tools/docker-compose/ansible/migrate.yml` - playbook for migrating data from Local Docker to the Development Environment\n\n### Prerequisites\n\n- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the `docker` group, refer to the documentation for details)\n- [Docker Compose](https://docs.docker.com/compose/install/).\n- [Ansible](https://docs.ansible.com/projects/ansible/latest/installation_guide/intro_installation.html) will need to be installed as we use it to template files needed for the docker-compose.\n- OpenSSL.\n\n### Tested Operating Systems\n\nThe docker-compose development environment is regularly used and should work on x86_64 systems running:\n\n- Fedora (maintained versions)\n- Ubuntu LTS (18, 20)\n- Red Hat Enterprise Linux 8, CentOS Stream 8\n- macOS 11\n\nUse on other platforms is untested, and may require local changes.\n\n### How to Disable Color?\n\nThere are several layers that might apply color:\n - docker compose coloring based on what container log comes from\n - supervisord coloring based on what server it comes from\n - general coloration from Django management commands\n\nThese can have color turned off by adding things to the\nenvironment variable to modify the call options.\n\n```\nDJANGO_COLORS=nocolor COMPOSE_UP_OPTS=\"--no-color\" SUPERVISOR_ARGS=\"-n -t\" make docker-compose\n```\n\nThis can be useful if this is ran in CI in any context.\n\n## Starting the Development Environment\n\n### Build the Image\n\nThe AWX base container image (defined in the Dockerfile templated from [Dockerfile.j2](./../ansible/roles/dockerfile/templates/Dockerfile.j2)) contains basic OS dependencies and symbolic links into the development environment that make running the services easy.\n\nRun the following to build the image:\n\n```bash\n$ make docker-compose-build\n```\n\n> The image will need to be rebuilt if there are any changes to Dockerfile.j2 or any of the files used by the templated Dockerfile.\n\nOnce the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows:\n\n```bash\n(host)$ docker images\n\nREPOSITORY TAG IMAGE ID CREATED SIZE\nansible/awx_devel latest ba9ec3e8df74 26 minutes ago 1.42GB\n```\n\n> By default, this image will be tagged with your branch name. You can specify a custom tag by setting an environment variable, for example: `DEVEL_IMAGE_NAME=quay.io/your_user/awx_devel:17.0.1`\n\n#### Customizing the Receptor Image\n\nBy default, the development environment will use the `devel` image from receptor.\nThis is used directly in `docker-compose.yml` for the hop nodes.\nThe receptor binary is also copied over to the main awx_devel image, used in all other AWX nodes.\n\nBecause of this, the `RECEPTOR_IMAGE` environment variable must be set when running\nboth docker-compose-build and docker-compose in order to use the correct receptor in all containers.\n\nIf you need to create a new receptor image, you can check out receptor and build it like this:\n\n```bash\nCONTAINERCMD=docker TAG=quay.io/ansible/receptor:release_1.1 make container\n```\n\nThen that can be used by AWX like this:\n\n```bash\nexport RECEPTOR_IMAGE=quay.io/ansible/receptor:release_1.1\nmake docker-compose-build\nmake docker-compose\n```\n\n### Run AWX\n\n##### Start the containers\n\nRun the awx, postgres and redis containers. This utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the awx container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process.\n\n```bash\n$ make docker-compose\n```\n\n> The make target assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG= `in your environment. For example, you might be working in a feature branch, but you want to run the containers using the devel image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose`\n\n> For running docker-compose detached mode, start the containers using the following command: `$ make docker-compose COMPOSE_UP_OPTS=-d`\n\n###### Solving database-related issues during initial startup\n\nIf you have encountered the infinitely-repeating `Waiting for postgres to be ready to accept connections` message during the execution, try to do the following:\n\n1. Stop and delete AWX-related docker containers.\n2. Delete all associated docker volumes.\n3. Delete all associated docker networks.\n4. Repeat the process from scratch.\n\nIf you have **only** AWX-related container entities in your system, you can simply stop and delete everything using the following commands:\n\n```bash\ndocker stop $(docker ps -a -q)\ndocker system prune -a\ndocker volume prune\ndocker network prune\n```\n\n##### _(alternative method)_ Spin up a development environment with customized mesh node cluster\n\nA cluster (of containers) with execution nodes and a hop node can be created by the docker-compose Makefile target.\nBy default, it will create 1 hybrid node.\nYou can switch the type of AWX nodes between hybrid and control with `MAIN_NODE_TYPE`.\n\n```\nMAIN_NODE_TYPE=control EXECUTION_NODE_COUNT=2 COMPOSE_TAG=devel make docker-compose\n```\n\nRunning the above command will create a cluster of 1 control node, 1 hop node, and 2 execution nodes.\nA hop node is automatically created whenever there are 1 or more execution nodes.\n\nThe number of nodes can be changed:\n\n```\nCONTROL_PLANE_NODE_COUNT=2 EXECUTION_NODE_COUNT=3 COMPOSE_TAG=devel make docker-compose\n```\n\nThis will spin up a topology represented below.\n(names are the receptor node names, which differ from the AWX Instance names and network address in some cases)\n\n```\n ┌──────────────┐\n │ │\n┌──────────────┐ ┌──────────┤ receptor-1 │\n│ │ │ │ │\n│ awx_1 │◄──────────┐ │ └──────────────┘\n│ │ │ ▼\n└──────┬───────┘ ┌──────┴───────┐ ┌──────────────┐\n │ │ │ │ │\n │ │ receptor-hop │◄───────┤ receptor-2 │\n ▼ │ │ │ │\n┌──────────────┐ └──────────────┘ └──────────────┘\n│ │ ▲\n│ awx_2 │ │ ┌──────────────┐\n│ │ │ │ │\n└──────────────┘ └──────────┤ receptor-3 │\n │ │\n └──────────────┘\n```\n\nAll execution (`receptor-*`) nodes connect to the hop node.\nOnly the `awx_1` node connects to the hop node out of the AWX cluster.\n`awx_1` connects to `awx_2`, fulfilling the requirement that the AWX cluster is fully connected.\n\nFor example, if a job is launched with `awx_2` as the `controller_node` and `receptor-3` as the `execution_node`,\nthen `awx_2` communicates to `receptor-3` via `awx_1` and then `receptor-hop`.\n\n\n##### Wait for migrations to complete\n\nThe first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following:\n\n```bash\nawx_1 | Operations to perform:\nawx_1 | Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic\nawx_1 | Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, contenttypes, auth, conf, main\nawx_1 | Synchronizing apps without migrations:\nawx_1 | Creating tables...\nawx_1 | Running deferred SQL...\nawx_1 | Installing custom SQL...\nawx_1 | Running migrations:\nawx_1 | Rendering model states... DONE\nawx_1 | Applying contenttypes.0001_initial... OK\nawx_1 | Applying contenttypes.0002_remove_content_type_name... OK\nawx_1 | Applying auth.0001_initial... OK\n...\n```\n\n##### Clean and build the UI\n\nPrerequisites (on your local machine)\n- npm\n- nodejs\n\nRequired versions listed here https://github.com/ansible/ansible-ui/blob/main/README.md\n\nOn your local machine (not in awx container)\n\n```bash\nmake clean/ui ui\n```\n\nThis will clone the ansible-ui into the `awx/ui/src` directory and build the static files. Then when the containers come up, awx-manage collectstatic will copy those files into the proper place.\n\nYou can also use `UI_LOCAL` to build from a locally cloned ansible-ui repo.\n\nSee [the ui development documentation](https://github.com/ansible/ansible-ui/blob/main/CONTRIBUTING.md) for more information on using the frontend development, build, and test tooling.\n\nOnce migrations are completed and the UI is built, you can begin using AWX. The UI can be reached in your browser at `https://localhost:8043/`, and the API can be found at `https://localhost:8043/api/v2`.\n\n##### Create an admin user\n\nBefore you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command:\n\n```bash\n$ docker exec -ti tools_awx_1 awx-manage createsuperuser\n```\n\n> Remember the username and password, as you will use them to log into the web interface for the first time.\n\n##### Load demo data\n\nOptionally, you may also want to load some demo data. This will create a demo project, inventory, and job template.\n\n```bash\n$ docker exec tools_awx_1 awx-manage create_preload_data\n```\n\n> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate\n> this container, and thus the database, if the database schema changes in an upstream commit.\n\n## Migrating Data from Local Docker\n\nIf you are migrating data from a Local Docker installation (17.0.1 and prior), you can\nmigrate your data to the development environment via the migrate.yml playbook using the steps described [here](./docs/data_migration.md).\n\n## Upgrading the Development Environment\n\nUpgrading AWX involves checking out the new source code and re-running the make target. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables.\n\nAfter updating the inventory file with any custom values, run the make target from the root of your AWX clone.\n\n```bash\n$ make docker-compose\n```\n\n## Extras\n\n- [Start a shell](#start-a-shell)\n- [Start AWX from the container shell](#start-awx-from-the-container-shell)\n- [Using Logstash](./docs/logstash.md)\n- [Start a Cluster](#start-a-cluster)\n- [Start with Minikube](#start-with-minikube)\n- [Splunk Integration](#splunk-integration)\n\n### Start a Shell\n\nTo run `awx-manage` commands and modify things inside the container, you will want to start a shell session on the _awx_ container. In a new terminal session, use the `docker exec` command to start the shell session:\n\n```bash\n(host)$ docker exec -it tools_awx_1 bash\n```\n\nThis creates a session in the _awx_ containers, just as if you were using `ssh`, and allows you execute commands within the running container.\n\n### Start AWX from the container shell\n\nOften times you'll want to start the development environment without immediately starting all of the services in the _awx_ container, and instead be taken directly to a shell. You can do this with the following:\n\n```bash\n(host)$ make docker-compose-test\n```\n\nUsing `docker exec`, this will create a session in the running _awx_ container, and place you at a command prompt, where you can run shell commands inside the container.\n\nIf you want to start and use the development environment, you'll first need to bootstrap it by running the following command:\n\n```bash\n(container)# /usr/bin/bootstrap_development.sh\n```\n\nThe above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it\nwill drop you back to the shell.\n\nIn order to launch all developer services:\n\n```bash\n(container)# /usr/bin/launch_awx.sh\n```\n\n`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't\nneed to call `bootstrap_development.sh` first.\n\n### Start a Cluster\n\nCertain features or bugs are only applicable when running a cluster of AWX nodes. To bring up a 3 node cluster development environment simply run the below command.\n\n```bash\n(host)$ CONTROL_PLANE_NODE_COUNT=3 make docker-compose\n```\n\n`CONTROL_PLANE_NODE_COUNT` is configurable and defaults to 1, effectively a non-clustered AWX.\n\nNote that you may see multiple messages of the form `2021-03-04 20:11:47,666 WARNING [-] awx.main.wsrelay Connection from awx_2 to awx_5 failed: 'Cannot connect to host awx_5:8013 ssl:False [Name or service not known]'.`. This can happen when you bring up a cluster of many nodes, say 10, then you bring up a cluster of less nodes, say 3. In this example, there will be 7 `Instance` records in the database that represent AWX instances. The AWX development environment mimics the VM deployment (vs. kubernetes) and expects the missing nodes to be brought back to healthy by the admin. The warning message you are seeing is all of the AWX nodes trying to connect the websocket backplane. You can manually delete the `Instance` records from the database i.e. `Instance.objects.get(hostname='awx_9').delete()` to stop the warnings.\n\n### Start with Minikube\n\nTo bring up a 1 node AWX + minikube that is accessible from AWX run the following.\n\nStart minikube\n\n```bash\n(host)$minikube start --cpus=4 --memory=8g --addons=ingress\n```\n\nStart AWX\n\n```bash\n(host)$ make docker-compose-container-group\n```\n\nAlternatively, you can set the env var `MINIKUBE_CONTAINER_GROUP=true` to use the default dev env bring up. his way you can use other env flags like the cluster node count. Set `MINIKUBE_SETUP=true` to make the roles download, install and run minikube for you, but if you run into issues with this just start minikube yourself.\n\n```bash\n(host)$ MINIKUBE_CONTAINER_GROUP=true make docker-compose\n```\n\nIf you want to clean all things once your are done, you can do:\n\n```bash\n(host)$ make docker-compose-container-group-clean\n```\n\n### Splunk Integration\n\nSplunk is a log aggregation tool that can be used to test AWX with external logging integration. This section describes how to build a reference Splunk instance and plumb it with your AWX for testing purposes.\n\nFirst, be sure that you have the awx.awx collection installed by running `make install_collection`.\n\nNext, install the splunk.es collection by running `ansible-galaxy collection install splunk.es`.\n\nAnytime you want to run a Splunk instance alongside AWX we can start docker-compose with the SPLUNK option to get a Splunk instance with the command:\n```bash\nSPLUNK=true make docker-compose\n```\n\nOnce the containers come up three new ports (8000, 8089 and 9199) should be exposed and the Splunk server should be running on some of those ports (the 9199 will be created later by the plumbing playbook). The first port (8000) is the non-SSL admin port and you can log into splunk with the credentials admin/splunk_admin. The url will be like http://:8000/ this will be referenced below. The 8089 is the API port that the ansible modules will use to connect to and configure splunk. The 9199 port will be used to construct a TCP listener in Splunk that AWX will forward messages to.\n\nOnce the containers are up we are ready to configure and plumb Splunk with AWX. To do this we have provided a playbook which will:\n* Backup and configure the External Logging adapter in AWX. NOTE: this will back up your existing settings but the password fields can not be backed up through the API, you need a DB backup to recover this.\n* Create a TCP port in Splunk for log forwarding\n\nFor routing traffic between AWX and Splunk we will use the internal docker compose network. The `Logging Aggregator` will be configured using the internal network machine name of `splunk`.\n\nOnce you have have the collections installed (from above) you can run the playbook like:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_splunk.yml\n```\n\nOnce the playbook is done running Splunk should now be setup in your development environment. You can log into the admin console (see above for username/password) and click on \"Searching and Reporting\" in the left hand navigation. In the search box enter `source=\"http:tower_logging_collections\"` and click search.\n\n### HashiVault Integration\n\nRun a HashiVault container alongside of AWX.\n\n```bash\nVAULT=true make docker-compose\n```\n\nYou can find the initialization data at `tools/docker-compose/_sources/secrets/vault_init.yml`,\nThis includes the unseal keys and a root token.\n\nYou will need to unseal the HashiVault each time the container is started.\nThe easiest way to do that is to run:\n```bash\nansible-playbook tools/docker-compose/ansible/unseal_vault.yml\n```\nThis will perform the unseal and also display the root token for login.\n\nFor demo purposes, Vault will be auto-configured to include a Key Value (KV) vault called `my_engine` along with a secret called `my_key` in `/my_engine/my_root/my_folder`.\nThe secret value is `this_is_the_secret_value`.\n\nTo create a secret connected to this vault in AWX you can run the following playbook:\n```bash\nexport CONTROLLER_USERNAME=\nexport CONTROLLER_PASSWORD=\nansible-playbook tools/docker-compose/ansible/plumb_vault.yml\n```\n\nThis will create the following items in your AWX instance:\n* A credential called `Vault Lookup Cred` tied to the vault instance.\n* A credential called `Vault UserPass Lookup Cred` tied to the vault instance.\n* A custom credential type called `Vault Custom Cred Type`.\n* A credential called `Credential From HashiCorp Vault via Token Auth` which is of the created type using the `Vault Lookup Cred` to get the secret.\n* A credential called `Credential From HashiCorp Vault via UserPass Auth` which is of the created type using the `Vault Userpass Lookup Cred` to get the secret.\n\nThe custom credential type adds a variable when used in a playbook called `the_secret_from_vault`.\nIf you have a playbook like:\n```\n---\n- name: Show a vault secret\n hosts: localhost\n connection: local\n gather_facts: False\n tasks:\n - debug:\n var: the_secret_from_vault\n```\n\nAnd run it through AWX with the credential `Credential From Vault via Token Auth` tied to it, the debug should result in `this_is_the_secret_value`. If you run it through AWX with the credential `Credential From Vault via Userpass Auth`, the debug should result in `this_is_the_userpass_secret_value`.\n\n### Prometheus and Grafana integration\n\nSee docs at https://github.com/ansible/awx/blob/devel/tools/grafana/README.md\n\n### OpenTelemetry Integration\n\n```bash\nOTEL=true GRAFANA=true LOKI=true PROMETHEUS=true make docker-compose\n```\n\nThis will start the sidecar container `tools_otel_1` and configure AWX logging to send to it. The OpenTelemetry Collector is configured to export logs to Loki. Grafana is configured with Loki as a datasource. AWX logs can be viewed in Grafana.\n\n`http://localhost:3001` grafana\n\n\n### Prometheus and Grafana\nPrometheus is a metrics collecting tool, and we support prometheus formatted data at the`api/v2/metrics` endpoint.\n\nPrometheus is configured to poll targets on some interval and can use various authentication methods.\nWe configure prometheus to use basic auth and poll AWX every 5 seconds by default. This configuration is created using the `docker-compose/ansible/roles/sources/templates/prometheus.yml.j2` template. To change the values, modify the template or set the ansible variables used in that template (see use of `EXTRA_SOURCES_ANSIBLE_OPTS` in the next section).\n\nGrafana provides dashboards to visualize this data as well as alerting features.\n\nWe use the \"provisioning files\" functionality in Grafana to be able to deploy alert and dashboard configs from file.\n\nThese config files are located in directories that are mounted into the container in `tools/grafana`.\n\n### How to run docker-compose with Grafana and Prometheus\n\nTo run the development environment (see [docs](https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md)) with Prometheus and Grafana enabled, set the following variables:\n\n```\n$ PROMETHEUS=yes GRAFANA=yes make docker-compose\n```\n\nIf we want to pass additional ansible vars used in the deployment process, we can pass `EXTRA_SOURCES_ANSIBLE_OPTS`:\n\n```\nGRAFANA=true PROMETHEUS=true EXTRA_SOURCES_ANSIBLE_OPTS=\"-e scrape_interval=1 admin_password=foobar\" make docker-compose\n```\n\n### Where to view grafana and prometheus\n\n1. navigate to Prometheus at `http://localhost:9090/targets` and check that the metrics endpoint State is Up.\n2. Click the Graph tab, start typing a metric name, or use the Open metrics explorer button to find a metric to display (next to `Execute` button)\n3. Navigate to Grafana at `http://localhost:3001`. Sign in, using `admin` for both username and password.\n4. In the left navigation menu go to Dashboards->Browse, find the \"awx-demo\" and click. These should have graphs.\n5. Now you can modify these and add panels for whichever metrics you like.\n\n### Alerts in Grafana\n\nWe are configuring alerts in grafana using the provisioning files method. This feature is new in Grafana as of August, 2022. Documentation can be found: https://grafana.com/docs/grafana/latest/administration/provisioning/#alerting however it does not fully show all parameters to the config.\n\nOne way to understand how to build rules is to build them in the UI and use chrometools to inspect the payload as you save the rules. It appears that the \"data\" portion of the payload for each rule is the same syntax as needed in the provisioning file config. To reload the alerts without restarting the container, from your terminal you can send a POST with `curl -X POST http://admin:admin@localhost:3001/api/admin/provisioning/alerting/reload`.\n\nAnother way to export rules is explore the api.\n1. Get all the folders: `GET` to `/api/folders`\n2. Get the rules `GET` to `/api/ruler/grafana/api/v1/rules/{{ Folder }}`\n\nYou can do this via curl or in the web browser.\n\n### Included Alerts\n\n#### Alert if remaining capacity low and pending jobs exist\n\nWe want to know if jobs are in pending but we lack capacity in the cluster to run them. Our approach is to sum all remaining capacity in the cluster and compare it to the total capacity of the cluster. If less than 10% of our capacity is remaining and we have pending jobs, and this is true for more than 180s, we will fire the alert.\n\nThis alert is named \"capacity_below_10_percent\" and can be found in this directory in https://github.com/ansible/awx/blob/devel/tools/grafana/alerting/alerts.yml\n\n\n# ig-hotfix Documentation\n\nPath: `tools/scripts/ig-hotfix/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/ansible/awx/HEAD/tools/scripts/ig-hotfix/README.md)\n"}