## File: README.md [](https://app.fossa.io/projects/git%2Bgithub.com%2FTykTechnologies%2Ftyk?ref=badge_shield) [](https://github.com/TykTechnologies/tyk/releases) [](https://github.com/TykTechnologies/tyk/releases) [](https://hub.docker.com/r/tykio/tyk-gateway/) [](https://github.com/TykTechnologies/tyk/actions/workflows/ci-tests.yml) [](https://goreportcard.com/report/github.com/TykTechnologies/tyk) [](https://github.com/TykTechnologies/tyk/stargazers) [](https://github.com/TykTechnologies/tyk/fork) --- [Documentation](https://tyk.io/docs/) | [Forum](https://community.tyk.io) | [Blog](https://tyk.io/blog/) | [About](https://tyk.io) # Tyk API Gateway **Tyk Gateway** is the cloud-native, open source, enterprise-ready API Gateway supporting REST, GraphQL, TCP and gRPC protocols. Built from the ground up, as the [fastest API Gateway](https://tyk.io/performance-benchmarks/) on the planet since 2014. _Tyk Gateway_ is provided ‘Batteries-included’, with no feature lockout. Enabling your organization to rate limit, auth, gather analytics, apply microservice patterns [and more](#open-source-api-gateway-features) with ease. Tyk runs natively on _Kubernetes_, if you prefer, thanks to the _[Tyk Kubernetes Operator](https://github.com/TykTechnologies/tyk-operator)_ | [ ](https://tyk.io/docs/deployment-and-operations/tyk-open-source-api-gateway/quick-start) Everything you need to manage APIs. Follow the simple Get Started guide below 👇 | [ ](https://tyk.io/docs/tyk-self-managed/install) The Enterprise API Management platform: Management Control Plane, Dashboard GUI & Developer Portal. [Install Tyk Self Managed](https://tyk.io/api-lifecycle-management/) | [ ](https://tyk.io/docs/tyk-cloud) The Enterprise API Management platform SaaS: Management Control Plane, Dashboard GUI & Developer Portal. [Deploy Tyk Cloud ](https://tyk.io/docs/deployment-and-operations/tyk-cloud-platform/quick-start) | | --- | --- | --- | --- ## Get Started We’ll install Tyk, add auth, analytics, quotas and rate limiting to your API in under 5 minutes. We recommend [Tyk Gateway Docker](https://github.com/TykTechnologies/tyk-gateway-docker#start-up-the-deployment) as the quickest way to get started now. Later, you can move to one of our other [supported distributions](https://tyk.io/docs/apim/open-source/installation/) if you prefer. #### Step 1 - Clone the docker-compose repository ```console git clone https://github.com/TykTechnologies/tyk-gateway-docker ``` #### Step 2 - Change to the new directory ```console cd tyk-gateway-docker ``` #### Step 3 - Deploy Tyk Gateway and Redis ```console docker-compose up ``` You can run this in detach mode using the `-d` flag: `docker-compose up -d` **Congratulations, you’re done!** Your Tyk Gateway is now configured and ready to use. Confirm this by checking against the ‘hello’ endpoint: ```console curl localhost:8080/hello ``` Output: ```json {"status": "pass", "version": "v3.2.1", "description": "Tyk GW"} ``` Next, visit [adding your first API](https://tyk.io/docs/getting-started/create-api/) to Tyk and follow the Open Source instructions. --- Other Installations are available: 1. [Docker](https://tyk.io/docs/tyk-oss/ce-docker/) 2. [Kubernetes-Native ](https://github.com/TykTechnologies/tyk-oss-k8s-deployment) 3. [Kubernetes-Helm](https://github.com/TykTechnologies/tyk-helm-chart#install-tyk-community-edition) 4. [Ansible](https://tyk.io/docs/tyk-oss/ce-ansible/) 5. [Red Hat](https://tyk.io/docs/tyk-oss/ce-redhat/) 6. [Ubuntu](https://tyk.io/docs/tyk-oss/ce-ubuntu/) 7. [CentOS](https://tyk.io/docs/tyk-oss/ce-centos/) 8. [Compile Tyk from Source](#compiling-tyk-gateway) ## Open Source API Gateway Features Use any protocol: REST, SOAP, [GraphQL](https://tyk.io/docs/tyk-apis/tyk-gateway-api/api-definition-objects/graphql/), [gRPC](https://tyk.io/docs/key-concepts/grpc-proxy/), and [TCP](https://tyk.io/docs/key-concepts/tcp-proxy/). Industry Standard Authentication: [OIDC](https://tyk.io/docs/advanced-configuration/integrate/api-auth-mode/open-id-connect/#setting-up-oidc), [JWT,](https://tyk.io/docs/tyk-apis/tyk-gateway-api/api-definition-objects/jwt/) [bearer Tokens](https://tyk.io/docs/basic-config-and-security/security/authentication-authorization/bearer-tokens/), [Basic Auth](https://tyk.io/docs/tyk-apis/tyk-dashboard-api/basic-authentication/), Client Certificates and more. [Open API Standards:](https://tyk.io/docs/getting-started/using-oas-definitions/import-an-oas-api/) Import your Swagger and OpenAPI Documents (OAS 2.X and OAS 3.0.1) to scaffold APIs in Tyk. [Ultra performant](https://tyk.io/performance-tuning-your-tyk-api-gateway/): Low latency, and thousands of rps with just a single CPU, horizontally and vertically scalable. [Content mediation](https://tyk.io/docs/advanced-configuration/transform-traffic/): Transform all the things, from request or response headers to converting between SOAP and GraphQL. [Extensible Plugin Architecture](https://tyk.io/docs/plugins/): Customize Tyk’s middleware chain by writing plugins in your language of choice - from Python to Javascript to Go, or any language which supports gRPC. [Rate Limiting](https://tyk.io/docs/basic-config-and-security/control-limit-traffic/rate-limiting/#setting-rate-limits-in-the-tyk-community-edition-gateway-ce) & Quotas: Protect your upstreams from becoming overloaded and/or apply limits for each consumer. [API Versioning](https://tyk.io/docs/tyk-apis/tyk-gateway-api/api-definition-objects/versioning-endpoint/) - API Versions can be easily set and sunset (deprecated) at a specific time and date. [Granular Access Control](https://tyk.io/docs/security/security-policies/secure-apis-method-path/) - Grant access to one or more APIs on a per version and operation basis. [Blocklist](https://tyk.io/docs/advanced-configuration/transform-traffic/endpoint-designer/#blocklist)/[Allowlist](https://tyk.io/docs/advanced-configuration/transform-traffic/endpoint-designer/#allowlist)/[Ignore](https://tyk.io/docs/advanced-configuration/transform-traffic/endpoint-designer/#ignore) endpoint access - Enforce strict security models on a version-by-version basis to your access points. Analytics logging - Record detailed usage data on who is using your APIs (raw data only) [CORS](https://tyk.io/docs/tyk-apis/tyk-gateway-api/api-definition-objects/cors/) - Enable CORS for certain APIs so users can make browser-based requests [Webhooks](https://tyk.io/docs/basic-config-and-security/report-monitor-trigger-events/webhooks/) - Trigger webhooks against events such as Quota Violations and Authentication failures [IP AllowListing](https://tyk.io/docs/tyk-apis/tyk-gateway-api/api-definition-objects/ip-whitelisting/) - Block access to non-trusted IP addresses for more secure interactions [Hitless reloads](https://tyk.io/docs/tyk-configuration-reference/hot-restart-tyk-gateway-process/) - Tyk configurations can be altered dynamically and the service restarted without affecting any active request [Kubernetes native](https://tyk.io/docs/tyk-oss/ce-helm-chart/) declarative API: using Open Source [Tyk Operator](https://github.com/TykTechnologies/tyk-operator) (more info in OSS section) Tyk Technologies uses the same API Gateway for all it’s applications. Protecting, securing, and processing APIs for thousands of organizations and businesses around the world. Ideal for Open Banking, building software in the clouds as well as exposing APIs to teams, partners & consumers. ## Tyk OSS Integrations Tyk Technologies maintains other Open Source Software which can be used in conjunction with Tyk API Gateway: [Tyk Pump](https://github.com/TykTechnologies/tyk-pump) - Pluggable analytics purger to move Analytics generated by your Tyk nodes to any back-end. [Tyk Operator](https://github.com/TykTechnologies/tyk-operator) - Brings API Management capabilities to Kubernetes. Configure Ingress, APIs, Security Policies, Authentication, Authorization, Mediation and more - all using Custom Resources and Kubernetes Native primitives [Tyk Identity Broker](https://github.com/TykTechnologies/tyk-identity-broker) - Tyk Authentication Proxy for third-party login [Tyk Sync ](https://tyk.io/docs/tyk-sync/)- Command line tool and library to manage and synchronise a Tyk installation with your version control system (VCS). [Tyk Mserv](https://github.com/TykTechnologies/mserv) - Asset Server and gRPC host ## Documentation All the documentation for Tyk Gateway and other OSS-related topics can be found at [https://tyk.io/docs/tyk-oss-gateway/](https://tyk.io/docs/tyk-oss-gateway/) ## Community * [Tyk Community Board](https://community.tyk.io/) - Technical support from the Tyk Community * [Write a GitHub Issue](https://github.com/TykTechnologies/tyk/issues/new/choose) - Feature requests & bug reports welcome * [Technical blog](https://tyk.io/blog/) - Tyk announcements and updates * [Newsletters ](https://pages.tyk.io/newsletter)- Subscribe to our GraphQL & API newsletters * If you are using Tyk give us a star ⭐️ ## Licensing Tyk is dual-licensed: 1. Open Source License: The code in the root directory and all subdirectories except the 'ee' folder is released under the MPL v2.0. Please see [LICENSE](https://github.com/TykTechnologies/tyk/blob/master/LICENSE) for the full version of the open source license. 2. Commercial License: The code in the 'ee' folder is subject to a commercial license. For more information about obtaining a commercial license, please contact our sales team at sales@tyk.io. ## Compiling Tyk Gateway Compile from Source ```console git clone https://github.com/TykTechnologies/tyk go build ``` Go version 1.22 is required to build `master`, the current development version. Tyk is officially supported on `Linux/amd64`, `Linux/i386` and `Linux/arm64`. To run tests locally use the following command: ```console go test ./... ``` Note that tests require Redis to be running on the same machine (default port). To write your own test please use this guide [https://github.com/TykTechnologies/tyk/blob/master/TESTING.md](https://github.com/TykTechnologies/tyk/blob/master/TESTING.md) ## Contributing For more information about contributing PRs and issues, see [CONTRIBUTING.md](https://github.com/TykTechnologies/tyk/blob/master/CONTRIBUTING.md). — --- ## File: .vale/styles/write-good/README.md Based on [write-good](https://github.com/btford/write-good). > Naive linter for English prose for developers who can't write good and wanna learn to do other stuff good too. ``` The MIT License (MIT) Copyright (c) 2014 Brian Ford Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` --- ## File: apidef/oas/README.md # OAS Package This package provides support for OpenAPI Specification (OAS) schema validation and integration with Tyk's custom extensions. ## Overview The OAS package handles: - Loading and validating OAS schemas (3.0, 3.1, and future versions) - Injecting Tyk-specific extensions (`x-tyk-api-gateway`) into OAS schemas - Validating OAS documents and templates against schemas - Managing schema versions and defaults ## How to Add a New OAS Version When a new OAS version is released (e.g., OAS 4.0), follow these steps to add support: ### Step 1: Add the OAS Schema File 1. Download or create the official JSON Schema for the new OAS version 2. Save it to `apidef/oas/schema/` directory with the naming convention: `{major}.{minor}.json` - Example: `3.1.json` for OAS 3.1.x - Example: `4.0.json` for OAS 4.0.x ```bash # Example curl -o apidef/oas/schema/4.0.json https://spec.openapis.org/oas/4.0/schema/... ``` ### Step 2: Understand Schema Structure Differences Different OAS/JSON Schema versions may use different keys for definitions: - **OAS 3.0** uses `"definitions"` (JSON Schema Draft 04) - **OAS 3.1+** uses `"$defs"` (JSON Schema 2020-12) - **Future versions** may introduce new keys Check the schema file to identify which key is used: ```bash # Check for definitions key grep -E '"definitions"|"\$defs"' apidef/oas/schema/4.0.json | head -5 ``` ### Step 3: Update Code (if needed) The code is designed to automatically detect and handle different definition keys using the `GetDefinitionsKey()` function. **If the new version uses an existing key (`definitions` or `$defs`):** - ✅ No code changes needed! The schema will load automatically. **If the new version uses a NEW key (e.g., `$schemas`):** - Update the `GetDefinitionsKey()` function in `validator.go` to detect the new key: ```go func GetDefinitionsKey(schemaData []byte) string { // Try newest format first if _, _, _, err := jsonparser.Get(schemaData, "new-key"); err == nil { return "new-key" } // Try OAS 3.1+ format if _, _, _, err := jsonparser.Get(schemaData, keyDefs); err == nil { return keyDefs } // Fall back to OAS 3.0 format return keyDefinitions } ``` ### Step 4: Add Tests Add comprehensive tests to ensure the new schema loads correctly: #### 4.1 Update `Test_loadOASSchema` The test will automatically verify the new version if the schema file exists. It: - Checks that the schema loads - Verifies x-tyk extensions are injected correctly - Validates the correct definitions key is used #### 4.2 Add Version-Specific Validation Test Create a new test function to validate documents with the new version: ```go func TestValidateOASObject_4_0(t *testing.T) { t.Parallel() // Create minimal valid OAS 4.0 document validOAS40Doc := []byte(`{ "openapi": "4.0.0", "info": { "title": "Test API 4.0", "version": "1.0.0" }, "paths": { "/test": { "get": { "responses": { "200": { "description": "Success" } } } } }, "x-tyk-api-gateway": { "info": { "name": "test-api-4.0", "state": { "active": true } }, "upstream": { "url": "http://localhost:8080" }, "server": { "listenPath": { "value": "/test-api-4.0/" } } } }`) t.Run("valid OAS 4.0 document with version 4.0.0", func(t *testing.T) { t.Parallel() err := ValidateOASObject(validOAS40Doc, "4.0.0") assert.NoError(t, err) }) t.Run("valid OAS 4.0 document with version 4.0", func(t *testing.T) { t.Parallel() err := ValidateOASObject(validOAS40Doc, "4.0") assert.NoError(t, err) }) } ``` #### 4.3 Add Template Validation Test ```go func TestValidateOASTemplate_4_0(t *testing.T) { t.Parallel() // Minimal OAS 4.0 template (missing required x-tyk fields) template40 := []byte(`{ "openapi": "4.0.0", "info": { "title": "Template API 4.0", "version": "1.0.0" }, "paths": {}, "x-tyk-api-gateway": {} }`) t.Run("valid OAS 4.0 template", func(t *testing.T) { t.Parallel() err := ValidateOASTemplate(template40, "4.0") assert.NoError(t, err) }) } ``` #### 4.4 Add Schema Retrieval Test Update `TestGetOASSchema` with new test cases: ```go t.Run("return 4.0 schema when version 4.0 is requested", func(t *testing.T) { schema, err := GetOASSchema("4.0") assert.NoError(t, err) assert.NotEmpty(t, schema) // Verify it's the 4.0 schema by checking the definitions key defsKey := GetDefinitionsKey(schema) assert.Equal(t, "expected-key", defsKey, "OAS 4.0 should use 'expected-key'") }) t.Run("return 4.0 schema when version 4.0.0 is requested", func(t *testing.T) { schema, err := GetOASSchema("4.0.0") assert.NoError(t, err) assert.NotEmpty(t, schema) defsKey := GetDefinitionsKey(schema) assert.Equal(t, "expected-key", defsKey) }) ``` ### Step 5: Run Tests Run all tests to ensure the new version works correctly: ```bash # Run all OAS package tests cd apidef/oas go test -v # Run specific tests go test -v -run "Test_loadOASSchema|TestValidateOASObject_4_0|TestGetOASSchema" ``` ### Step 6: Update Default Version (Optional) The default OAS version is controlled in the `setDefaultVersion()` function in `validator.go`. **To keep the current default version:** - ✅ No changes needed. The code will automatically prefer the older version for stability. **To make the new version the default:** - Update or remove the override logic in `setDefaultVersion()`: ```go func setDefaultVersion() { var versions []string for k := range oasJSONSchemas { versions = append(versions, k) } latestVersion := findDefaultVersion(versions) // Remove or update this override when ready to use newer version if latestVersion == "4.0" { defaultVersion = "3.0" // Keep 3.0 as default for now } else { defaultVersion = latestVersion } } ``` ### Step 7: Document Breaking Changes If the new OAS version has breaking changes or requires updates to existing code: 1. Document the changes in the main project CHANGELOG 2. Add migration notes for users upgrading to the new version 3. Update examples and documentation to use the new version (if applicable) ## Example: Adding OAS 3.1 Support Here's what was done to add OAS 3.1 support: 1. ✅ Added `schema/3.1.json` file 2. ✅ Identified that OAS 3.1 uses `$defs` instead of `definitions` 3. ✅ Updated `GetDefinitionsKey()` to detect `$defs` 4. ✅ Updated `loadOASSchema()` to use detected key 5. ✅ Updated `ValidateOASTemplate()` to use detected key 6. ✅ Added comprehensive tests: - `TestGetDefinitionsKey` - Tests the helper function - `TestValidateOASObject_3_1` - Tests validation of 3.1 documents - `TestValidateOASTemplate_3_1` - Tests template validation - Updated `Test_loadOASSchema` - Verifies both 3.0 and 3.1 - Updated `TestGetOASSchema` - Added 3.1 retrieval tests 7. ✅ Kept default version at 3.0 for stability 8. ✅ All tests pass ✅ ## Key Architecture Decisions ### Automatic Schema Detection The `GetDefinitionsKey()` function automatically detects which key a schema uses, making the code robust across versions. ### Public API `GetDefinitionsKey()` is a public function, allowing other products to leverage this detection logic. ### Version Management - Schemas are stored by minor version (e.g., `3.0`, `3.1`) - Patch versions (e.g., `3.1.2`) map to their minor version - Default version can be overridden for stability during transitions ### Schema Injection X-Tyk extensions are injected from `schema/x-tyk-api-gateway.json` into each OAS schema during load time, ensuring all schemas have consistent Tyk-specific validation. ## Troubleshooting ### Schema Not Loading **Problem:** New schema file is not being loaded. **Solutions:** - Ensure filename follows the pattern `{major}.{minor}.json` - Verify the file is valid JSON - Check that the file is in the `schema/` directory - Ensure the file is included in the embed directive at the top of `validator.go` ### X-Tyk Extensions Not Working **Problem:** X-Tyk extensions are not being validated. **Solutions:** - Check that `GetDefinitionsKey()` correctly detects the schema's definitions key - Verify that `x-tyk-api-gateway.json` uses the correct definitions key - Add debug logging to `loadOASSchema()` to see which key is being used ### Tests Failing **Problem:** Existing tests fail after adding new schema. **Solutions:** - Check if `Test_setDefaultVersion` needs updating (it may expect a specific default) - Verify that the new schema doesn't conflict with existing schemas - Run tests in verbose mode to see detailed error messages: `go test -v` ## References - [OpenAPI Specification](https://spec.openapis.org/) - [JSON Schema Specification](https://json-schema.org/) - [Tyk OAS Documentation](https://tyk.io/docs/getting-started/using-oas-definitions/) ## Contributing When adding a new OAS version: 1. Follow this guide 2. Ensure all tests pass 3. Update this README if you discover any missing steps 4. Submit a PR with clear description of changes --- ## File: ci/images/hybrid/README.md ## Usage Please refer to the official tyk.io docs for information about how to run this Multi-Cloud docker image: [Tyk Docs - Multi Cloud Usage](https://tyk.io/docs/get-started/with-tyk-multi-cloud/tutorials/install-multicloud-gateway/) ## What's different about this Image and the Gateway Image? The main differences between the Docker Hybrid image and the Docker Gateway Image are the following: ### Includes Redis Server This Hybrid image runs with an integrated Redis Server. You can see the dependency for the service installed in the Dockerfile, and then you can see the service being started in the entrypoint.sh file. This is not optional. Should you choose to run your own Redis configuration, you will need to run the Gateway Image instead, and tweak the `tyk.conf` to run in multi-cloud. ### Includes Nginx Server This Hybrid image runs with an included Nginx server. In order to disable it, just add an environment variable `-e DISABLENGINX=1` If you are running multi-cloud through the start.sh script, you will need to modify it there. ### Tyk.conf Of course, the `tyk.conf` included in this repo will already be setup to connect as a slave node to the MDCB( Multi Data Center Bridge) instance. You shouldn't need to make any changes to this file. --- ## File: ci/images/README.md # plugin-compiler ~~Does not support go modules. If your plugin has vendored modules that are [also used by tyk gateway](https://github.com/TykTechnologies/tyk/tree/master/vendor) then your module will be overridden by the version that Tyk uses.~~ Since 3.2, tyk has started using go.mod and thus your vendor'd code will no longer be overridden by tyk's versions. ## Using the image Assuming that you are in the plugin source directory and that you want to build a plugin for v3.0.4 of the the gateway, ``` shell % docker run --rm -v `pwd`:/plugin-source tykio/tyk-plugin-compiler:v3.0.4 testplugin.so ``` You will find a `testplugin.so` in the current directory which is the file that goes into the API definition ## Testing the image ```shell % export tag=v2.9.5 % rm -v testplugin/*.so % docker run --rm -v `pwd`/testplugin:/plugin-source tykio/tyk-plugin-compiler:${tag} testplugin.so % docker-compose -f test.yml up .... ``` Look for `msg="API Loaded" api_id= api_name="Goplugin test"` in the output. Test that the plugin is working correctly by, ```shell % curl http://localhost:8080/goplugin/headers { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip", "Foo": "Bar", "Host": "httpbin.org", "User-Agent": "curl/7.68.0", "X-Amzn-Trace-Id": "Root=1-606f4317-18581ac0164b5496739a5b32" } } ``` The `Foo: Bar` header indicates all is well. Can be tested with `jq` like: ``` shell % curl http://localhost:8080/goplugin/headers | jq '.headers.Foo == "Bar"' true ``` ## Building the image This will build the image that will be used in the plugin build step. This section is for only for informational purposes. In the root of the repo: ``` shell docker build --build-arg TYK_GW_TAG=v2.8.4 -t tykio/tyk-plugin-compiler:v2.8.4 -f images/plugin-compiler/Dockerfile . ``` `TYK_GW_TAG` can be any github ref. --- ## File: coprocess/grpc/README.md # Coprocess (gRPC) This feature makes it possible to write Tyk middleware using a [gRPC](http://www.grpc.io/) backend. gRPC is a very interesting framework that has official support for many languages: C++, Java, Python, Go, Ruby, C#, Node.JS, Java, Objective C & PHP. The Tyk Coprocess feature uses Protocol Buffers for dispatching messages (mostly requests and events), this makes it easier to connect Tyk to a gRPC server (which is based on Protocol Buffers too), it works by specifying a [gRPC service definition](../proto/coprocess_object.proto) that covers the dispatcher logic: ``` service Dispatcher { rpc Dispatch (Object) returns (Object) {} rpc DispatchEvent (Event) returns (EventReply) {} } ``` ## gRPC backend A very simple use case is as follows: you write a gRPC server in a language of your choice, using the Tyk's Protocol Buffer definitions. When Tyk starts it performs a connection to your gRPC server (this is specified as a global setting in `tyk.conf`, you could use a UNIX socket -local- or even a TCP connection -over the network-). When Tyk receives a request, it performs a call to your gRPC server, which is responsible for doing the actual tasks (middleware tasks like transformations or even authentication) ## Global settings This is a section of `tyk.conf` that will: * Enable the Coprocess feature. * Indicate your gRPC server address. ```json "coprocess_options": { "enable_coprocess": true, "coprocess_grpc_server": "tcp://127.0.0.1:5555" }, "enable_bundle_downloader": true, "bundle_base_url": "http://my-bundle-server.com/bundles/", "public_key_path": "/path/to/my/pubkey", ``` * `enable_coprocess`: Enables the rich plugins feature. * `coprocess_grpc_server`: Sets the gRPC server host address. This is only required for gRPC plugins. * `enable_bundle_downloader`: Enables the bundle downloader. * `bundle_base_url`: A base URL that will be used to download the bundle, in this example we have "test-bundle" specified in the API settings, Tyk will fetch the following URL: "http://my-bundle-server.com/bundles/test-bundle". * `public_key_path`: Sets a public key, this is used for verifying signed bundles, you may omit this if unsigned bundles are used. ## API settings This is a sample configuration that will authenticate your API through a Coprocess (gRPC in this case, see `driver`), and a hook a "pre" middleware. ```json "enable_coprocess_auth": true, "custom_middleware": { "pre": [ { "name": "MyPreMiddleware", "require_session": false } ], "auth_check": { "name": "MyAuthCheck" }, "driver": "grpc" }, ``` ## Examples (Ruby) You may find a Ruby sample [here](ruby/sample_server.rb). --- ## File: coprocess/lua/README.md # Coprocess (Lua) This feature makes it possible to write Tyk middleware using [Lua](https://www.lua.org/). The requirement is to have [lua-cjson](http://www.kyne.com.au/~mark/software/lua-cjson.php) installed, an easier way of getting it is by using `luarocks`: ``` % luarocks install lua-cjson ``` --- ## File: coprocess/python/README.md # Coprocess (Python) This feature makes it possible to write Tyk middleware using [Python](https://www.python.org/), the current binding supports Python 3.x. The purpose of this README is to provide an overview of the architecture and a few implementation notes. ## Usage You'll need to build Tyk with specific build tags, see build notes below. Basically `go build -tags 'coprocess python'`. ### Setting up custom Python middleware The custom middleware should be specified in your API definition file, under `custom_middleware` (see [coprocess_app_sample.json](../../apps/coprocess_app_sample.json)): ```json "custom_middleware": { "pre": [ { "name": "MyPreMiddleware", "require_session": false } ], "post": [ { "name": "MyPostMiddleware", "require_session": false } ], "driver": "python" } ``` You can chain multiple hook functions when the hook type is Pre, Post or PostAuthCheck. Tyk will load all the modules inside `middleware/python`. The "name" field represents the name of a Python function, a sample Python middleware matching the sample definition above will look like (see [middleware/python](../../middleware/python)): ```python from tyk.decorators import * @Pre def MyPreMiddleware(request, session, spec): print("my_middleware: MyPreMiddleware") return request, session @Post def MyPreMiddleware(request, session, spec): print("my_middleware: MyPreMiddleware") return request, session ``` ### Authenticating an API with Python See example: https://tyk.io/docs/plugins/supported-languages/rich-plugins/python/custom-auth-python-tutorial/ ### Writing events handlers with Python It's also possible to write a Tyk event listener with Python. The first step is to set a custom event handler inside your API definition (see [coprocess_app_sample_protected.json](../../apps/coprocess_app_sample_protected.json)): ```json ... "event_handlers": { "events": { "AuthFailure": [ { "handler_name": "cp_dynamic_handler", "handler_meta": { "name": "my_handler" } } ] } }, ... ``` In the above sample we're setting an event handler for `AuthFailure` events, an event that's triggered everytime a failed authentication occurs. The `handler_name` must be `cp_dynamic_handler`. The `name` field inside `handler_meta` refers to a Python function name that can be written inside `event_handlers` (see [event_handlers/my_handler.py](../../event_handlers/my_handler.py)): ```python from tyk.decorators import Event @Event def my_handler(event, spec): print("-- my_handler:") print(" Event:", event) print(" Spec:", spec) ``` This function will be called when the specified event occurs, Tyk will pass a Python object like this: ```json { "TimeStamp": "2016-08-19 11:13:31.537047694 -0400 PYT", "Meta":{ "Path":"/coprocess-auth-tyk-api-test/", "Origin":"127.0.0.1", "Message":"Auth Failure", "OriginatingRequest":"R0VUIC9jb3Byb2Nlc3MtYXV0aC10eWstYXBpLXRlc3QvIEhUVFAvMS4xDQpIb3N0OiAxMjcuMC4wLjE6ODA4MA0KVXNlci1BZ2VudDogY3VybC83LjQzLjANCkFjY2VwdDogKi8qDQpBdXRob3JpemF0aW9uOiAxDQoNCg==", "Key":"" }, "Type": "AuthFailure" } ``` The above handler can be tested by sending a HTTP request to the protected Coprocess API, with an invalid authorization header: ``` curl http://127.0.0.1:8080/coprocess-auth-tyk-api-test/ -H 'Authorization: invalidtoken' ``` ## Build requirements * [Python 3.x](https://www.python.org/) * [Go](https://golang.org) * [Cython](http://cython.org/) (required if you need to modify and re-compile the gateway API binding) * [protobuf](https://pypi.python.org/pypi/protobuf/3.20.2) (Python module): `pip3 install protobuf==3.20.2` * [grpc](https://www.grpc.io/) (gRPC module): `pip3 install grpcio` ## Build steps To build Tyk with the Coprocess + Python support, use: ``` go build -tags 'coprocess python' ``` To compile the gateway API binding (assuming you're on the repository root): ```sh cd coprocess/python ./cythonize gateway ``` This will "cythonize" `gateway.pyx`, generating `gateway.c` and `gateway.h`. To compile some other binding (where `mybinding.pyx` is your Cython input file): ```sh cd coprocess/python ./cythonize mybinding ``` [cythonize](cythonize) is a helper script for compiling Python source files with Cython and patching the resulting source with the specific build tags used by this Coprocess feature. This is important in order to keep Tyk build-able when a standard build is needed (and make the Go compiler ignore C binding files based on the specified build tags). The top of a standard Cython binding file will look like this: ``` /* Generated by Cython 0.24.1 */ /* BEGIN: Cython Metadata { "distutils": { "depends": [] }, "module_name": "gateway" } END: Cython Metadata */ ``` After running `cythonize` the binding will have the correct build tags, and will be ignored if you don't build Tyk with these (`go build -tags 'coprocess python'`): ``` // +build coprocess // +build python /* Generated by Cython 0.24.1 */ /* BEGIN: Cython Metadata { "distutils": { "depends": [] }, "module_name": "gateway" } END: Cython Metadata */ ``` After re-compiling a binding, the C source code will change and you may want to build Tyk again. ## CPython [Python](https://www.python.org/) has a very popular and well-documented [C API](https://docs.python.org/3/c-api/index.html), this coprocess feature makes a heavy use of it. ## Built-in modules All the standard Python modules are available and it's also possible to load additional ones, if you add them to your local Python installation (for example, using pip). ### Coprocess Gateway API There's a Python binding for the [Coprocess Gateway API](../README.md), this is written using the Cython syntax, it's basically a single file: [`gateway.pyx`](tyk/gateway.pyx). This binding exposes some functions, like a storage handler that allows you to get/set Redis keys: ```python from tyk.decorators import * from gateway import TykGateway as tyk @Pre def SetKeyOnRequest(request, session, spec): tyk.store_data( "my_key", "expiring_soon", 15 ) val = tyk.get_data("cool_key") return request, session ``` ### Cython bindings Cython takes a `.pyx` file and generates a C source file (`.c`) with its corresponding header (`.h`), after this process, we use these two files as part of the `cgo` build process. This approach has been used as an alternative to `cffi`, which introduced an additional step into the setup, requiring the user to install the module first. So in practice, we don't use the `.pyx` files directly (and they aren't required at runtime!). When the build process is over, the bindings are part of the Tyk binary and can be loaded and accessed from Go code. The bindings [declare an initialization function](tyk/gateway.h) that should be called after the Python interpreter is invoked, this will load the actual module and make it possible to import it using `import mymodule`. This is how [`gateway.pyx`](tyk/gateway.pyx) and its functions become available. ### Middleware and wrappers There are [quick wrappers](tyk/) for the HTTP request and session objects, written in Python, the idea is to provide an idiomatic way of writing middleware: ```python from tyk.decorators import * @Pre def AppendHeader(request, session, spec): request.add_header("custom_header", "custom_value") return request, session ``` The decorators provide a simple way of indicating when it's the right moment to execute your handlers, a handler that is decorated with `Pre` will be called before any authentication occurs, `Post` will occur after authentication and will have access to the `session` object. You may find more information about Tyk middleware [here](https://tyk.io/docs/tyk-api-gateway-v1-9/javascript-plugins/middleware-scripting/). --- ## File: coprocess/README.md # Coprocess - PoC This feature makes it possible to write Tyk middleware using your favorite languages. ## Proto files To change the proto files and update the bindings, see proto/ and proto/update_bindings.sh. ## Python support [Python](https://www.python.org/) support is available, more notes [here](python/README.md). ## Lua support [Lua](https://www.lua.org/) support is available as well. Notes [here](lua/README.md). ## gRPC support Tyk provides support for [gRPC](http://www.grpc.io/), you may use any of the gRPC supported languages (e.g. Ruby, Java, etc.). Check the [gRPC support README](grpc/README.md). ## ID Extractor & auth cache The ID extractor is a very useful mechanism that will let you cache your authentication IDs and prevent certain requests from hitting your CP backend. It takes a set of rules from your API configuration (the rules are set per API). A sample usage will look like this: ```json "custom_middleware": { "pre": [ { "name": "MyPreMiddleware", "require_session": false } ], "id_extractor": { "extract_from": "header", "extract_with": "value", "extractor_config": { "header_name": "Authorization" } }, "driver": "grpc" }, ``` Tyk provides a set of ID extractors that aim to cover the most common use cases, a very simple one is the **value extractor**. ## Interoperability This feature implements an in-process message passing mechanism, based on [Protocol Buffers](https://developers.google.com/protocol-buffers/), any supported languages should provide a function to receive, unmarshal and process this kind of messages. The main interoperability task is achieved by using [cgo](https://golang.org/cmd/cgo/) as a bridge between a supported language -like Python- and the Go codebase. Your C bridge function must accept and return a `CoProcessMessage` data structure like the one described in [`api.h`](api.h), where `p_data` is a pointer to the serialized data and `length` indicates the length of it. ```c struct CoProcessMessage { void* p_data; int length; }; ``` The unpacked data will hold the actual `CoProcessObject` data structure, where `HookType` represents the hook type (see below), `Request` represents the HTTP request and `Session` is the Tyk session data. The `Spec` field holds the API specification data, like organization ID, API ID, etc. ```go type CoProcessObject struct { HookType string Request CoProcessMiniRequestObject Session SessionState Metadata map[string]string Spec map[string]string } ``` ## Coprocess Dispatcher `coprocess.Dispatcher` describes a very simple interface for implementing the dispatcher logic, the required methods are: `Dispatch`, `DispatchEvent` and `Reload`. `Dispatch` accepts a pointer to a `struct CoProcessObject` (as described above) and must return an object of the same type. This method will be called for every configured hook, on every request. Traditionally this method will perform a single function call on the target language side (like `Python_DispatchHook` in `coprocess_python`), and the corresponding logic will be handled from there (mostly because different languages have different ways of loading, referencing or calling middlewares). `DispatchEvent` provides a way of dispatching Tyk events to a target language. This method doesn't return any variables but does receive a JSON-encoded object containing the event data. For extensibility purposes, this method doesn't use Protocol Buffers, the input is a `[]byte`, the target language will take this (as a `char`) and perform the JSON decoding operation. `Reload` is called when triggering a hot reload, this method could be useful for reloading scripts or modules in the target language. ## Coprocess Dispatcher - Hooks This component is in charge of dispatching your HTTP requests to the custom middlewares, in the right order. The dispatcher follows the standard middleware chain logic and provides a simple mechanism for "hooking" your custom middleware behavior, the supported hooks are: **Pre:** gets executed before any authentication information is extracted from the header or parameter list of the request. **Post:** gets executed after the authentication, validation, throttling, and quota-limiting middleware has been executed, just before the request is proxied upstream. Use this to post-process a request before sending it to your upstream API. **PostKeyAuth:** gets executed right after the autentication process. **CustomAuthCheck:** gets executed as a custom authentication middleware, instead of the standard ones provided by Tyk. Use this to provide your own authentication mechanism. ## Coprocess Gateway API [`coprocess_api.go`](../gateway/coprocess_api.go) provides a bridge between the gateway API and C, any function that needs to be exported should have the `export` keyword: ```go //export TykTriggerEvent func TykTriggerEvent( CEventName *C.char, CPayload *C.char ) { eventName := C.GoString(CEventName) payload := C.GoString(CPayload) FireSystemEvent(tykcommon.TykEvent(eventName), EventMetaDefault{ Message: payload, }) } ``` You should also expect a header file declaration of this function in [`api.h`](api.h), like this: ```c #ifndef TYK_COPROCESS_API #define TYK_COPROCESS_API extern void TykTriggerEvent(char* event_name, char* payload); #endif ``` The language binding will include this header file (or declare the function inline) and perform the necessary steps to call it with the appropriate arguments (like a `ffi` mechanism could do). As a reference, this is how this could be achieved if you're building a [Cython](http://cython.org/) module: ```python cdef extern: void TykTriggerEvent(char* event_name, char* payload); def call(): event_name = 'my event'.encode('utf-8') payload = 'my payload'.encode('utf-8') TykTriggerEvent( event_name, payload ) ``` ## Basic usage The intended way of using a Coprocess middleware is to specify it as part of an API definition: ```json "custom_middleware": { "pre": [ { "name": "MyPreMiddleware", "require_session": false }, { "name": "AnotherPreMiddleware", "require_session": false } ], "post": [ { "name": "MyPostMiddleware", "require_session": false } ], "post_key_auth": [ { "name": "MyPostKeyAuthMiddleware", "require_session": true } ], "auth_check": { "name": "MyAuthCheck" }, "driver": "python" } ``` It's important to note that all hook types support chaining except the custom auth check (`auth_check`). ## Build notes It's possible to use a [build tag](https://golang.org/pkg/go/build/#hdr-Build_Constraints): ``` go build -tags 'coprocess python' ``` ``` go build -tags 'coprocess somelanguage' ``` Each language should implement a ```CoProcessInit``` function, this will be called from the main function when the ```coprocess``` build tag is used. Using the ```coprocess``` build tag with no language tag will fail. A standard build is still possible: ``` go build ``` ```coprocess_dummy.go``` provides a dummy ```CoProcessInit``` function that will be called if you perform a standard Tyk build. This file will be ignored when using the ```coprocess``` build tag, as we expect it to be implemented by a language. ## Tests You must use the `coprocess` build tag to run the tests: ``` go test -tags 'coprocess' go test -run CoProcess -tags 'coprocess' ``` ## References [Trello note](https://trello.com/c/6QNWnF2n/265-coprocess-handlers-middleware-replacements-and-hooks) --- ## File: docs/dev/apidef-oas.md # Translating Tyk OAS API Definition to Tyk Classic API Definition (and vice versa) To ensure feature parity between Tyk OAS APIs and Tyk classic API definitions, follow these guidelines: ## Define necessary Structs Define the necessary structs or add the necessary fields in the `apidef/oas` package. Make sure `json` and `bson` tags are added to the fields. If an `enabled` flag is specified in the OAS contract, make sure a corresponding `disabled` or `enabled` flag is added in the classic API definition. Also make sure that `disabled`/`enabled` flag toggles the feature on or off. ### Why `disabled` or `enabled` in classic API definition? Historically, almost every feature/middleware in Tyk is considered enabled by default when value for feature controls are non zero values. It is disabled when the feature controls are having zero values. For this reason, whenever an existing feature is migrated to OAS, and OAS has an `enabled` flag then a `disabled` flag is added to give explicit control to turn off the feature. Please also make sure that the disabled flags are set to true in `APIDefinition.SetDisabledFlags()`, so that it is not enabled in OAS by default. ## Use camelCase Notation Ensure OAS fields follow `camelCase` notation for `json` and `bson` tags. ## Handle Required Fields For fields that are required: 1. Do not use `omitempty` in struct tags. 2. Do not use pointer types for required fields. 3. Add a comment `// required` towards the end of a required field so that automation generates docs accordingly. 4. As a convention, we also try to add the corresponding classic API definition fields in godoc in the following format ``` // Tyk classic API definition: `!use_keyless`. ``` This might not be perfect at this moment, but we aim to keep this link so that customers find it easier to follow the docs. ## Handle Optional Fields For optional fields: 1. Add the `omitempty` tag 2. Use pointer types for structs. 3. Make sure that `omitempty` tag is added for slice fields that are optional. ## Add Go Doc Comments Add comments in Go doc format for each field to enable automated documentation generation (this is validated by linter). ## Implement Fill and ExtractTo Methods Every OAS struct should follow the convention of having `Fill(apidef.APIDefinition)` and `ExtractTo(*apidef.APIDefinition)` methods: `Fill` populates the struct from a classic API definition. `ExtractTo` extracts the contents of an OAS API definition into a classic API definition. ## Implement Fill Method Pattern Each `Fill` method should follow this pattern: ```go if u.RateLimit == nil { u.RateLimit = &RateLimit{} } u.RateLimit.Fill(api) if ShouldOmit(u.RateLimit) { u.RateLimit = nil } ``` This ensures the field is reset to empty when not configured in the classic API definition. ### Working with `VersionData` A `Main` version will be provided that can be used for `Fill`. ```go api.VersionData.Versions[Main] ``` ## Implement ExtractTo Method Pattern Similarly, follow this pattern with `ExtractTo`: ```go if u.RateLimit == nil { u.RateLimit = &RateLimit{} defer func() { u.RateLimit = nil }() } u.RateLimit.ExtractTo(api) ``` ### Working with `VersionData` There are 2 helper functions for `ExtractTo` that will help to handle `VersionData`. You can use them like this: ```go func (g *GlobalRequestSizeLimit) ExtractTo(api *apidef.APIDefinition) { mainVersion := requireMainVersion(api) defer func() { updateMainVersion(api, mainVersion) }() // manipulate the Main VersionInfo here } ``` ## Write Tests Write tests for conversion functions. Refer to the example: https://github.com/TykTechnologies/tyk/pull/5979/files#diff-222cc254c0c6c09fa0cf50087860b837a0873e2aef3c84ec7d80b1014c149057R97 ## Update TestOAS_ExtractTo_ResetAPIDefinition Maintain and update the list of fields that are not OAS compatible in the `TestOAS_ExtractTo_ResetAPIDefinition` test. ## Update JSON Schema Update the JSON schema for the `x-tyk-api-gateway` struct in: https://github.com/TykTechnologies/tyk/blob/master/apidef/oas/schema/x-tyk-api-gateway.json Ensure this schema is updated whenever the OAS API definition is modified.