logstash Documentation

File: README.md Copy Chapter

Logstash

Logstash is part of the Elastic Stack along with Beats, Elasticsearch and Kibana. Logstash is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.). Logstash has over 200 plugins, and you can write your own very easily as well.

For more info, see https://www.elastic.co/products/logstash

Documentation and Getting Started Copy Chapter

You can find the documentation and getting started guides for Logstash
on the elastic.co site

For information about building the documentation, see the README in https://github.com/elastic/docs

Downloads Copy Chapter

You can download officially released Logstash binaries, as well as debian/rpm packages for the
supported platforms, from downloads page.

Need Help? Copy Chapter

• Logstash Forum
• Logstash Documentation
• Logstash Product Information
• Elastic Support

Logstash Plugins Copy Chapter

Logstash plugins are hosted in separate repositories under the logstash-plugins github organization. Each plugin is a self-contained Ruby gem which gets published to RubyGems.org.

Writing your own Plugin Copy Section

Logstash is known for its extensibility. There are hundreds of plugins for Logstash and you can write your own very easily! For more info on developing and testing these plugins, please see the working with plugins section

Plugin Issues and Pull Requests Copy Section

Please open new issues and pull requests for plugins under its own repository

For example, if you have to report an issue/enhancement for the Elasticsearch output, please do so here.

Logstash core will continue to exist under this repository and all related issues and pull requests can be submitted here.

Developing Logstash Core Copy Chapter

Prerequisites Copy Section

• Install JDK version 21. Make sure to set the JAVA_HOME environment variable to the path to your JDK installation directory. For example set JAVA_HOME=<JDK_PATH>
• Install JRuby 10.0.5.0 It is recommended to use a Ruby version manager such as RVM or rbenv.
• Install rake and bundler tool using gem install rake and gem install bundler respectively.

RVM install (optional) Copy Section

If you prefer to use rvm (ruby version manager) to manage Ruby versions on your machine, follow these directions. In the Logstash folder:

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash -s stable --ruby=$(cat .ruby-version)

Check Ruby version Copy Section

Before you proceed, please check your ruby version by:

$ ruby -v

The printed version should be the same as in the .ruby-version file.

Building Logstash Copy Section

The Logstash project includes the source code for all of Logstash, including the Elastic-Licensed X-Pack features and functions; to run Logstash from source using only the OSS-licensed code, export the OSS environment variable with a value of true:

export OSS=true

• Set up the location of the source code to build

export LOGSTASH_SOURCE=1
export LOGSTASH_PATH=/YOUR/LOGSTASH/DIRECTORY

Install dependencies with `gradle` **(recommended)**[^1]

• Install development dependencies

./gradlew installDevelopmentGems

• Install default plugins and other dependencies

./gradlew installDefaultGems

Verify the installation Copy Section

To verify your environment, run the following to start Logstash and send your first event:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

This should start Logstash with stdin input waiting for you to enter an event

hello world
2016-11-11T01:22:14.405+0000 0.0.0.0 hello world

Advanced: Drip Launcher

Drip is a tool that solves the slow JVM startup problem while developing Logstash. The drip script is intended to be a drop-in replacement for the java command. We recommend using drip during development, in particular for running tests. Using drip, the first invocation of a command will not be faster but the subsequent commands will be swift.

To tell logstash to use drip, set the environment variable JAVACMD=`which drip`.

Example (but see the Testing section below before running rspec for the first time):

JAVACMD=`which drip` bin/rspec

Caveats

Drip does not work with STDIN. You cannot use drip for running configs which use the stdin plugin.

Building Logstash Documentation Copy Chapter

To build the Logstash Reference (open source content only) on your local
machine, clone the following repos:

logstash - contains main docs about core features

logstash-docs - contains generated plugin docs

docs - contains doc build files

Make sure you have the same branch checked out in logstash and logstash-docs.
Check out master in the docs repo.

Run the doc build script from within the docs repo. For example:

./build_docs.pl --doc ../logstash/docs/index.asciidoc --chunk=1 -open

Testing Copy Chapter

Most of the unit tests in Logstash are written using rspec for the Ruby parts. For the Java parts, we use junit. For testing you can use the test rake tasks and the bin/rspec command, see instructions below:

Core tests Copy Section

1- To run the core tests you can use the Gradle task:

./gradlew test

or use the rspec tool to run all tests or run a specific test:

bin/rspec
bin/rspec spec/foo/bar_spec.rb

Note that before running the rspec command for the first time you need to set up the RSpec test dependencies by running:

./gradlew bootstrap

2- To run the subset of tests covering the Java codebase only run:

./gradlew javaTests

3- To execute the complete test-suite including the integration tests run:

./gradlew check

4- To execute a single Ruby test run:

SPEC_OPTS="-fd -P logstash-core/spec/logstash/api/commands/default_metadata_spec.rb" ./gradlew :logstash-core:rubyTests --tests org.logstash.RSpecTests

5- To execute single spec for integration test, run:

./gradlew integrationTests -PrubyIntegrationSpecs=specs/slowlog_spec.rb

Sometimes you might find a change to a piece of Logstash code causes a test to hang. These can be hard to debug.

If you set LS_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" you can connect to a running Logstash with your IDEs debugger which can be a great way of finding the issue.

Plugins tests Copy Section

To run the tests of all currently installed plugins:

rake test:plugins

You can install the default set of plugins included in the logstash package:

rake test:install-default

Note that if a plugin is installed using the plugin manager bin/logstash-plugin install ... do not forget to also install the plugins development dependencies using the following command after the plugin installation:

bin/logstash-plugin install --development

Building Artifacts Copy Chapter

Built artifacts will be placed in the LS_HOME/build directory, and will create the directory if it is not already present.

You can build a Logstash snapshot package as tarball or zip file

./gradlew assembleTarDistribution
./gradlew assembleZipDistribution

OSS-only artifacts can similarly be built with their own gradle tasks:

./gradlew assembleOssTarDistribution
./gradlew assembleOssZipDistribution

You can also build .rpm and .deb, but the fpm tool is required.

rake artifact:rpm
rake artifact:deb

and:

rake artifact:rpm_oss
rake artifact:deb_oss

Using a Custom JRuby Distribution Copy Chapter

If you want the build to use a custom JRuby you can do so by setting a path to a custom
JRuby distribution's source root via the custom.jruby.path Gradle property.

E.g.

./gradlew clean test -Pcustom.jruby.path="/path/to/jruby"

Project Principles Copy Chapter

• Community: If a newbie has a bad time, it's a bug.
• Software: Make it work, then make it right, then make it fast.
• Technology: If it doesn't do a thing today, we can make it do it tomorrow.

Contributing Copy Chapter

All contributions are welcome: ideas, patches, documentation, bug reports,
complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and
maintainers or community members saying "send patches or die" - you will not
see that here.

It is more important that you are able to contribute.

For more information about contributing, see the
CONTRIBUTING file.

Footnotes Copy Chapter

[^1]: Use bundle instead of gradle to install dependencies

#### Alternatively, instead of using `gradle` you can also use `bundle`:

* Install development dependencies

    ```sh
    bundle config set --local path vendor/bundle
    bundle install
    ```

* Bootstrap the environment:

    ```sh
    rake bootstrap
    ```

* You can then use `bin/logstash` to start Logstash, but there are no plugins installed. To install default plugins, you can run:

    ```sh
    rake plugin:install-default
    ```

This will install the 80+ default plugins which makes Logstash ready to connect to multiple data sources, perform transformations and send the results to Elasticsearch and other destinations.

File: ci/serverless/README.md Copy Chapter

The test cases against serverless Elasticsearch cover the following scenarios

• es-output
• es-input
• es-filter
• elastic-integration-filter (Logstash run ingest pipeline)
• DLQ
• central pipeline management
• Kibana API for pipeline management (CPM)
• Metricbeat monitoring
• Logstash legacy monitoring

Setup testing endpoint Copy Section

1. Go to https://console.qa.cld.elstc.co
2. Create deployment. Choose AWS as cloud provider.
3. Create fully-managed project
   • save the credentials of superuser
4. Create API key
   • save the api keys for metricbeat and elasticsearch
   • get the elasticsearch and kibana endpoint from UI or API
5. Save credentials to Vault

Create API key Copy Section

Logstash

Plugin user

POST /_security/api_key
{
  "name": "plugin_user",
  "expiration": "365d",   
  "role_descriptors": { 
    "plugin_user_role": {
      "cluster": ["manage_index_templates", "monitor"], 
      "indices": [
        {
          "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "serverless*", "logs-*", "metrics-*", "synthetics-*", "traces-*" ], 
          "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"]  
        }
      ]
    }
  }
}

Integration User

POST /_security/api_key
{
  "name": "integration_user",
  "expiration": "365d",   
  "role_descriptors": { 
    "integration_user_role": {
      "cluster": ["manage_index_templates", "read_pipeline", "monitor"]
    }
  }
}

CPM User

POST /_security/api_key
{
  "name": "cpm_user",
  "expiration": "365d",   
  "role_descriptors": { 
    "cpm_user_role": {
      "cluster": ["manage_logstash_pipelines", "monitor"]
    }
  }
}

Tester

POST /_security/api_key
{
  "name": "tester_user",
  "expiration": "365d",   
  "role_descriptors": { 
    "tester_user_role": {
      "cluster": ["manage_index_templates", "manage_logstash_pipelines","manage_ingest_pipelines"], 
      "indices": [
        {
          "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "serverless*", "logs-*", "metrics-*", "synthetics-*", "traces-*", "*test*" ], 
          "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"]  
        }
      ]
    }
  }
}

MetricBeat

Grant metricbeat write permission.

POST /_security/api_key
{
  "name": "metricbeat_user", 
  "role_descriptors": {
    "metricbeat_user_role": { 
      "cluster": ["monitor", "read_ilm", "read_pipeline"],
      "index": [
        {
          "names": ["metricbeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

Save credentials to Vault Copy Section

Setup Vault if you haven't.

The username, password, API key and hosts are stored in secret/ci/elastic-logstash/serverless-test.

vault write secret/ci/elastic-logstash/serverless-test es_host="REDACTED" es_superuser="REDACTED" es_superuser_pw="REDACTED" " kb_host="REDACTED" mb_api_key="REDACTED" plugin_api_key="REDACTED" integration_api_key_encoded="REDACTED" tester_api_key_encoded="REDACTED" cpm_api_key="REDACTED"

File: docker/ironbank/README.md Copy Chapter

Logstash

Logstash is part of the Elastic Stack along with Elasticsearch, Kibana, and Beats. Logstash is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.). Logstash has over 200 plugins, and you can write your own very easily as well.

For more info, see https://www.elastic.co/products/logstash

Installation instructions Copy Section

Please follow the documentation on how to install Logstash with Docker.

Documentation and Getting Started Copy Chapter

You can find the documentation and getting started guides for Logstash
on the elastic.co site.

Where to file issues and PRs Copy Section

• Issues
• PRs

Please open new issues and pull requests for plugins under its own repository

For example, if you have to report an issue/enhancement for the Elasticsearch output, please do so here.

Need Help? Copy Chapter

• Logstash Forum
• Logstash Documentation
• Elastic Support

Project Principles Copy Chapter

• Community: If a newbie has a bad time, it's a bug.
• Software: Make it work, then make it right, then make it fast.
• Technology: If it doesn't do a thing today, we can make it do it tomorrow.

File: docker/README.md Copy Chapter

Description Copy Chapter

This repository contains the official Logstash Docker image from
Elastic.

Documentation can be found on the Elastic website.

Supported Docker versions Copy Chapter

The images have been tested on Docker version 18.09.2, build 6247962

Requirements Copy Chapter

A full build requires:

• Docker
• GNU Make
• Python 3.5 with Virtualenv
• JRuby 9.1+

Running a build Copy Chapter

To build an image check out the corresponding branch for the version and run the rake task
Like this:

git checkout 7.0
rake artifact:docker
# and for the OSS package
rake artifact:docker_oss

This image is built on Ubuntu 20.04.

File: docs/extend/codec-new-plugin.md Copy Chapter

mapped_pages:

• https://www.elastic.co/guide/en/logstash/current/codec-new-plugin.html

How to write a Logstash codec plugin [codec-new-plugin]

To develop a new codec for Logstash, build a self-contained Ruby gem whose source code lives in its own GitHub repository. The Ruby gem can then be hosted and shared on RubyGems.org. You can use the example codec implementation as a starting point. (If you’re unfamiliar with Ruby, you can find an excellent quickstart guide at https://www.ruby-lang.org/en/documentation/quickstart/.)

Get started [_get_started_2] Copy Chapter

Let’s step through creating a codec plugin using the example codec plugin.

Create a GitHub repo for your new plugin [_create_a_github_repo_for_your_new_plugin_2] Copy Section

Each Logstash plugin lives in its own GitHub repository. To create a new repository for your plugin:

1. Log in to GitHub.

2. Click the Repositories tab. You’ll see a list of other repositories you’ve forked or contributed to.

3. Click the green New button in the upper right.

4. Specify the following settings for your new repo:

   • Repository name — a unique name of the form logstash-codec-pluginname.
   • Public or Private — your choice, but the repository must be Public if you want to submit it as an official plugin.
   • Initialize this repository with a README — enables you to immediately clone the repository to your computer.

5. Click Create Repository.

Use the plugin generator tool [_use_the_plugin_generator_tool_2] Copy Section

You can create your own Logstash plugin in seconds! The generate subcommand of bin/logstash-plugin creates the foundation for a new Logstash plugin with templatized files. It creates the correct directory structure, gemspec files, and dependencies so you can start adding custom code to process data with Logstash.

For more information, see Generating plugins

Copy the codec code [_copy_the_codec_code] Copy Section

Alternatively, you can use the examples repo we host on github.com

1. Clone your plugin. Replace GITUSERNAME with your github username, and MYPLUGINNAME with your plugin name.

   • git clone https://github.com/GITUSERNAME/logstash-``codec-MYPLUGINNAME.git

     • alternately, via ssh: git clone [email protected]:GITUSERNAME/logstash``-codec-MYPLUGINNAME.git

   • cd logstash-codec-MYPLUGINNAME

2. Clone the codec plugin example and copy it to your plugin branch.

   You don’t want to include the example .git directory or its contents, so delete it before you copy the example.

   • cd /tmp
   • git clone https://github.com/logstash-plugins/logstash``-codec-example.git
   • cd logstash-codec-example
   • rm -rf .git
   • cp -R * /path/to/logstash-codec-mypluginname/

3. Rename the following files to match the name of your plugin.

   • logstash-codec-example.gemspec

   • example.rb

   • example_spec.rb

     txt Copy

     cd /path/to/logstash-codec-mypluginname
     mv logstash-codec-example.gemspec logstash-codec-mypluginname.gemspec
     mv lib/logstash/codecs/example.rb lib/logstash/codecs/mypluginname.rb
     mv spec/codecs/example_spec.rb spec/codecs/mypluginname_spec.rb

Your file structure should look like this:

$ tree logstash-codec-mypluginname
├── Gemfile
├── LICENSE
├── README.md
├── Rakefile
├── lib
│   └── logstash
│       └── codecs
│           └── mypluginname.rb
├── logstash-codec-mypluginname.gemspec
└── spec
    └── codecs
        └── mypluginname_spec.rb

For more information about the Ruby gem file structure and an excellent walkthrough of the Ruby gem creation process, see http://timelessrepo.com/making-ruby-gems

See what your plugin looks like [_see_what_your_plugin_looks_like_2] Copy Section

Before we dive into the details, open up the plugin file in your favorite text editor and take a look.

require "logstash/codecs/base"
require "logstash/codecs/line"

# Add any asciidoc formatted documentation here
class LogStash::Codecs::Example < LogStash::Codecs::Base

  # This example codec will append a string to the message field
  # of an event, either in the decoding or encoding methods
  #
  # This is only intended to be used as an example.
  #
  # input {
  #   stdin { codec => example }
  # }
  #
  # or
  #
  # output {
  #   stdout { codec => example }
  # }
  config_name "example"

  # Append a string to the message
  config :append, :validate => :string, :default => ', Hello World!'

  public
  def register
    @lines = LogStash::Codecs::Line.new
    @lines.charset = "UTF-8"
  end

  public
  def decode(data)
    @lines.decode(data) do |line|
      replace = { "message" => line["message"].to_s + @append }
      yield LogStash::Event.new(replace)
    end
  end # def decode

  public
  def encode(event)
    @on_event.call(event, event.get("message").to_s + @append + NL)
  end # def encode

end # class LogStash::Codecs::Example

Coding codec plugins [_coding_codec_plugins] Copy Chapter

Now let’s take a line-by-line look at the example plugin.

`require` Statements [_require_statements_2] Copy Section

Logstash codec plugins require parent classes defined in logstash/codecs/base and logstash/namespace:

require "logstash/codecs/base"
require "logstash/namespace"

Of course, the plugin you build may depend on other code, or even gems. Just put them here along with these Logstash dependencies.

Plugin Body [_plugin_body_2] Copy Chapter

Let’s go through the various elements of the plugin itself.

`class` Declaration [_class_declaration_2] Copy Section

The codec plugin class should be a subclass of LogStash::Codecs::Base:

class LogStash::Codecs::Example < LogStash::Codecs::Base

The class name should closely mirror the plugin name, for example:

LogStash::Codecs::Example

`config_name` [_config_name_2] Copy Section

config_name "example"

This is the name your plugin will call inside the codec configuration block.

If you set config_name "example" in your plugin code, the corresponding Logstash configuration block would need to look like this:

Configuration Parameters [_configuration_parameters_2] Copy Chapter

config :variable_name, :validate => :variable_type, :default => "Default value", :required => boolean, :deprecated => boolean, :obsolete => string

The configuration, or config section allows you to define as many (or as few) parameters as are needed to enable Logstash to process events.

There are several configuration attributes:

• :validate - allows you to enforce passing a particular data type to Logstash for this configuration option, such as :string, :password, :boolean, :number, :array, :hash, :path (a file-system path), uri, :codec (since 1.2.0), :bytes. Note that this also works as a coercion in that if I specify "true" for boolean (even though technically a string), it will become a valid boolean in the config. This coercion works for the :number type as well where "1.2" becomes a float and "22" is an integer.
• :default - lets you specify a default value for a parameter
• :required - whether or not this parameter is mandatory (a Boolean true or
• :list - whether or not this value should be a list of values. Will typecheck the list members, and convert scalars to one element lists. Note that this mostly obviates the array type, though if you need lists of complex objects that will be more suitable. false)
• :deprecated - informational (also a Boolean true or false)
• :obsolete - used to declare that a given setting has been removed and is no longer functioning. The idea is to provide an informed upgrade path to users who are still using a now-removed setting.

Plugin Methods [_plugin_methods_2] Copy Chapter

Logstash codecs must implement the register method, and the decode method or the encode method (or both).

`register` Method [_register_method_2] Copy Section

public
  def register
  end # def register

The Logstash register method is like an initialize method. It was originally created to enforce having super called, preventing headaches for newbies. (Note: It may go away in favor of initialize, in conjunction with some enforced testing to ensure super is called.)

public means the method can be called anywhere, not just within the class. This is the default behavior for methods in Ruby, but it is specified explicitly here anyway.

You can also assign instance variables here (variables prepended by @). Configuration variables are now in scope as instance variables, like @message

`decode` Method [_decode_method] Copy Section

public
  def decode(data)
    @lines.decode(data) do |line|
      replace = { "message" => line["message"].to_s + @append }
      yield LogStash::Event.new(replace)
    end
  end # def decode

The codec’s decode method is where data coming in from an input is transformed into an event. There are complex examples like the collectd codec, and simpler examples like the spool codec.

There must be a yield statement as part of the decode method which will return decoded events to the pipeline.

`encode` Method [_encode_method] Copy Section

public
  def encode(event)
    @on_event.call(event, event.get("message").to_s + @append + NL)
  end # def encode

The encode method takes an event and serializes it (encodes) into another format. Good examples of encode methods include the simple plain codec, the slightly more involved msgpack codec, and even an avro codec.

In most cases, your encode method should have an @on_event.call() statement. This call will output data per event in the described way.

Building the Plugin [_building_the_plugin_2] Copy Chapter

At this point in the process you have coded your plugin and are ready to build a Ruby Gem from it. The following information will help you complete the process.

External dependencies [_external_dependencies_2] Copy Section

A require statement in Ruby is used to include necessary code. In some cases your plugin may require additional files. For example, the collectd plugin uses the types.db file provided by collectd. In the main directory of your plugin, a file called vendor.json is where these files are described.

The vendor.json file contains an array of JSON objects, each describing a file dependency. This example comes from the collectd codec plugin:

[{
        "sha1": "a90fe6cc53b76b7bdd56dc57950d90787cb9c96e",
        "url": "http://collectd.org/files/collectd-5.4.0.tar.gz",
        "files": [ "/src/types.db" ]
}]

• sha1 is the sha1 signature used to verify the integrity of the file referenced by url.
• url is the address from where Logstash will download the file.
• files is an optional array of files to extract from the downloaded file. Note that while tar archives can use absolute or relative paths, treat them as absolute in this array. If files is not present, all files will be uncompressed and extracted into the vendor directory.

Another example of the vendor.json file is the geoip filter

The process used to download these dependencies is to call rake vendor. This will be discussed further in the testing section of this document.

Another kind of external dependency is on jar files. This will be described in the "Add a gemspec file" section.

Deprecated features [_deprecated_features_2] Copy Section

As a plugin evolves, an option or feature may no longer serve the intended purpose, and the developer may want to deprecate its usage. Deprecation warns users about the option’s status, so they aren’t caught by surprise when it is removed in a later release.

{{ls}} 7.6 introduced a deprecation logger to make handling those situations easier. You can use the adapter to ensure that your plugin can use the deprecation logger while still supporting older versions of {{ls}}. See the readme for more information and for instructions on using the adapter.

Deprecations are noted in the logstash-deprecation.log file in the log directory.

Add a Gemfile [_add_a_gemfile_2] Copy Section

Gemfiles allow Ruby’s Bundler to maintain the dependencies for your plugin. Currently, all we’ll need is the Logstash gem, for testing, but if you require other gems, you should add them in here.

::::{tip}
See Bundler’s Gemfile page for more details.
::::

source 'https://rubygems.org'
gemspec
gem "logstash", :github => "elastic/logstash", :branch => "master"

Add a `gemspec` file [_add_a_gemspec_file_2] Copy Chapter

Gemspecs define the Ruby gem which will be built and contain your plugin.

::::{tip}
More information can be found on the Rubygems Specification page.
::::

Gem::Specification.new do |s|
  s.name = 'logstash-codec-example'
  s.version = '0.1.0'
  s.licenses = ['Apache License (2.0)']
  s.summary = "This codec does x, y, z in Logstash"
  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
  s.authors = ["Elastic"]
  s.email = '[email protected]'
  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
  s.require_paths = ["lib"]

  # Files
  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
   # Tests
  s.test_files = s.files.grep(%r{^(test|spec|features)/})

  # Special flag to let us know this is actually a logstash plugin
  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" }

  # Gem dependencies
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
  s.add_development_dependency 'logstash-devutils'
end

It is appropriate to change these values to fit your plugin. In particular, s.name and s.summary should reflect your plugin’s name and behavior.

s.licenses and s.version are also important and will come into play when you are ready to publish your plugin.

Logstash and all its plugins are licensed under Apache License, version 2 ("ALv2"). If you make your plugin publicly available via RubyGems.org, please make sure to have this line in your gemspec:

• s.licenses = ['Apache License (2.0)']

The gem version, designated by s.version, helps track changes to plugins over time. You should use semver versioning strategy for version numbers.

Runtime and Development Dependencies [_runtime_and_development_dependencies_2] Copy Section

At the bottom of the gemspec file is a section with a comment: Gem dependencies. This is where any other needed gems must be mentioned. If a gem is necessary for your plugin to function, it is a runtime dependency. If a gem are only used for testing, then it would be a development dependency.

::::{note}
You can also have versioning requirements for your dependencies—​including other Logstash plugins:

# Gem dependencies
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
  s.add_development_dependency 'logstash-devutils'

This gemspec has a runtime dependency on the logstash-core-plugin-api and requires that it have a version number greater than or equal to version 1.60 and less than or equal to version 2.99.

::::

::::{important}
All plugins have a runtime dependency on the logstash-core-plugin-api gem, and a development dependency on logstash-devutils.
::::

Jar dependencies [_jar_dependencies_2] Copy Section

In some cases, such as the Elasticsearch output plugin, your code may depend on a jar file. In cases such as this, the dependency is added in the gemspec file in this manner:

# Jar dependencies
  s.requirements << "jar 'org.elasticsearch:elasticsearch', '5.0.0'"
  s.add_runtime_dependency 'jar-dependencies'

With these both defined, the install process will search for the required jar file at http://mvnrepository.com and download the specified version.

Document your plugin [_document_your_plugin_2] Copy Chapter

Documentation is an important part of your plugin. All plugin documentation is rendered and placed in the Logstash Reference and the Versioned plugin docs.

See Document your plugin for tips and guidelines.

Add Tests [_add_tests_2] Copy Chapter

Logstash loves tests. Lots of tests. If you’re using your new codec plugin in a production environment, you’ll want to have some tests to ensure you are not breaking any existing functionality.

::::{note}
A full exposition on RSpec is outside the scope of this document. Learn more about RSpec at http://rspec.info
::::

For help learning about tests and testing, look in the spec/codecs/ directory of several other similar plugins.

Clone and test! [_clone_and_test_2] Copy Chapter

Now let’s start with a fresh clone of the plugin, build it and run the tests.

• Clone your plugin into a temporary location Replace GITUSERNAME with your github username, and MYPLUGINNAME with your plugin name.

  • git clone https://github.com/GITUSERNAME/logstash-``codec-MYPLUGINNAME.git

    • alternately, via ssh: git clone [email protected]:GITUSERNAME/logstash-``codec-MYPLUGINNAME.git

  • cd logstash-codec-MYPLUGINNAME

Then, you’ll need to install your plugins dependencies with bundler:

bundle install

::::{important}
If your plugin has an external file dependency described in vendor.json, you must download that dependency before running or testing. You can do this by running:

rake vendor

::::

And finally, run the tests:

bundle exec rspec

You should see a success message, which looks something like this:

Finished in 0.034 seconds
1 example, 0 failures

Hooray! You’re almost there! (Unless you saw failures…​ you should fix those first).

Building and Testing [_building_and_testing_2] Copy Chapter

Now you’re ready to build your (well-tested) plugin into a Ruby gem.

Build [_build_2] Copy Section

You already have all the necessary ingredients, so let’s go ahead and run the build command:

gem build logstash-codec-example.gemspec

That’s it! Your gem should be built and be in the same path with the name

logstash-codec-mypluginname-0.1.0.gem

The s.version number from your gemspec file will provide the gem version, in this case, 0.1.0.

Test installation [_test_installation_2] Copy Section

You should test install your plugin into a clean installation of Logstash. Download the latest version from the Logstash downloads page.

1. Untar and cd in to the directory:

   sh Copy

   curl -O https://download.elastic.co/logstash/logstash/logstash-9.0.0.tar.gz
   tar xzvf logstash-9.0.0.tar.gz
   cd logstash-9.0.0

2. Using the plugin tool, we can install the gem we just built.

   • Replace /my/logstash/plugins with the correct path to the gem for your environment, and 0.1.0 with the correct version number from the gemspec file.

     sh Copy

     bin/logstash-plugin install /my/logstash/plugins/logstash-codec-example/logstash-codec-example-0.1.0.gem

   • After running this, you should see feedback from Logstash that it was successfully installed:

     sh Copy

     validating /my/logstash/plugins/logstash-codec-example/logstash-codec-example-0.1.0.gem >= 0
     Valid logstash plugin. Continuing...
     Successfully installed 'logstash-codec-example' with version '0.1.0'

     ::::{tip}
     You can also use the Logstash plugin tool to determine which plugins are currently available:

     sh Copy

     bin/logstash-plugin list

     Depending on what you have installed, you might see a short or long list of plugins: inputs, codecs, filters and outputs.

     ::::

3. Now try running Logstash with a simple configuration passed in via the command-line, using the -e flag.

   ::::{note}
   Your results will depend on what your codec plugin is designed to do.
   ::::

bin/logstash -e 'input { stdin{ codec => example{}} } output {stdout { codec => rubydebug }}'

The example codec plugin will append the contents of append (which by default appends ", Hello World!")

After starting Logstash, type something, for example "Random output string". The resulting output message field contents should be, "Random output string, Hello World!":

Random output string
{
       "message" => "Random output string, Hello World!",
      "@version" => "1",
    "@timestamp" => "2015-01-27T19:17:18.932Z",
          "host" => "cadenza"
}

Feel free to experiment and test this by changing the append parameter:

bin/logstash -e 'input { stdin{ codec => example{ append => ", I am appending this! }} } output {stdout { codec => rubydebug }}'

Congratulations! You’ve built, deployed and successfully run a Logstash codec.

Submitting your plugin to [RubyGems.org](http://rubygems.org) and [logstash-plugins](https://github.com/logstash-plugins) [_submitting_your_plugin_to_rubygems_orghttprubygems_org_and_logstash_pluginshttpsgithub_comlogstash_plugins_2] Copy Chapter

Logstash uses RubyGems.org as its repository for all plugin artifacts. Once you have developed your new plugin, you can make it available to Logstash users by simply publishing it to RubyGems.org.

Licensing [_licensing_2] Copy Section

Logstash and all its plugins are licensed under Apache License, version 2 ("ALv2"). If you make your plugin publicly available via RubyGems.org, please make sure to have this line in your gemspec:

• s.licenses = ['Apache License (2.0)']

Publishing to [RubyGems.org](http://rubygems.org) [_publishing_to_rubygems_orghttprubygems_org_2] Copy Section

To begin, you’ll need an account on RubyGems.org

• Sign-up for a RubyGems account.

After creating an account, obtain an API key from RubyGems.org. By default, RubyGems uses the file ~/.gem/credentials to store your API key. These credentials will be used to publish the gem. Replace username and password with the credentials you created at RubyGems.org:

curl -u username:password https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
chmod 0600 ~/.gem/credentials

Before proceeding, make sure you have the right version in your gemspec file and commit your changes.

• s.version = '0.1.0'

To publish version 0.1.0 of your new logstash gem:

bundle install
bundle exec rake vendor
bundle exec rspec
bundle exec rake publish_gem

::::{note}
Executing rake publish_gem:

1. Reads the version from the gemspec file (s.version = '0.1.0')
2. Checks in your local repository if a tag exists for that version. If the tag already exists, it aborts the process. Otherwise, it creates a new version tag in your local repository.
3. Builds the gem
4. Publishes the gem to RubyGems.org

::::

That’s it! Your plugin is published! Logstash users can now install your plugin by running:

bin/logstash-plugin install logstash-codec-mypluginname

Contributing your source code to [logstash-plugins](https://github.com/logstash-plugins) [_contributing_your_source_code_to_logstash_pluginshttpsgithub_comlogstash_plugins_2] Copy Chapter

It is not required to contribute your source code to logstash-plugins github organization, but we always welcome new plugins!

Benefits [_benefits_2] Copy Section

Some of the many benefits of having your plugin in the logstash-plugins repository are:

• Discovery. Your plugin will appear in the Logstash Reference, where Logstash users look first for plugins and documentation.
• Documentation. Your plugin documentation will automatically be added to the Logstash Reference.
• Testing. With our testing infrastructure, your plugin will be continuously tested against current and future releases of Logstash. As a result, users will have the assurance that if incompatibilities arise, they will be quickly discovered and corrected.

Acceptance Guidelines [_acceptance_guidelines_2] Copy Section

• Code Review. Your plugin must be reviewed by members of the community for coherence, quality, readability, stability and security.
• Tests. Your plugin must contain tests to be accepted. These tests are also subject to code review for scope and completeness. It’s ok if you don’t know how to write tests — we will guide you. We are working on publishing a guide to creating tests for Logstash which will make it easier. In the meantime, you can refer to http://betterspecs.org/ for examples.

To begin migrating your plugin to logstash-plugins, simply create a new issue in the Logstash repository. When the acceptance guidelines are completed, we will facilitate the move to the logstash-plugins organization using the recommended github process.

File: docs/extend/community-maintainer.md Copy Chapter

mapped_pages:

• https://www.elastic.co/guide/en/logstash/current/community-maintainer.html

Logstash Plugins Community Maintainer Guide [community-maintainer]

This document, to be read by new Maintainers, should explain their responsibilities. It was inspired by the C4 document from the ZeroMQ project. This document is subject to change and suggestions through Pull Requests and issues are strongly encouraged.

Contribution Guidelines [_contribution_guidelines] Copy Chapter

For general guidance around contributing to Logstash Plugins, see the Contributing to Logstash section.

Document Goals [_document_goals] Copy Chapter

To help make the Logstash plugins community participation easy with positive feedback.

To increase diversity.

To reduce code review, merge and release dependencies on the core team by providing support and tools to the Community and Maintainers.

To support the natural life cycle of a plugin.

To codify the roles and responsibilities of: Maintainers and Contributors with specific focus on patch testing, code review, merging and release.

Development Workflow [_development_workflow] Copy Chapter

All Issues and Pull Requests must be tracked using the Github issue tracker.

The plugin uses the Apache 2.0 license. Maintainers should check whether a patch introduces code which has an incompatible license. Patch ownership and copyright is defined in the Elastic Contributor License Agreement (CLA).

Terminology [_terminology_2] Copy Section

A "Contributor" is a role a person assumes when providing a patch. Contributors will not have commit access to the repository. They need to sign the Elastic Contributor License Agreement before a patch can be reviewed. Contributors can add themselves to the plugin Contributor list.

A "Maintainer" is a role a person assumes when maintaining a plugin and keeping it healthy, including triaging issues, and reviewing and merging patches.

Patch Requirements [_patch_requirements] Copy Section

A patch is a minimal and accurate answer to exactly one identified and agreed upon problem. It must conform to the code style guidelines and must include RSpec tests that verify the fitness of the solution.

A patch will be automatically tested by a CI system that will report on the Pull Request status.

A patch CLA will be automatically verified and reported on the Pull Request status.

A patch commit message has a single short (less than 50 character) first line summarizing the change, a blank second line, and any additional lines as necessary for change explanation and rationale.

A patch is mergeable when it satisfies the above requirements and has been reviewed positively by at least one other person.

Development Process [_development_process] Copy Section

A user will log an issue on the issue tracker describing the problem they face or observe with as much detail as possible.

To work on an issue, a Contributor forks the plugin repository and then works on their forked repository and submits a patch by creating a pull request back to the plugin.

Maintainers must not merge patches where the author has not signed the CLA.

Before a patch can be accepted it should be reviewed. Maintainers should merge accepted patches without delay.

Maintainers should not merge their own patches except in exceptional cases, such as non-responsiveness from other Maintainers or core team for an extended period (more than 2 weeks).

Reviewer’s comments should not be based on personal preferences.

The Maintainers should label Issues and Pull Requests.

Maintainers should involve the core team if help is needed to reach consensus.

Review non-source changes such as documentation in the same way as source code changes.

Branch Management [_branch_management] Copy Section

The plugin has a main branch that always holds the latest in-progress version and should always build. Topic branches should kept to the minimum.

Changelog Management [_changelog_management] Copy Section

Every plugin should have a changelog (https://www.elastic.co/guide/en/logstash/current/CHANGELOG.html). If not, please create one. When changes are made to a plugin, make sure to include a changelog entry under the respective version to document the change. The changelog should be easily understood from a user point of view. As we iterate and release plugins rapidly, users use the changelog as a mechanism for deciding whether to update.

Changes that are not user facing should be tagged as internal:. For example:

- internal: Refactored specs for better testing
 - config: Default timeout configuration changed from 10s to 5s

Detailed format of https://www.elastic.co/guide/en/logstash/current/CHANGELOG.html [_detailed_format_of_changelog_md]

Sharing a similar format of https://www.elastic.co/guide/en/logstash/current/CHANGELOG.html in plugins ease readability for users. Please see following annotated example and see a concrete example in logstash-filter-date.

## 1.0.x                              <1>
 - change description                 <2>
 - tag: change description            <3>
 - tag1,tag2: change description      <4>
 - tag: Multi-line description        <5>
   must be indented and can use
   additional markdown syntax
                                      <6>
## 1.0.0                              <7>
[...]

1. Latest version is the first line of https://www.elastic.co/guide/en/logstash/current/CHANGELOG.html. Each version identifier should be a level-2 header using ##
2. One change description is described as a list item using a dash - aligned under the version identifier
3. One change can be tagged by a word and suffixed by :. Common tags are bugfix, feature, doc, test or internal.
4. One change can have multiple tags separated by a comma and suffixed by :
5. A multi-line change description must be properly indented
6. Please take care to separate versions with an empty line
7. Previous version identifier

Continuous Integration [_continuous_integration] Copy Section

Plugins are setup with automated continuous integration (CI) environments and there should be a corresponding badge on each Github page. If it’s missing, please contact the Logstash core team.

Every Pull Request opened automatically triggers a CI run. To conduct a manual run, comment “Jenkins, please test this.” on the Pull Request.

Versioning Plugins [_versioning_plugins] Copy Chapter

Logstash core and its plugins have separate product development lifecycles. Hence the versioning and release strategy for the core and plugins do not have to be aligned. In fact, this was one of our goals during the great separation of plugins work in Logstash 1.5.

At times, there will be changes in core API in Logstash, which will require mass update of plugins to reflect the changes in core. However, this does not happen frequently.

For plugins, we would like to adhere to a versioning and release strategy that can better inform our users, about any breaking changes to the Logstash configuration formats and functionality.

Plugin releases follows a three-placed numbering scheme X.Y.Z. where X denotes a major release version which may break compatibility with existing configuration or functionality. Y denotes releases which includes features which are backward compatible. Z denotes releases which includes bug fixes and patches.

Changing the version [_changing_the_version] Copy Section

Version can be changed in the Gemspec, which needs to be associated with a changelog entry. Following this, we can publish the gem to RubyGem.org manually. At this point only the core developers can publish a gem.

Labeling [_labeling] Copy Section

Labeling is a critical aspect of maintaining plugins. All issues in GitHub should be labeled correctly so it can:

• Provide good feedback to users/developers
• Help prioritize changes
• Be used in release notes

Most labels are self explanatory, but here’s a quick recap of few important labels:

• bug: Labels an issue as an unintentional defect
• needs details: If a the issue reporter has incomplete details, please ask them for more info and label as needs details.
• missing cla: Contributor License Agreement is missing and patch cannot be accepted without it
• adopt me: Ask for help from the community to take over this issue
• enhancement: New feature, not a bug fix
• needs tests: Patch has no tests, and cannot be accepted without unit/integration tests
• docs: Documentation related issue/PR

Logging [_logging] Copy Chapter

Although it’s important not to bog down performance with excessive logging, debug level logs can be immensely helpful when diagnosing and troubleshooting issues with Logstash. Please remember to liberally add debug logs wherever it makes sense as users will be forever gracious.

@logger.debug("Logstash loves debug logs!", :actions => actions)

Contributor License Agreement (CLA) Guidance [_contributor_license_agreement_cla_guidance] Copy Chapter

Why is a CLA required?
: We ask this of all Contributors in order to assure our users of the origin and continuing existence of the code. We are not asking Contributors to assign copyright to us, but to give us the right to distribute a Contributor’s code without restriction.

Please make sure the CLA is signed by every Contributor prior to reviewing PRs and commits.
: Contributors only need to sign the CLA once and should sign with the same email as used in Github. If a Contributor signs the CLA after a PR is submitted, they can refresh the automated CLA checker by pushing another comment on the PR after 5 minutes of signing.

Need Help? [_need_help] Copy Chapter

Ping @logstash-core on Github to get the attention of the Logstash core team.

Community Administration [_community_administration] Copy Chapter

The core team is there to support the plugin Maintainers and overall ecosystem.

Maintainers should propose Contributors to become a Maintainer.

Contributors and Maintainers should follow the Elastic Community Code of Conduct. The core team should block or ban "bad actors".

File: docs/extend/contribute-to-core.md Copy Chapter

mapped_pages:

• https://www.elastic.co/guide/en/logstash/current/contribute-to-core.html

Extending Logstash core [contribute-to-core]

We also welcome contributions and bug fixes to the Logstash core feature set.

Please read through our contribution guide, and the Logstash readme document.

File: docs/extend/contributing-patch-plugin.md Copy Chapter

mapped_pages:

• https://www.elastic.co/guide/en/logstash/current/contributing-patch-plugin.html

Contributing a patch to a Logstash plugin [contributing-patch-plugin]

This section discusses the information you need to know to successfully contribute a patch to a Logstash plugin.

Each plugin defines its own configuration options. These control the behavior of the plugin to some degree. Configuration option definitions commonly include:

• Data validation
• Default value
• Any required flags

Plugins are subclasses of a Logstash base class. A plugin’s base class defines common configuration and methods.

Input plugins [contrib-patch-input] Copy Chapter

Input plugins ingest data from an external source. Input plugins are always associated with a codec. An input plugin always has an associated codec plugin. Input and codec plugins operate in conjunction to create a Logstash event and add that event to the processing queue. An input codec is a subclass of the LogStash::Inputs::Base class.

Input API [input-api] Copy Section

#register() -> nil
: Required. This API sets up resources for the plugin, typically the connection to the external source.

#run(queue) -> nil
: Required. This API fetches or listens for source data, typically looping until stopped. Must handle errors inside the loop. Pushes any created events to the queue object specified in the method argument. Some inputs may receive batched data to minimize the external call overhead.

#stop() -> nil
: Optional. Stops external connections and cleans up.

Codec plugins [contrib-patch-codec] Copy Chapter

Codec plugins decode input data that has a specific structure, such as JSON input data. A codec plugin is a subclass of LogStash::Codecs::Base.

Codec API [codec-api] Copy Section

#register() -> nil
: Identical to the API of the same name for input plugins.

#decode(data){|event| block} -> nil
: Must be implemented. Used to create an Event from the raw data given in the method argument. Must handle errors. The caller must provide a Ruby block. The block is called with the created Event.

#encode(event) -> nil
: Required. Used to create a structured data object from the given Event. May handle errors. This method calls a block that was previously stored as @on_event with two arguments: the original event and the data object.

Filter plugins [contrib-patch-filter] Copy Chapter

A mechanism to change, mutate or merge one or more Events. A filter plugin is a subclass of the LogStash::Filters::Base class.

Filter API [filter-api] Copy Section

#register() -> nil
: Identical to the API of the same name for input plugins.

#filter(event) -> nil
: Required. May handle errors. Used to apply a mutation function to the given event.

Output plugins [contrib-patch-output] Copy Chapter

A mechanism to send an event to an external destination. This process may require serialization. An output plugin is a subclass of the LogStash::Outputs::Base class.

Output API [output-api] Copy Section

#register() -> nil
: Identical to the API of the same name for input plugins.

#receive(event) -> nil
: Required. Must handle errors. Used to prepare the given event for transmission to the external destination. Some outputs may buffer the prepared events to batch transmit to the destination.

Process [patch-process] Copy Chapter

A bug or feature is identified. An issue is created in the plugin repository. A patch is created and a pull request (PR) is submitted. After review and possible rework the PR is merged and the plugin is published.

The Community Maintainer Guide explains, in more detail, the process of getting a patch accepted, merged and published. The Community Maintainer Guide also details the roles that contributors and maintainers are expected to perform.

Testing methodologies [test-methods] Copy Chapter

Test driven development [tdd] Copy Section

Test driven development (TDD) describes a methodology for using tests to guide evolution of source code. For our purposes, we are use only a part of it. Before writing the fix, we create tests that illustrate the bug by failing. We stop when we have written enough code to make the tests pass and submit the fix and tests as a patch. It is not necessary to write the tests before the fix, but it is very easy to write a passing test afterwards that may not actually verify that the fault is really fixed especially if the fault can be triggered via multiple execution paths or varying input data.

RSpec framework [rspec] Copy Section

Logstash uses Rspec, a Ruby testing framework, to define and run the test suite. What follows is a summary of various sources.

2 require "logstash/devutils/rspec/spec_helper"
 3 require "logstash/plugin"
 4
 5 describe "outputs/riemann" do
 6   describe "#register" do
 7     let(:output) do
 8       LogStash::Plugin.lookup("output", "riemann").new(configuration)
 9     end
10
11     context "when no protocol is specified" do
12       let(:configuration) { Hash.new }
13
14       it "the method completes without error" do
15         expect {output.register}.not_to raise_error
16       end
17     end
18
19     context "when a bad protocol is specified" do
20       let(:configuration) { {"protocol" => "fake"} }
21
22       it "the method fails with error" do
23         expect {output.register}.to raise_error
24       end
25     end
26
27     context "when the tcp protocol is specified" do
28       let(:configuration) { {"protocol" => "tcp"} }
29
30       it "the method completes without error" do
31         expect {output.register}.not_to raise_error
32       end
33     end
34   end
35
36   describe "#receive" do
37     let(:output) do
38       LogStash::Plugin.lookup("output", "riemann").new(configuration)
39     end
40
41     context "when operating normally" do
42       let(:configuration) { Hash.new }
43       let(:event) do
44         data = {"message"=>"hello", "@version"=>"1",
45                 "@timestamp"=>"2015-06-03T23:34:54.076Z",
46                 "host"=>"vagrant-ubuntu-trusty-64"}
47         LogStash::Event.new(data)
48       end
49
50       before(:example) do
51         output.register
52       end
53
54       it "should accept the event" do
55         expect { output.receive event }.not_to raise_error
56       end
57     end
58   end
59 end

describe(string){block} -> nil
describe(Class){block} -> nil

With RSpec, we are always describing the plugin method behavior. The describe block is added in logical sections and can accept either an existing class name or a string. The string used in line 5 is the plugin name. Line 6 is the register method, line 36 is the receive method. It is a RSpec convention to prefix instance methods with one hash and class methods with one dot.

context(string){block} -> nil

In RSpec, context blocks define sections that group tests by a variation. The string should start with the word when and then detail the variation. See line 11. The tests in the content block should should only be for that variation.

let(symbol){block} -> nil

In RSpec, let blocks define resources for use in the test blocks. These resources are reinitialized for every test block. They are available as method calls inside the test block. Define let blocks in describe and context blocks, which scope the let block and any other nested blocks. You can use other let methods defined later within the let block body. See lines 7-9, which define the output resource and use the configuration method, defined with different variations in lines 12, 20 and 28.

before(symbol){block} -> nil - symbol is one of :suite, :context, :example, but :all and :each are synonyms for :suite and :example respectively.

In RSpec, before blocks are used to further set up any resources that would have been initialized in a let block. You cannot define let blocks inside before blocks.

You can also define after blocks, which are typically used to clean up any setup activity performed by a before block.

it(string){block} -> nil

In RSpec, it blocks set the expectations that verify the behavior of the tested code. The string should not start with it or should, but needs to express the outcome of the expectation. When put together the texts from the enclosing describe, context and it blocks should form a fairly readable sentence, as in lines 5, 6, 11 and 14:

outputs/riemann
#register when no protocol is specified the method completes without error

Readable code like this make the goals of tests easy to understand.

expect(object){block} -> nil

In RSpec, the expect method verifies a statement that compares an actual result to an expected result. The expect method is usually paired with a call to the to or not_to methods. Use the block form when expecting errors or observing for changes. The to or not_to methods require a matcher object that encapsulates the expected value. The argument form of the expect method encapsulates the actual value. When put together the whole line tests the actual against the expected value.

raise_error(error class|nil) -> matcher instance
be(object) -> matcher instance
eq(object) -> matcher instance
eql(object) -> matcher instance
  for more see http://www.relishapp.com/rspec/rspec-expectations/docs/built-in-matchers

In RSpec, a matcher is an object generated by the equivalent method call (be, eq) that will be used to evaluate the expected against the actual values.

Putting it all together [all-together] Copy Chapter

This example fixes an issue in the ZeroMQ output plugin. The issue does not require knowledge of ZeroMQ.

The activities in this example have the following prerequisites:

• A minimal knowledge of Git and Github. See the Github boot camp.
• A text editor.
• A JRuby runtime environment. The chruby tool manages Ruby versions.
• JRuby 1.7.22 or later.
• The bundler and rake gems installed.
• ZeroMQ installed.

1. In Github, fork the ZeroMQ output plugin repository.

2. On your local machine, clone the fork to a known folder such as logstash/.

3. Open the following files in a text editor:

   • logstash-output-zeromq/lib/logstash/outputs/zeromq.rb
   • logstash-output-zeromq/lib/logstash/util/zeromq.rb
   • logstash-output-zeromq/spec/outputs/zeromq_spec.rb

4. According to the issue, log output in server mode must indicate bound. Furthermore, the test file contains no tests.

   ::::{note}
   Line 21 of util/zeromq.rb reads @logger.info("0mq: #{server? ? 'connected' : 'bound'}", :address => address)
   ::::

5. In the text editor, require zeromq.rb for the file zeromq_spec.rb by adding the following lines:

   ruby Copy

   require "logstash/outputs/zeromq"
   require "logstash/devutils/rspec/spec_helper"

6. The desired error message should read:

   ruby Copy

   LogStash::Outputs::ZeroMQ when in server mode a 'bound' info line is logged

   To properly generate this message, add a describe block with the fully qualified class name as the argument, a context block, and an it block.

   ruby Copy

   describe LogStash::Outputs::ZeroMQ do
     context "when in server mode" do
       it "a 'bound' info line is logged" do
       end
     end
   end

7. To add the missing test, use an instance of the ZeroMQ output and a substitute logger. This example uses an RSpec feature called test doubles as the substitute logger.

   Add the following lines to zeromq_spec.rb, after describe LogStash::Outputs::ZeroMQ do and before context "when in server mode" do:

   ruby Copy

   let(:output) { described_class.new("mode" => "server", "topology" => "pushpull" }
     let(:tracer) { double("logger") }

8. Add the body to the it block. Add the following five lines after the line context "when in server mode" do:

   ruby Copy

   allow(tracer).to receive(:debug)<1>
         output.logger = logger<2>
         expect(tracer).to receive(:info).with("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})<3>
         output.register<4>
         output.do_close<5>

9. Allow the double to receive debug method calls.

10. Make the output use the test double.

11. Set an expectation on the test to receive an info method call.

12. Call register on the output.

13. Call do_close on the output so the test does not hang.

At the end of the modifications, the relevant code section reads:

require "logstash/outputs/zeromq"
require "logstash/devutils/rspec/spec_helper"

describe LogStash::Outputs::ZeroMQ do
  let(:output) { described_class.new("mode" => "server", "topology" => "pushpull") }
  let(:tracer) { double("logger") }

  context "when in server mode" do
    it "a ‘bound’ info line is logged" do
      allow(tracer).to receive(:debug)
      output.logger = tracer
      expect(tracer).to receive(:info).with("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})
      output.register
      output.do_close
    end
  end
end

To run this test:

1. Open a terminal window
2. Navigate to the cloned plugin folder
3. The first time you run the test, run the command bundle install
4. Run the command bundle exec rspec

Assuming all prerequisites were installed correctly, the test fails with output similar to:

Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true,
:elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}

LogStash::Outputs::ZeroMQ
  when in server mode
    a ‘bound’ info line is logged (FAILED - 1)

Failures:

  1) LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged
     Failure/Error: output.register
       Double "logger" received :info with unexpected arguments
         expected: ("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})
              got: ("0mq: connected", {:address=>"tcp://127.0.0.1:2120"})
     # ./lib/logstash/util/zeromq.rb:21:in `setup'
     # ./lib/logstash/outputs/zeromq.rb:92:in `register'
     # ./lib/logstash/outputs/zeromq.rb:91:in `register'
     # ./spec/outputs/zeromq_spec.rb:13:in `(root)'
     # /Users/guy/.gem/jruby/1.9.3/gems/rspec-wait-0.0.7/lib/rspec/wait.rb:46:in `(root)'

Finished in 0.133 seconds (files took 1.28 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/outputs/zeromq_spec.rb:10 # LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged

Randomized with seed 2568

To correct the error, open the util/zeromq.rb file in your text editor and swap the positions of the words connected and bound on line 21. Line 21 now reads:

@logger.info("0mq: #{server? ? 'bound' : 'connected'}", :address => address)

Run the test again with the bundle exec rspec command.

The test passes with output similar to:

Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}

LogStash::Outputs::ZeroMQ
  when in server mode
    a ‘bound’ info line is logged

Finished in 0.114 seconds (files took 1.22 seconds to load)
1 example, 0 failures

Randomized with seed 45887

Commit the changes to git and Github.

Your pull request is visible from the Pull Requests section of the original Github repository. The plugin maintainers review your work, suggest changes if necessary, and merge and publish a new version of the plugin.

File: docs/extend/create-logstash-plugins.md Copy Chapter

mapped_pages:

• https://www.elastic.co/guide/en/logstash/current/contributing-java-plugin.html

Create Logstash plugins [contributing-java-plugin]

Now you can write your own Java plugin for use with {{ls}}. We have provided instructions and GitHub examples to give you a head start.

Native support for Java plugins in {{ls}} consists of several components:

• Extensions to the Java execution engine to support running Java plugins in Logstash pipelines
• APIs for developing Java plugins. The APIs are in the co.elastic.logstash.api package. A Java plugin might break if it references classes or specific concrete implementations of API interfaces outside that package. The implementation of classes outside of the API package may change at any time.
• Tooling to automate the packaging and deployment of Java plugins in Logstash.

Process overview [_process_overview] Copy Chapter

Here are the steps:

1. Choose the type of plugin you want to create: input, codec, filter, or output.
2. Set up your environment.
3. Code the plugin.
4. Package and deploy the plugin.
5. Run Logstash with your new plugin.

Let’s get started [_lets_get_started] Copy Section

Here are the example repos:

• Input plugin example
• Codec plugin example
• Filter plugin example
• Output plugin example

Here are the instructions:

• How to write a Java input plugin
• How to write a Java codec plugin
• How to write a Java filter plugin
• How to write a Java output plugin

File: docs/extend/filter-new-plugin.md Copy Chapter

mapped_pages:

• https://www.elastic.co/guide/en/logstash/current/filter-new-plugin.html

How to write a Logstash filter plugin [filter-new-plugin]

To develop a new filter for Logstash, build a self-contained Ruby gem whose source code lives in its own GitHub repository. The Ruby gem can then be hosted and shared on RubyGems.org. You can use the example filter implementation as a starting point. (If you’re unfamiliar with Ruby, you can find an excellent quickstart guide at https://www.ruby-lang.org/en/documentation/quickstart/.)

Get started [_get_started_3] Copy Chapter

Let’s step through creating a filter plugin using the example filter plugin.

Create a GitHub repo for your new plugin [_create_a_github_repo_for_your_new_plugin_3] Copy Section

Each Logstash plugin lives in its own GitHub repository. To create a new repository for your plugin:

1. Log in to GitHub.

2. Click the Repositories tab. You’ll see a list of other repositories you’ve forked or contributed to.

3. Click the green New button in the upper right.

4. Specify the following settings for your new repo:

   • Repository name — a unique name of the form logstash-filter-pluginname.
   • Public or Private — your choice, but the repository must be Public if you want to submit it as an official plugin.
   • Initialize this repository with a README — enables you to immediately clone the repository to your computer.

5. Click Create Repository.

Use the plugin generator tool [_use_the_plugin_generator_tool_3] Copy Section

You can create your own Logstash plugin in seconds! The generate subcommand of bin/logstash-plugin creates the foundation for a new Logstash plugin with templatized files. It creates the correct directory structure, gemspec files, and dependencies so you can start adding custom code to process data with Logstash.

For more information, see Generating plugins

Copy the filter code [_copy_the_filter_code] Copy Section

Alternatively, you can use the examples repo we host on github.com

1. Clone your plugin. Replace GITUSERNAME with your github username, and MYPLUGINNAME with your plugin name.

   • git clone https://github.com/GITUSERNAME/logstash-``filter-MYPLUGINNAME.git

     • alternately, via ssh: git clone [email protected]:GITUSERNAME/logstash``-filter-MYPLUGINNAME.git

   • cd logstash-filter-MYPLUGINNAME

2. Clone the filter plugin example and copy it to your plugin branch.

   You don’t want to include the example .git directory or its contents, so delete it before you copy the example.

   • cd /tmp
   • git clone https://github.com/logstash-plugins/logstash``-filter-example.git
   • cd logstash-filter-example
   • rm -rf .git
   • cp -R * /path/to/logstash-filter-mypluginname/

3. Rename the following files to match the name of your plugin.

   • logstash-filter-example.gemspec

   • example.rb

   • example_spec.rb

     txt Copy

     cd /path/to/logstash-filter-mypluginname
     mv logstash-filter-example.gemspec logstash-filter-mypluginname.gemspec
     mv lib/logstash/filters/example.rb lib/logstash/filters/mypluginname.rb
     mv spec/filters/example_spec.rb spec/filters/mypluginname_spec.rb

Your file structure should look like this:

$ tree logstash-filter-mypluginname
├── Gemfile
├── LICENSE
├── README.md
├── Rakefile
├── lib
│   └── logstash
│       └── filters
│           └── mypluginname.rb
├── logstash-filter-mypluginname.gemspec
└── spec
    └── filters
        └── mypluginname_spec.rb

For more information about the Ruby gem file structure and an excellent walkthrough of the Ruby gem creation process, see http://timelessrepo.com/making-ruby-gems

See what your plugin looks like [_see_what_your_plugin_looks_like_3] Copy Section

Before we dive into the details, open up the plugin file in your favorite text editor and take a look.

require "logstash/filters/base"
require "logstash/namespace"

# Add any asciidoc formatted documentation here
# This example filter will replace the contents of the default
# message field with whatever you specify in the configuration.
#
# It is only intended to be used as an example.
class LogStash::Filters::Example < LogStash::Filters::Base

  # Setting the config_name here is required. This is how you
  # configure this filter from your Logstash config.
  #
  # filter {
  #   example { message => "My message..." }
  # }
  config_name "example"

  # Replace the message with this value.
  config :message, :validate => :string, :default => "Hello World!"


  public
  def register
    # Add instance variables
  end # def register

  public
  def filter(event)

    if @message
      # Replace the event message with our message as configured in the
      # config file.
      event.set("message", @message)
    end

    # filter_matched should go in the last line of our successful code
    filter_matched(event)
  end # def filter

end # class LogStash::Filters::Example

Coding filter plugins [_coding_filter_plugins] Copy Chapter

Now let’s take a line-by-line look at the example plugin.

`require` Statements [_require_statements_3] Copy Section

Logstash filter plugins require parent classes defined in logstash/filters/base and logstash/namespace:

require "logstash/filters/base"
require "logstash/namespace"

Of course, the plugin you build may depend on other code, or even gems. Just put them here along with these Logstash dependencies.

Plugin Body [_plugin_body_3] Copy Chapter

Let’s go through the various elements of the plugin itself.

`class` Declaration [_class_declaration_3] Copy Section

The filter plugin class should be a subclass of LogStash::Filters::Base:

class LogStash::Filters::Example < LogStash::Filters::Base

The class name should closely mirror the plugin name, for example:

LogStash::Filters::Example

`config_name` [_config_name_3] Copy Section

config_name "example"

This is the name your plugin will call inside the filter configuration block.

If you set config_name "example" in your plugin code, the corresponding Logstash configuration block would need to look like this:

Configuration Parameters [_configuration_parameters_3] Copy Chapter

config :variable_name, :validate => :variable_type, :default => "Default value", :required => boolean, :deprecated => boolean, :obsolete => string

The configuration, or config section allows you to define as many (or as few) parameters as are needed to enable Logstash to process events.

There are several configuration attributes:

• :validate - allows you to enforce passing a particular data type to Logstash for this configuration option, such as :string, :password, :boolean, :number, :array, :hash, :path (a file-system path), uri, :codec (since 1.2.0), :bytes. Note that this also works as a coercion in that if I specify "true" for boolean (even though technically a string), it will become a valid boolean in the config. This coercion works for the :number type as well where "1.2" becomes a float and "22" is an integer.
• :default - lets you specify a default value for a parameter
• :required - whether or not this parameter is mandatory (a Boolean true or
• :list - whether or not this value should be a list of values. Will typecheck the list members, and convert scalars to one element lists. Note that this mostly obviates the array type, though if you need lists of complex objects that will be more suitable. false)
• :deprecated - informational (also a Boolean true or false)
• :obsolete - used to declare that a given setting has been removed and is no longer functioning. The idea is to provide an informed upgrade path to users who are still using a now-removed setting.

Plugin Methods [_plugin_methods_3] Copy Chapter

Logstash filters must implement the register and filter methods.

`register` Method [_register_method_3] Copy Section

public
  def register
  end # def register

The Logstash register method is like an initialize method. It was originally created to enforce having super called, preventing headaches for newbies. (Note: It may go away in favor of initialize, in conjunction with some enforced testing to ensure super is called.)

public means the method can be called anywhere, not just within the class. This is the default behavior for methods in Ruby, but it is specified explicitly here anyway.

You can also assign instance variables here (variables prepended by @). Configuration variables are now in scope as instance variables, like @message

`filter` Method [_filter_method] Copy Section

public
  def filter(event)

    if @message
      # Replace the event message with our message as configured in the
      # config file.
      event.set("message", @message)
    end

  # filter_matched should go in the last line of our successful code
  filter_matched(event)
end # def filter

The plugin’s filter method is where the actual filtering work takes place! Inside the filter method you can refer to the event data using the Event object. Event is the main object that encapsulates data flow internally in Logstash and provides an API for the plugin developers to interact with the event’s content.

The filter method should also handle any event dependent configuration by explicitly calling the sprintf method available in Event class. For example:

field_foo = event.sprintf(field)

Note that configuration variables are now in scope as instance variables, like @message

filter_matched(event)

Calling the filter_matched method upon successful execution of the plugin will ensure that any fields or tags added through the Logstash configuration for this filter will be handled correctly. For example, any add_field, remove_field, add_tag and/or remove_tag actions will be performed at this time.

Event methods such as event.cancel are now available to control the workflow of the event being processed.

Building the Plugin [_building_the_plugin_3] Copy Chapter

At this point in the process you have coded your plugin and are ready to build a Ruby Gem from it. The following information will help you complete the process.

External dependencies [_external_dependencies_3] Copy Section

A require statement in Ruby is used to include necessary code. In some cases your plugin may require additional files. For example, the collectd plugin uses the types.db file provided by collectd. In the main directory of your plugin, a file called vendor.json is where these files are described.

The vendor.json file contains an array of JSON objects, each describing a file dependency. This example comes from the collectd codec plugin:

[{
        "sha1": "a90fe6cc53b76b7bdd56dc57950d90787cb9c96e",
        "url": "http://collectd.org/files/collectd-5.4.0.tar.gz",
        "files": [ "/src/types.db" ]
}]

• sha1 is the sha1 signature used to verify the integrity of the file referenced by url.
• url is the address from where Logstash will download the file.
• files is an optional array of files to extract from the downloaded file. Note that while tar archives can use absolute or relative paths, treat them as absolute in this array. If files is not present, all files will be uncompressed and extracted into the vendor directory.

Another example of the vendor.json file is the geoip filter

The process used to download these dependencies is to call rake vendor. This will be discussed further in the testing section of this document.

Another kind of external dependency is on jar files. This will be described in the "Add a gemspec file" section.

Deprecated features [_deprecated_features_3] Copy Section

As a plugin evolves, an option or feature may no longer serve the intended purpose, and the developer may want to deprecate its usage. Deprecation warns users about the option’s status, so they aren’t caught by surprise when it is removed in a later release.

{{ls}} 7.6 introduced a deprecation logger to make handling those situations easier. You can use the adapter to ensure that your plugin can use the deprecation logger while still supporting older versions of {{ls}}. See the readme for more information and for instructions on using the adapter.

Deprecations are noted in the logstash-deprecation.log file in the log directory.

Add a Gemfile [_add_a_gemfile_3] Copy Section

Gemfiles allow Ruby’s Bundler to maintain the dependencies for your plugin. Currently, all we’ll need is the Logstash gem, for testing, but if you require other gems, you should add them in here.

::::{tip}
See Bundler’s Gemfile page for more details.
::::

source 'https://rubygems.org'
gemspec
gem "logstash", :github => "elastic/logstash", :branch => "master"

Add a `gemspec` file [_add_a_gemspec_file_3] Copy Chapter

Gemspecs define the Ruby gem which will be built and contain your plugin.

::::{tip}
More information can be found on the Rubygems Specification page.
::::

Gem::Specification.new do |s|
  s.name = 'logstash-filter-example'
  s.version = '0.1.0'
  s.licenses = ['Apache License (2.0)']
  s.summary = "This filter does x, y, z in Logstash"
  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
  s.authors = ["Elastic"]
  s.email = '[email protected]'
  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
  s.require_paths = ["lib"]

  # Files
  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
   # Tests
  s.test_files = s.files.grep(%r{^(test|spec|features)/})

  # Special flag to let us know this is actually a logstash plugin
  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }

  # Gem dependencies
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
  s.add_development_dependency 'logstash-devutils'
end

It is appropriate to change these values to fit your plugin. In particular, s.name and s.summary should reflect your plugin’s name and behavior.

s.licenses and s.version are also important and will come into play when you are ready to publish your plugin.

Logstash and all its plugins are licensed under Apache License, version 2 ("ALv2"). If you make your plugin publicly available via RubyGems.org, please make sure to have this line in your gemspec:

• s.licenses = ['Apache License (2.0)']

The gem version, designated by s.version, helps track changes to plugins over time. You should use semver versioning strategy for version numbers.

Runtime and Development Dependencies [_runtime_and_development_dependencies_3] Copy Section

At the bottom of the gemspec file is a section with a comment: Gem dependencies. This is where any other needed gems must be mentioned. If a gem is necessary for your plugin to function, it is a runtime dependency. If a gem are only used for testing, then it would be a development dependency.

::::{note}
You can also have versioning requirements for your dependencies—​including other Logstash plugins:

# Gem dependencies
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
  s.add_development_dependency 'logstash-devutils'

This gemspec has a runtime dependency on the logstash-core-plugin-api and requires that it have a version number greater than or equal to version 1.60 and less than or equal to version 2.99.

::::

::::{important}
All plugins have a runtime dependency on the logstash-core-plugin-api gem, and a development dependency on logstash-devutils.
::::

Jar dependencies [_jar_dependencies_3] Copy Section

In some cases, such as the Elasticsearch output plugin, your code may depend on a jar file. In cases such as this, the dependency is added in the gemspec file in this manner:

# Jar dependencies
  s.requirements << "jar 'org.elasticsearch:elasticsearch', '5.0.0'"
  s.add_runtime_dependency 'jar-dependencies'

With these both defined, the install process will search for the required jar file at http://mvnrepository.com and download the specified version.

Document your plugin [_document_your_plugin_3] Copy Chapter

Documentation is an important part of your plugin. All plugin documentation is rendered and placed in the Logstash Reference and the Versioned plugin docs.

See Document your plugin for tips and guidelines.

Add Tests [_add_tests_3] Copy Chapter

Logstash loves tests. Lots of tests. If you’re using your new filter plugin in a production environment, you’ll want to have some tests to ensure you are not breaking any existing functionality.

::::{note}
A full exposition on RSpec is outside the scope of this document. Learn more about RSpec at http://rspec.info
::::

For help learning about tests and testing, look in the spec/filters/ directory of several other similar plugins.

Clone and test! [_clone_and_test_3] Copy Chapter

Now let’s start with a fresh clone of the plugin, build it and run the tests.

• Clone your plugin into a temporary location Replace GITUSERNAME with your github username, and MYPLUGINNAME with your plugin name.

  • git clone https://github.com/GITUSERNAME/logstash-``filter-MYPLUGINNAME.git

    • alternately, via ssh: git clone [email protected]:GITUSERNAME/logstash-``filter-MYPLUGINNAME.git

  • cd logstash-filter-MYPLUGINNAME

Then, you’ll need to install your plugins dependencies with bundler:

bundle install

::::{important}
If your plugin has an external file dependency described in vendor.json, you must download that dependency before running or testing. You can do this by running:

rake vendor

::::

And finally, run the tests:

bundle exec rspec

You should see a success message, which looks something like this:

Finished in 0.034 seconds
1 example, 0 failures

Hooray! You’re almost there! (Unless you saw failures…​ you should fix those first).

Building and Testing [_building_and_testing_3] Copy Chapter

Now you’re ready to build your (well-tested) plugin into a Ruby gem.

Build [_build_3] Copy Section

You already have all the necessary ingredients, so let’s go ahead and run the build command:

gem build logstash-filter-example.gemspec

That’s it! Your gem should be built and be in the same path with the name

logstash-filter-mypluginname-0.1.0.gem

The s.version number from your gemspec file will provide the gem version, in this case, 0.1.0.

Test installation [_test_installation_3] Copy Section

You should test install your plugin into a clean installation of Logstash. Download the latest version from the Logstash downloads page.

1. Untar and cd in to the directory:

   sh Copy

   curl -O https://download.elastic.co/logstash/logstash/logstash-9.0.0.tar.gz
   tar xzvf logstash-9.0.0.tar.gz
   cd logstash-9.0.0

2. Using the plugin tool, we can install the gem we just built.

   • Replace /my/logstash/plugins with the correct path to the gem for your environment, and 0.1.0 with the correct version number from the gemspec file.

     sh Copy

     bin/logstash-plugin install /my/logstash/plugins/logstash-filter-example/logstash-filter-example-0.1.0.gem

   • After running this, you should see feedback from Logstash that it was successfully installed:

     sh Copy

     validating /my/logstash/plugins/logstash-filter-example/logstash-filter-example-0.1.0.gem >= 0
     Valid logstash plugin. Continuing...
     Successfully installed 'logstash-filter-example' with version '0.1.0'

     ::::{tip}
     You can also use the Logstash plugin tool to determine which plugins are currently available:

     sh Copy

     bin/logstash-plugin list

     Depending on what you have installed, you might see a short or long list of plugins: inputs, codecs, filters and outputs.

     ::::

3. Now try running Logstash with a simple configuration passed in via the command-line, using the -e flag.

   ::::{note}
   Your results will depend on what your filter plugin is designed to do.
   ::::

bin/logstash -e 'input { stdin{} } filter { example {} } output {stdout { codec => rubydebug }}'

Test your filter by sending input through stdin and output (after filtering) through stdout with the rubydebug codec, which enhances readability.

In the case of the example filter plugin, any text you send will be replaced by the contents of the message configuration parameter, the default value being "Hello World!":

Testing 1, 2, 3
{
       "message" => "Hello World!",
      "@version" => "1",
    "@timestamp" => "2015-01-27T19:17:18.932Z",
          "host" => "cadenza"
}

Feel free to experiment and test this by changing the message parameter:

bin/logstash -e 'input { stdin{} } filter { example { message => "This is a new message!"} } output {stdout { codec => rubydebug }}'

Congratulations! You’ve built, deployed and successfully run a Logstash filter.

Submitting your plugin to [RubyGems.org](http://rubygems.org) and [logstash-plugins](https://github.com/logstash-plugins) [_submitting_your_plugin_to_rubygems_orghttprubygems_org_and_logstash_pluginshttpsgithub_comlogstash_plugins_3] Copy Chapter

Logstash uses RubyGems.org as its repository for all plugin artifacts. Once you have developed your new plugin, you can make it available to Logstash users by simply publishing it to RubyGems.org.

Licensing [_licensing_3] Copy Section

Logstash and all its plugins are licensed under Apache License, version 2 ("ALv2"). If you make your plugin publicly available via RubyGems.org, please make sure to have this line in your gemspec:

• s.licenses = ['Apache License (2.0)']

Publishing to [RubyGems.org](http://rubygems.org) [_publishing_to_rubygems_orghttprubygems_org_3] Copy Section

To begin, you’ll need an account on RubyGems.org

• Sign-up for a RubyGems account.

After creating an account, obtain an API key from RubyGems.org. By default, RubyGems uses the file ~/.gem/credentials to store your API key. These credentials will be used to publish the gem. Replace username and password with the credentials you created at RubyGems.org:

curl -u username:password https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
chmod 0600 ~/.gem/credentials

Before proceeding, make sure you have the right version in your gemspec file and commit your changes.

• s.version = '0.1.0'

To publish version 0.1.0 of your new logstash gem:

bundle install
bundle exec rake vendor
bundle exec rspec
bundle exec rake publish_gem

::::{note}
Executing rake publish_gem:

1. Reads the version from the gemspec file (s.version = '0.1.0')
2. Checks in your local repository if a tag exists for that version. If the tag already exists, it aborts the process. Otherwise, it creates a new version tag in your local repository.
3. Builds the gem
4. Publishes the gem to RubyGems.org

::::

That’s it! Your plugin is published! Logstash users can now install your plugin by running:

bin/logstash-plugin install logstash-filter-mypluginname

Contributing your source code to [logstash-plugins](https://github.com/logstash-plugins) [_contributing_your_source_code_to_logstash_pluginshttpsgithub_comlogstash_plugins_3] Copy Chapter

It is not required to contribute your source code to logstash-plugins github organization, but we always welcome new plugins!

Benefits [_benefits_3] Copy Section

Some of the many benefits of having your plugin in the logstash-plugins repository are:

• Discovery. Your plugin will appear in the Logstash Reference, where Logstash users look first for plugins and documentation.
• Documentation. Your plugin documentation will automatically be added to the Logstash Reference.
• Testing. With our testing infrastructure, your plugin will be continuously tested against current and future releases of Logstash. As a result, users will have the assurance that if incompatibilities arise, they will be quickly discovered and corrected.

Acceptance Guidelines [_acceptance_guidelines_3] Copy Section

• Code Review. Your plugin must be reviewed by members of the community for coherence, quality, readability, stability and security.
• Tests. Your plugin must contain tests to be accepted. These tests are also subject to code review for scope and completeness. It’s ok if you don’t know how to write tests — we will guide you. We are working on publishing a guide to creating tests for Logstash which will make it easier. In the meantime, you can refer to http://betterspecs.org/ for examples.

To begin migrating your plugin to logstash-plugins, simply create a new issue in the Logstash repository. When the acceptance guidelines are completed, we will facilitate the move to the logstash-plugins organization using the recommended github process.