Repository: elastic/logstash
Stars: 14836
CLAUDE.md
See AGENTS.md.
README.md
Logstash
Logstash is part of the Elastic Stack along with Beats, Elasticsearch and Kibana. Logstash is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.). Logstash has over 200 plugins, and you can write your own very easily as well.
For more info, see <https://www.elastic.co/products/logstash>
Documentation and Getting Started
You can find the documentation and getting started guides for Logstash
on the elastic.co site
For information about building the documentation, see the README in https://github.com/elastic/docs
Downloads
You can download officially released Logstash binaries, as well as debian/rpm packages for the
supported platforms, from downloads page.
Need Help?
- Logstash Forum
- Logstash Documentation
- Logstash Product Information
- Elastic Support
Logstash Plugins
Logstash plugins are hosted in separate repositories under the logstash-plugins github organization. Each plugin is a self-contained Ruby gem which gets published to RubyGems.org.
Writing your own Plugin
Logstash is known for its extensibility. There are hundreds of plugins for Logstash and you can write your own very easily! For more info on developing and testing these plugins, please see the working with plugins section
Plugin Issues and Pull Requests
Please open new issues and pull requests for plugins under its own repository
For example, if you have to report an issue/enhancement for the Elasticsearch output, please do so here.
Logstash core will continue to exist under this repository and all related issues and pull requests can be submitted here.
Developing Logstash Core
Prerequisites
* Install JDK version 21. Make sure to set the JAVA_HOME environment variable to the path to your JDK installation directory. For example set JAVA_HOME=<JDK_PATH>
* Install JRuby 10.0.5.0 It is recommended to use a Ruby version manager such as RVM or rbenv.
* Install rake and bundler tool using gem install rake and gem install bundler respectively.
RVM install (optional)
If you prefer to use rvm (ruby version manager) to manage Ruby versions on your machine, follow these directions. In the Logstash folder:
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash -s stable --ruby=$(cat .ruby-version)Check Ruby version
Before you proceed, please check your ruby version by:
$ ruby -vThe printed version should be the same as in the .ruby-version file.
Building Logstash
The Logstash project includes the source code for all of Logstash, including the Elastic-Licensed X-Pack features and functions; to run Logstash from source using only the OSS-licensed code, export the OSS environment variable with a value of true:
`` * Install development dependencies To verify your environment, run the following to start Logstash and send your first event: Drip is a tool that solves the slow JVM startup problem while developing Logstash. The drip script is intended to be a drop-in replacement for the java command. We recommend using drip during development, in particular for running tests. Using drip, the first invocation of a command will not be faster but the subsequent commands will be swift. To tell logstash to use drip, set the environment variable Example (but see the Testing section below before running rspec for the first time): JAVACMD= Caveats Drip does not work with STDIN. You cannot use drip for running configs which use the stdin plugin. To build the Logstash Reference (open source content only) on your local logstash - contains main docs about core features logstash-docs - contains generated plugin docs docs - contains doc build files Make sure you have the same branch checked out in Run the doc build script from within the Most of the unit tests in Logstash are written using rspec for the Ruby parts. For the Java parts, we use junit. For testing you can use the test 1- To run the core tests you can use the Gradle task: ./gradlew test or use the bin/rspec Note that before running the ./gradlew bootstrap 2- To run the subset of tests covering the Java codebase only run: ./gradlew javaTests 3- To execute the complete test-suite including the integration tests run: ./gradlew check 4- To execute a single Ruby test run: SPEC_OPTS="-fd -P logstash-core/spec/logstash/api/commands/default_metadata_spec.rb" ./gradlew :logstash-core:rubyTests --tests org.logstash.RSpecTests 5- To execute single spec for integration test, run: ./gradlew integrationTests -PrubyIntegrationSpecs=specs/slowlog_spec.rb Sometimes you might find a change to a piece of Logstash code causes a test to hang. These can be hard to debug. If you set To run the tests of all currently installed plugins: rake test:plugins You can install the default set of plugins included in the logstash package: rake test:install-default --- bin/logstash-plugin install --development Built artifacts will be placed in the You can build a Logstash snapshot package as tarball or zip file sh
export OSS=true* Set up the location of the source code to build
export LOGSTASH_SOURCE=1
export LOGSTASH_PATH=/YOUR/LOGSTASH/DIRECTORY#### Install dependencies with gradle (recommended)[^1]
./gradlew installDevelopmentGems* Install default plugins and other dependencies
./gradlew installDefaultGemsVerify the installation
bin/logstash -e 'input { stdin { } } output { stdout {} }'This should start Logstash with stdin input waiting for you to enter an event
hello world
2016-11-11T01:22:14.405+0000 0.0.0.0 hello worldAdvanced: Drip Launcher JAVACMD=which drip .which drip bin/rspeclogstashBuilding Logstash Documentation
machine, clone the following repos: and logstash-docs.master
Check out in the docs repo.docs repo. For example:
./build_docs.pl --doc ../logstash/docs/index.asciidoc --chunk=1 -openrakeTesting
tasks and the bin/rspec command, see instructions below:rspecCore tests
tool to run all tests or run a specific test:rspec
bin/rspec spec/foo/bar_spec.rb command for the first time you need to set up the RSpec test dependencies by running:LS_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" you can connect to a running Logstash with your IDEs debugger which can be a great way of finding the issue.bin/logstash-plugin install ...Plugins tests
Note that if a plugin is installed using the plugin manager do not forget to also install the plugins development dependencies using the following command after the plugin installation:LS_HOME/buildBuilding Artifacts
directory, and will create the directory if it is not already present.
./gradlew assembleTarDistribution
./gradlew assembleZipDistributionOSS-only artifacts can similarly be built with their own gradle tasks:
./gradlew assembleOssTarDistribution
./gradlew assembleOssZipDistribution
You can also build .rpm and .deb, but the fpm tool is required.rake artifact:rpm
rake artifact:deb
and:rake artifact:rpm_oss
rake artifact:deb_oss
custom.jruby.pathUsing a Custom JRuby Distribution
If you want the build to use a custom JRuby you can do so by setting a path to a custom
JRuby distribution's source root via theGradle property.E.g.
./gradlew clean test -Pcustom.jruby.path="/path/to/jruby"
gradleProject Principles
* Community: If a newbie has a bad time, it's a bug.
* Software: Make it work, then make it right, then make it fast.
* Technology: If it doesn't do a thing today, we can make it do it tomorrow.Contributing
All contributions are welcome: ideas, patches, documentation, bug reports,
complaints, and even something you drew up on a napkin.Programming is not a required skill. Whatever you've seen about open source and
maintainers or community members saying "send patches or die" - you will not
see that here.It is more important that you are able to contribute.
For more information about contributing, see the
CONTRIBUTING file.Footnotes
[^1]: <details><summary>Use bundle instead of gradle to install dependencies</summary>
#### Alternatively, instead of using
you can also usebundle:* Install development dependencies
bundle config set --local path vendor/bundle
bundle install
* Bootstrap the environment:rake bootstrap
* You can then usebin/logstashto start Logstash, but there are no plugins installed. To install default plugins, you can run:
rake plugin:install-default
``
This will install the 80+ default plugins which makes Logstash ready to connect to multiple data sources, perform transformations and send the results to Elasticsearch and other destinations.
</details>