vector Documentation

A high-performance observability data pipeline.

RAW Doc

File: README.md



What is Vector?

Vector is a high-performance, end-to-end (agent & aggregator) observability data
pipeline that puts you in control of your observability data.
Collect, transform, and route
all your logs and metrics to any vendors you want today and any other
vendors you may want tomorrow. Vector enables dramatic cost reduction, novel
data enrichment, and data security where you need it, not where it is most
convenient for your vendors. Additionally, it is open source and up to 10x
faster than every alternative in the space.

To get started, follow our quickstart guide or install
Vector
.

Vector is maintained by Datadog's Community Open Source Engineering team.

Principles

  • Reliable - Built in Rust, Vector's primary design goal is reliability.
  • End-to-end - Deploys as an agent or aggregator. Vector is a complete platform.
  • Unified - Logs, metrics (beta), and traces (coming soon). One tool for all of your data.

Use cases

  • Reduce total observability costs.
  • Transition vendors without disrupting workflows.
  • Enhance data quality and improve insights.
  • Consolidate agents and eliminate agent fatigue.
  • Improve overall observability performance and reliability.

Community

  • Vector is relied on by startups and enterprises like Atlassian, T-Mobile,
    Comcast, Zendesk, Discord, Fastly, CVS, Trivago,
    Tuple, Douban, Visa, Mambu, Blockfi, Claranet,
    Instacart, Forcepoint, and many more.
  • Vector is downloaded over 100,000 times per day.
  • Vector's largest user processes over 500TB daily.
  • Vector has over 500 contributors and growing.

Documentation

All user documentation is available at vector.dev/docs.

Other Resources:

Comparisons

Performance

The following performance tests demonstrate baseline performance between
common protocols with the exception of the Regex Parsing test.

Test Vector Filebeat FluentBit FluentD Logstash SplunkUF SplunkHF
TCP to Blackhole 86mib/s n/a 64.4mib/s 27.7mib/s 40.6mib/s n/a n/a
File to TCP 76.7mib/s 7.8mib/s 35mib/s 26.1mib/s 3.1mib/s 40.1mib/s 39mib/s
Regex Parsing 13.2mib/s n/a 20.5mib/s 2.6mib/s 4.6mib/s n/a 7.8mib/s
TCP to HTTP 26.7mib/s n/a 19.6mib/s <1mib/s 2.7mib/s n/a n/a
TCP to TCP 69.9mib/s 5mib/s 67.1mib/s 3.9mib/s 10mib/s 70.4mib/s 7.6mib/s

To learn more about our performance tests, please see the Vector test harness.

Correctness

The following correctness tests are not exhaustive, but they demonstrate
fundamental differences in quality and attention to detail:

Test Vector Filebeat FluentBit FluentD Logstash Splunk UF Splunk HF
Disk Buffer Persistence โœ“ โœ“ โš  โœ“ โœ“
File Rotate (create) โœ“ โœ“ โœ“ โœ“ โœ“ โœ“ โœ“
File Rotate (copytruncate) โœ“ โœ“ โœ“
File Truncation โœ“ โœ“ โœ“ โœ“ โœ“ โœ“ โœ“
Process (SIGHUP) โœ“ โš  โœ“ โœ“
JSON (wrapped) โœ“ โœ“ โœ“ โœ“ โœ“ โœ“ โœ“

To learn more about our correctness tests, please see the Vector test harness.

Features

Vector is an end-to-end, unified, open data platform.

Vector Beats Fluentbit Fluentd Logstash Splunk UF Splunk HF Telegraf
End-to-end โœ“ โœ“
Agent โœ“ โœ“ โœ“ โœ“ โœ“
Aggregator โœ“ โœ“ โœ“ โœ“ โœ“
Unified โœ“ โœ“
Logs โœ“ โœ“ โœ“ โœ“ โœ“ โœ“ โœ“ โœ“
Metrics โœ“ โš  โš  โš  โš  โš  โš  โœ“
Traces ๐Ÿšง
Open โœ“ โœ“ โœ“ โœ“
Open-source โœ“ โœ“ โœ“ โœ“ โœ“ โœ“
Vendor-neutral โœ“ โœ“ โœ“ โœ“
Reliability โœ“
Memory-safe โœ“ โœ“
Delivery guarantees โœ“ โœ“ โœ“
Multi-core โœ“ โœ“ โœ“ โœ“ โœ“ โœ“ โœ“ โœ“

โš  = Not interoperable, metrics are represented as structured logs



File: docs/README.md

Vector Internal Documentation

This folder contains internal documentation for Vector contributors. If
you're a Vector user, please visit https://vector.dev/docs, which is powered by
the website directory.

Getting started

Whether you're a Vector team member, or an outside contributor, this is the best
place to start. This folder contains internal documentation to help with the
development of Vector and ensuring your change gets approved in a timely manner.

  1. CONTRIBUTING.md - Start here, contributor basics and workflow
  2. DEVELOPING.md - Everything necessary to develop
  3. DOCUMENTING.md - Preparing your change for Vector users

Vector team members

Vector team members have additional responsibilities beyond outside
contributors:

Project policies

Vector's policies are located in the root directory: