ConfigGenerator

Vector Config Generator

Build production-ready Vector pipelines with Sources, Transforms, Sinks, and VRL expressions.

Vector Pipeline

Global Options

Sources

Transforms

Sinks

vector.toml
1data_dir = "/var/lib/vector"
2

3[log_schema]
4timestamp_key = "timestamp"
5level_key = "level"
6message_key = "message"
7service_key = "service"
8

9[sources.app_logs]
10type = "file"
11include = ["/var/log/app/*.log"]
12read_from = "beginning"
13

14[transforms.parse_json]
15type = "remap"
16inputs = ["app_logs"]
17source = '''
18. = parse_json!(.message)
19.timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%dT%H:%M:%S%.fZ")
20.environment = "production"
21'''
22

23[sinks.console_output]
24type = "console"
25inputs = ["parse_json"]
26target = "stdout"
27[sinks.console_output.encoding]
28codec = "json"
Syntax
Security
Best Practices

Overview

The Vector Config Generator helps you build production-ready Vector pipeline configurations in TOML format. Vector is a high-performance observability data router written in Rust, capable of collecting, transforming, and routing logs, metrics, and traces with sub-millisecond latency.

Unlike legacy tools like Fluentd or Logstash, Vector operates as a unified data pipeline for all telemetry signals. Its Vector Remap Language (VRL) provides a safety-focused transformation language with compile-time error detection, deterministic error handling, and zero runtime panics. This generator produces syntactically correct vector.toml files with Sources, Transforms, and Sinks fully configured.

How It Works

1. Define Sources: Vector sources ingest data from various origins — files (file), stdin, exec commands, journald, Kafka topics, Docker logs, Kubernetes logs, S3 buckets, GCP Pub/Sub, HTTP endpoints, syslog, TCP/UDP sockets, and Vector's own internal metrics.

2. Add Transforms: Process and enrich data in-flight using transforms. Use remap for VRL-based field mapping, filter to drop unwanted events, route for conditional splitting, reduce for aggregation, dedupe to remove duplicates, sample for rate limiting, geoip for IP enrichment, and various parsers (parse_json, parse_regex).

3. Configure Sinks: Route processed data to destinations — Elasticsearch, Loki, Kafka, S3, Datadog, Splunk, Prometheus, ClickHouse, HTTP endpoints, Google Cloud Storage, AWS CloudWatch Logs, or chain to another Vector instance. Each sink supports encoding, batching, and healthchecks.

4. Tune Global Options: Set data_dir for persistent state and log_schema for default field names across the pipeline.

Best Practices

  • Always set a dedicated data_dir for Vector's internal state to survive restarts without data loss.
  • Use VRL's try() and assert!() patterns to validate data shapes at compile time rather than runtime.
  • Configure healthcheck.enabled = true on all sinks to detect unreachable backends early.
  • Use the Vector agent/aggregator split pattern: lightweight agents on each node forward to centralized aggregators.
  • Enable batch configuration (batch.max_events, batch.timeout_secs) on sinks to optimize throughput.

Common Mistakes

  • Forgetting to set data_dir, which causes Vector to fail when sinks need persistent buffer state.
  • Using .message as a raw string instead of parsing JSON with parse_json!(), leading to unstructured data in downstream sinks.
  • Not configuring acknowledgments, which can cause data loss when the downstream sink is temporarily unavailable.
  • Over-configuring transform inputs without considering the DAG execution order, creating circular dependencies.

Security Recommendations

  • Use environment variables for sensitive values like API keys and passwords — Vector supports ${ENV_VAR} syntax in TOML.
  • Enable TLS on all network-facing sources (TCP, HTTP, syslog) and sinks to protect data in transit.
  • Run Vector as a non-root user with minimal file system permissions on the data_dir directory.
  • Restrict HTTP source endpoints to internal networks or use authentication plugins for public-facing endpoints.

Production Tips

  • Deploy Vector as a DaemonSet in Kubernetes with the kubernetes_logs source for automatic pod log collection.
  • Use the aggregator pattern with Vector sources to fan-in logs from multiple agents before routing to backends.
  • Monitor Vector's own health via the internal_metrics source exposed through a Prometheus sink.
  • Tune buffer.max_size and buffer.type (memory vs disk) based on your throughput requirements and acceptable latency.
  • Use VRL's redact!() function to sanitize sensitive fields like passwords and tokens before shipping logs.

Frequently Asked Questions

What is Vector and how does it compare to Fluentd or Logstash?
Vector is a high-performance observability data router written in Rust. Unlike Fluentd (Ruby) or Logstash (JRuby), Vector offers sub-millisecond latency, zero-copy architecture, and a safety-focused transformation language (VRL). It handles logs, metrics, and traces in a single binary with significantly lower resource consumption.
What is VRL (Vector Remap Language)?
VRL is Vector's domain-specific language for transforming observability data. It features compile-time error detection, deterministic error handling with try() and assert!(), and operators for parsing, merging, truncating, and enriching log events. VRL prevents runtime panics by catching all error paths at configuration time.
What configuration format does Vector use?
Vector uses TOML (Tom's Obvious Minimal Language) for configuration. The config file defines a data_dir, Sources (inputs), Transforms (processors), and Sinks (outputs) as named components in a pipeline DAG.
What are Vector's main source types?
Vector supports file, stdin, exec, journald, kafka, demo_logs, docker_logs, aws_s3, gcp_pubsub, http, syslog, tcp, udp, and vector (internal metrics) sources. Each source has specific configuration options for connection details, authentication, and data format.
How do I deploy Vector in Kubernetes?
Deploy Vector as a DaemonSet using the kubernetes_logs source for automatic pod log collection, or as a Deployment using the aggregator pattern with Vector sources for centralized log processing. Helm charts are available from the Vector project for both patterns.
Can Vector handle high throughput without data loss?
Yes. Vector uses disk-based buffers, acknowledgments, and end-to-end flow control. Configure buffer.type = 'disk' with adequate buffer.max_size, enable acknowledgments on sinks, and use the retry mechanism to handle transient failures without dropping events.

Related Generators

Fluentd Config Generator

Generate Fluentd .conf files for centralized logging pipelines.

Docker Compose Generator

Deploy Vector alongside your applications in a Docker Compose stack.