Prometheus Config Generator
Build production-ready prometheus.yml with scrape targets, alerting rules, recording rules, and service discovery.
Prometheus Configuration
Global Configuration
Alertmanager
Rule Files
Scrape Jobs
Alerting Rules
prometheus.yml
Syntax
Security
Best Practices
Overview
The Prometheus Config Generator helps you build production-ready prometheus.yml configuration files with scrape targets, alerting rules, recording rules, remote write/read, and service discovery for Kubernetes, Docker, Consul, and cloud environments.
Prometheus uses a pull-based scraping model that actively connects to HTTP endpoints to collect time-series metrics. It supports multiple service discovery mechanisms (Kubernetes, Consul, EC2, Docker, DNS, file-based SD) and provides powerful relabeling for dynamic label manipulation. This generator produces syntactically correct prometheus.yml files with all components fully configured.
How It Works
1. Configure Global Settings: Set scrape_interval (how often to scrape targets), evaluation_interval (how often to evaluate rules), scrape_timeout, and external_labels for all scrape jobs.
2. Define Scrape Jobs: Create scrape_configs entries for each target type — static endpoints, Kubernetes pods/nodes/services, Consul services, EC2 instances, Docker containers, DNS records, or file-based service discovery.
3. Add Relabeling: Use relabel_configs to dynamically modify labels before scraping (filter targets, extract instance names, add metadata). Use metric_relabel_configs to modify labels on scraped metrics.
4. Configure Rules: Define alerting_rules for conditions that trigger alerts (sent to Alertmanager) and recording_rules for precomputing expensive PromQL expressions.
5. Enable Remote Storage: Configure remote_write to send samples to long-term storage (Thanos, Cortex, Mimir) and remote_read to query historical data from external backends.
Best Practices
- Set scrape_interval to 15s for most production workloads; use 5-10s only for critical high-resolution metrics.
- Always set evaluation_interval to a value greater than or equal to scrape_interval for reliable alerting.
- Use recording rules for frequently evaluated expensive PromQL expressions to reduce query-time load.
- Configure scrape_timeout to be less than scrape_interval to prevent overlapping scrapes.
- Use relabel_configs with keep/drop actions to filter targets and reduce cardinality.
- Set external_labels on all Prometheus instances in HA setups for proper deduplication in Thanos/Cortex.
- Use metric_relabel_configs to drop high-cardinality labels and reduce storage costs.
- Always set rule_files paths to valid locations accessible by the Prometheus process.
Common Mistakes
- Setting scrape_interval below 10s without justification, causing high storage and network overhead.
- Forgetting to configure rule_files, which prevents alerting and recording rules from being evaluated.
- Using relabel_configs and metric_relabel_configs interchangeably — relabel_configs apply before scrape, metric_relabel_configs after.
- Not setting scrape_timeout, causing Prometheus to use the default 10s which may be too short for slow endpoints.
- Configuring remote_write without proper queue_config, leading to data loss under high load.
- Missing external_labels in HA setups, causing deduplication failures in Thanos or Cortex.
Security Recommendations
- Enable TLS on the Prometheus web endpoint (web.config.file) for encrypted access to the UI and API.
- Use basic_auth or bearer_token on scrape endpoints that expose sensitive metrics.
- Restrict network access to port 9090 using firewalls or security groups.
- Run Prometheus as a non-root user with minimal filesystem permissions.
- Use relabel_configs with drop action to exclude sensitive targets from scraping.
- Avoid exposing Prometheus directly to the internet — use a reverse proxy with authentication.
Production Tips
- Deploy Prometheus with persistent storage (PVC in Kubernetes) to survive restarts without data loss.
- Use the Thanos sidecar pattern for long-term storage and global query view across multiple Prometheus instances.
- Monitor Prometheus itself using the prometheus_target_interval_length_seconds and up metrics.
- Configure Alertmanager in a cluster with --cluster.peers for high availability and deduplication.
- Use kube-prometheus-stack Helm chart for a production-ready Kubernetes monitoring setup with pre-built dashboards.
- Set retention flags (--storage.tsdb.retention.time, --storage.tsdb.retention.size) to manage disk usage.
- Use file_sd_configs for dynamic target management in non-Kubernetes environments.
Frequently Asked Questions
What is Prometheus and how does it work?
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It collects metrics from configured endpoints at set intervals, evaluates rule expressions, displays results, and can trigger alerts when conditions are met. Unlike agent-based monitoring, Prometheus uses a pull model — it scrapes HTTP endpoints to gather time-series data.
What is the scrape model in Prometheus?
Prometheus uses a pull-based scraping model where it actively connects to HTTP endpoints (targets) to collect metrics at regular intervals. Each scrape target exposes metrics in a simple text-based format at a metrics endpoint (default: /metrics). This model eliminates the need for agents and makes it easy to detect when a target is down.
How does service discovery work in Prometheus?
Prometheus supports multiple service discovery mechanisms including Kubernetes (pods, nodes, services, endpoints), Consul, EC2, Docker, DNS, and file-based SD. These mechanisms automatically discover and update scrape targets without manual configuration. For example, kubernetes_sd_configs can monitor all pods in a namespace automatically.
What are relabel_configs in Prometheus?
Relabel_configs allow you to dynamically modify labels on metrics before they are stored or scraped. They use source_labels, regex matching, target_label, and replacement to transform labels. Common uses include filtering targets with keep/drop, extracting instance names with replace, and generating labels with labelmap.
What are recording rules in Prometheus?
Recording rules allow you to precompute frequently used or expensive PromQL expressions and save the result as a new time series. They are defined in rule_files and evaluated at the evaluation_interval. Recording rules improve dashboard performance by reducing query-time computation and are ideal for aggregations that run frequently.
What are alerting rules in Prometheus?
Alerting rules define conditions that trigger alerts sent to Alertmanager. Each rule has an alert name, a PromQL expression (expr), a duration (for) that specifies how long the condition must be true, and optional labels/annotations. When the condition is met for the specified duration, Prometheus sends the alert to configured Alertmanager instances.
What is remote write and remote read in Prometheus?
Remote write allows Prometheus to send scraped samples to an external storage system (Thanos, Cortex, Mimir, VictoriaMetrics). Remote read allows Prometheus to query data from an external storage backend as if it were local. These features enable long-term storage, horizontal scaling, and global query views across multiple Prometheus instances.
How do I achieve high availability with Prometheus?
Run two or more identical Prometheus instances scraping the same targets. Use Alertmanager in a cluster with gossip protocol for deduplication. For long-term storage and global view, use Thanos, Cortex, or Mimir with remote_write. Ensure consistent external_labels across HA pairs to enable proper deduplication.
What is the recommended scrape interval for Prometheus?
15 seconds is the standard for most production environments. Use 5-10 seconds for high-resolution metrics on critical services. Use 30-60 seconds for cost savings on low-priority targets. Always ensure scrape_interval is less than evaluation_interval for alerting rules to evaluate correctly.
How do I secure Prometheus in production?
Enable TLS for the Prometheus web UI and API endpoints. Use basic_auth or bearer tokens on scrape endpoints that require authentication. Restrict network access to the Prometheus port (9090). Use relabel_configs with keep/drop to filter sensitive targets. Run Prometheus as a non-root user with minimal filesystem permissions.
How do I troubleshoot Prometheus scrape failures?
Check the /targets page to see scrape status and error messages. Common issues include network connectivity, firewall rules blocking the scrape port, incorrect metrics_path, TLS certificate validation failures, and DNS resolution problems. Use the up metric to monitor scrape health and the prometheus_target_sync_length metric for discovery issues.
Can I use Prometheus to monitor Kubernetes?
Yes. Prometheus has native Kubernetes service discovery (kubernetes_sd_configs) that automatically discovers pods, nodes, services, and endpoints. Use the kube-prometheus-stack Helm chart for a production-ready deployment with Prometheus Operator, Alertmanager, Grafana, and pre-built dashboards and alerts.