Guide to the Prometheus Blackbox Exporter

Prometheus is a popular solution for monitoring modern application and infrastructure stacks. A key reason Prometheus is so popular — and versatile — is the availability of Prometheus Exporters. In simple terms, Prometheus Exporters are utilities that run alongside the application environment that a Prometheus instance monitors. The Prometheus instance scrapes the exporter for crucial application and infrastructure-level metrics.

There are multiple different exporters, and here we’ll take a deep dive on Prometheus Blackbox exporter. We’ll also provide an overview of four other exporters you can use.

Why do you need exporters?

Of course, you shouldn’t just use a tool because it’s popular. Tools should solve a problem you have. So, why might you need an exporter like Prometheus Blackbox Exporter?

Exporters are necessary since they are the fundamental Prometheus functionality that collects metrics. Prometheus scrapes these metrics to provide you with application and infrastructure visibility. The diagram below shows how all the pieces, including exporters, fit together in an example architecture.

Prometheus Exporters: A Bird’s Eye View

Now that we know why exporters are useful, let’s look at some of our most popular options and their value.

Blackbox Exporter	Metrics for “blackbox” probing of endpoints over HTTP, HTTPS, DNS, TCP, and ICMP
Kube-state Metrics Exporter	Kubernetes (K8s) add-on agent which provides metrics about various Kubernetes objects, such as deployments, nodes, and pods
Node Exporter	Exposes hardware and OS level metrics for *NIX kernels
Elasticsearch Exporter	Connects to local or remote Elasticsearch (ES) instance to provide various Elasticsearch metrics in Prometheus format
Redis Exporter	Connects to local or remote Redis instance to provide various Redis metrics in Prometheus format

De-mystifying the Prometheus Blackbox Exporter

Now, let’s take a closer look at these exporters, beginning with one of the most popular: Prometheus Blackbox Exporter.

Blackbox Exporter is used for endpoint monitoring and can help generate meaningful uptime and availability metrics. Blackbox Exporter can probe endpoints over HTTP, HTTPS, DNS, TCP, and ICMP.

While most exporters accept static configurations and expose metrics accordingly, Blackbox Exporter works a little differently. Inside the Blackbox Exporter config, you define modules. Then, Prometheus can query each of those modules for a set of specific targets. As a response to that query, Blackbox Exporter generates metrics for the queried endpoint.

Prometheus Blackbox Exporter: A real-world example

To better understand how the Blackbox Exporter works, let’s walk through a practical example.

Here is a sample module defined in the Blackbox Exporter config:

modules: 
  http_2xx: 
    http: 
      fail_if_not_ssl: true
      ip_protocol_fallback: false
      method: GET
      no_follow_redirects: false
      preferred_ip_protocol: ip4
      valid_http_versions: 
        - HTTP/1.1
        - HTTP/2.0
    prober: http
    timeout: 15s

The above module is named http_2xx. It is based on the HTTP Probe offered by Blackbox Exporter. Other configuration parameters, and their meaning, are detailed in the table below:

Fail_if_not_ssl: true	Fail if the endpoint is not SSL secured
method: GET	Use HTTP GET method to access the endpoint
no_follow_redirects: false	Do not follow HTTP redirects
preferred_ip_protocol: ip4	Use IPv4 protocol
valid_http_versions	Both HTTP/1.1 and HTTP/2.0 are valid
timeout: 15s	Timeout after 15s if no response received

Now that we have configured the Blackbox Exporter let’s see how Prometheus gets metrics relevant to this module for an endpoint. Inside the Prometheus config, we have a section as follows:

- job_name: 'blackbox-https'
  scrape_timeout: 15s
  scrape_interval: 15s
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
   - targets:
      - https://example.com
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115

Most of the above Prometheus config is self-explanatory. However, it’s let’s look closer at these two points:

1. Note how we chose the Blackbox Export module inside the params section. This module name should correspond to the module name defined in the Blackbox Exporter config (http_2xx in this case).
2. Under the static configs section, we provide a target as “https://example.com”. This configuration means that Prometheus will query Blackbox Exporter’s http_2xx module with http://example.com as a parameter.

Stitching all of this together, we should be able to see https metrics for https://example.com by visiting http://prometheus-blackbox-exporter:9115/probe?target=https://example.com&module=http_2xx . The probe_success metric indicates if the probe succeeded. Adding a debug=true parameter will return debug information for that probe.

Let’s look at another example for a TCP-level probe:

modules: 
  tcp_connect: 
    prober: tcp
    tcp: 
      ip_protocol_fallback: false
      preferred_ip_protocol: ip4
      tls: true
      tls_config: 
        insecure_skip_verify: true
        server_name: myserver
    timeout: 15s

The module defined in the snippet above is named tcp_connect and uses BlackBox Exporter’s TCP Probe. Most of the config options are similar to the http_2xx probe except for some TLS settings. Let’s look at those:

tls: true	Enable TLS verification for TCP connection
insecure_skip_verify: true	Disables SSL certificate verification. This means that your certificate does not need to be a valid one and can be self-signed.
server_name: myserver	The presented certificate should have ‘myserver’ added as one of the SANs (Subject Alternative Name) to the certificate

Similar to our earlier config, we can have another job config added to our Prometheus configuration file. This job uses the tcp_connect module and gets metrics for a TCP endpoint using self-signed certificates.

- job_name: 'blackbox-tcp'
  scrape_timeout: 15s
  scrape_interval: 15s
  metrics_path: /probe
  params:
    module: [tcp_connect]
  static_configs:
   - targets:
      - <my_tcp_server>:<my_tcp_server_port>
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115

Similar to the case above, we should be able to see tcp metrics for endpoint <my_tcp_server>:<my_tcp_server_port> by visiting http://prometheus-blackbox-exporter:9115/probe?target=<my_tcp_server>:<my_tcp_server_port>&module=tcp_connect . The probe_success metric indicates if the probe succeeded. Adding a debug=true parameter will return debug information for that probe.

With these examples, you should now understand how Blackbox Exporter works and how you can configure Prometheus to scrape HTTPS and TCP metrics from your endpoint. Now, let’s learn a bit more about the other exporters.

Kube-state Metrics Exporter

If you want to monitor your Kubernetes workloads using a Prometheus instance,  deploying Kube-state Metrics is an absolute must.

This exporter will generate crucial container and pod level metrics for all kinds of K8s workloads. These metrics include vital stats like container CPU/Memory limits and requests, number of replicas for deployment to cronjob, and ingress-level metrics. For more detail, you can find the complete list of metrics here

You can easily deploy Kube-state Metrics by using a helm chart.

Node Exporter

Node Exporter provides OS and Hardware level metrics for the VMs. You can deploy Node Exporter as a process running over each VM or as a Daemonset if you want to monitor metrics for K8s nodes.

It provides metrics like disk usage, IOPS, CPU / Memory usage, overall system load, etc. For more detail, you can find the complete list of metrics here.

You can find in-depth installation instructions for different platforms here.

Elasticsearch Exporter

Elasticsearch is one of the most popular document stores and is used extensively for power search-based backends. It also has a K8s operator named ECK for easy orchestration of Elasticsearch workloads over Kubernetes. The Elasticsearch Exporter enables you to use Prometheus with ES.

This exporter can connect to your ES instance and get crucial performance and health metrics about the database. It provides metrics like cluster health, the total number of shards, initializing or unassigned shards, etc. You can find the complete list of metrics here.

You can find a detailed installation guide for different platforms here

Redis Exporter

Redis is hands down the most popular cache used by modern software applications. Given its crucial position in the tech stack. It is imperative to monitor your Redis instances.

The Redis Exporter connects to your Redis instance like any other Redis client and generates metrics in a Prometheus-friendly format.

You can find a detailed view of Redis Exporter metrics here . It is straightforward to install this exporter on Kubernetes or other platforms. Just follow the configuration guide here.

Writing your own Prometheus Exporter

Sometimes using existing open-source Prometheus Exporters is not enough. There might be scenarios where you need Prometheus to pull specific metrics relevant to your application. While many standard use cases are covered, there won’t always be an existing exporter to do the job for you.  Fortunately, you can simply write a Prometheus Exporter yourself.

Where do you start? The official Prometheus Documentation provides a very detailed guide.

Final thoughts

Prometheus Exporters are an excellent means of de-coupling metrics generation and storage in a Prometheus-based monitoring system. While you may not use all of the exporters out there, some are fundamental to monitoring any modern infrastructure and application environment. For example, Prometheus Blackbox Exporter is crucial for endpoint monitoring.

However, simply setting up exporters is not helpful if your Prometheus instance is not scalable, highly available, or fault-tolerant. To solve that problem, we would like to introduce Opsramp.

Based on the scalable Prometheus Cortex technology, OpsRamp off-loads Prometheus maintenance from DevOps teams and allows them to focus on more important tasks like correctly configuring exporters and other metric sources. Opsramp also seamlessly integrates with your public cloud subscription to provide you with cloud provider-specific metrics. You can get started controlling chaos and explore all of Opsramp for free. You can sign-up here.