Prometheus: relabel your scrape_config

Prometheus labels every data point — the most well-known example of a label is (probably) instance.

Take a look at this query result (query: up{job="prometheus"}):

up{instance="127.0.0.1:9090",job="prometheus"} 1

So what does this tell me?

I queried for the “up” metric and filtered it for “prometheus” — yay. The “1” says, my service is alive. So far so gut.

Readability

Since we are in the process of running a few Prometheus servers (in federation), each of those metrics will report back with instance="127.0.0.1:9090" (along with other labels of course).

While this works, I’m not a computer. If the “instance” reported an FQDN or some other readable name, it would make any dashboard or alert more approachable. Or readable, if you will.

The instance label

instance is a standard field used in various Grafana dashboards out there. Dashboards often use the value in instance to provide you with a dropdown list of (well) instances (or nodes) to select from.

To not end up with a dropdown full of 127.0.0.1:9000, here is a snippet on how to work with labels to make life a little easier.

Rewriting labels

Consider the following scrape_config:

- job_name: "prometheus"
  metrics_path: "/metrics"
  static_configs:
  - targets:
    - "127.0.0.1:9090"

It produces the result above.

Now, extend it slightly to include a name and relabel the instance field with it:

- job_name: "prometheus"
  metrics_path: "/metrics"
  relabel_configs:
    - source_labels: [name]
      target_label: instance
  static_configs:
  - targets:
    - "127.0.0.1:9090"
    labels:
      name: my-prometheus.example.org

Query again:

up{instance="my-prometheus.example.org",job="prometheus",name="my-prometheus.example.org"} 1

Now “instance” is set to something I can grok by glancing over it. Which makes me happy.

Fin

Thanks for following along!