Relabel instance to hostname in Prometheus
I just came across this problem and the solution is to use a group_left to resolve this problem. You can't relabel with a nonexistent value in the request, you are limited to the different parameters that you gave to Prometheus or those that exists in the module use for the request (gcp,aws...).
So the solution I used is to combine an existing value containing what we want (the hostnmame
) with a metric from the node exporter. Our answer exist inside the node_uname_info
metric which contains the nodename
value.
I used the answer to this post as a model for my request: https://stackoverflow.com/a/50357418 .
The solution is this one:
node_memory_Active_bytes
* on(instance) group_left(nodename)
node_uname_info
With this, the node_memory_Active_bytes
metric which contains only instance
and job
labels by default, gets an additional nodename
label that you can use in the description field of Grafana.
Hope that this will help others.
This solution stores data at scrape-time with the desired labels, no need for funny PromQL queries or hardcoded hacks.
It does so by replacing the labels for scraped data by regexes with relabel_configs
.
By default, instance
is set to __address__
, which is $host:$port
.
First attempt: In order to set the instance
label to $host
, one can use relabel_configs
to get rid of the port of your scaping target:
- job_name: 'whatever'
static_configs:
- targets: [
'yourhost.lol:9001'
]
relabel_configs:
- source_labels: [__address__]
target_label: instance
regex: '([^:]+)(:[0-9]+)?'
replacement: '${1}'
But the above would also overwrite labels you wanted to set e.g. in the file_sd_configs
:
[
{
"targets": ['yourhost.lol:9001'],
"labels": {
"instance": 'node42'
}
}
]
Solution: If you want to retain these labels, the relabel_configs
can rewrite the label multiple times be done the following way:
- job_name: 'whatever'
metrics_path: /metric/rolf
file_sd_configs:
- files:
- rolf_exporter_targets.yml
relabel_configs:
- source_labels: [instance]
target_label: __tmp_instance
regex: '(.+)'
replacement: '${1};'
- source_labels: [__tmp_instance, __address__]
separator: ''
target_label: instance
regex: '([^:;]+)((:[0-9]+)?|;(.*))'
replacement: '${1}'
Doing it like this, the manually-set instance
in sd_configs
takes precedence, but if it's not set the port is still stripped away.
I found hardcode solution:
global: scrape_interval: 5s scrape_timeout: 5s external_labels: monitor: 'Prometheus' scrape_configs: - job_name: 'shelby' static_configs: - targets: - 10.100.0.01:9100 relabel_configs: - source_labels: [__address__] regex: '.*' target_label: instance replacement: 'shelby' - job_name: 'camaro' static_configs: - targets: - 10.101.0.02:9100 relabel_configs: - source_labels: [__address__] regex: '.*' target_label: instance replacement: 'camaro' - job_name: 'verona' static_configs: - targets: - 10.101.0.03:9100 relabel_configs: - source_labels: [__address__] regex: '.*' target_label: instance replacement: 'verona'
Result:
node_load15{instance="camaro",job="camaro"} 0.16 node_load15{instance="shelby",job="shelby"} 0.4 node_load15{instance="verona",job="verona"} 0.07