Prometheus/PromQL subtract two gauge metrics
By default Prometheus performs a - b
in the following way:
- It selects all the time series for
a
query. - It selects all the time series for
b
query. - It searches pairs of time series at
a
andb
results with identical sets of labels. - It calculates the difference between time series in the found pairs with identical sets of labels.
See these docs for more details.
The metric_awesome{instance="one"} - metric_awesome{instance="two"}
returns empty result, because all the matching time series on the left side of -
contain instance="one"
label, while all the time series on the right side of -
contain instance="two"
label. There are no time series pairs with identical sets of labels here.
This behavior can be changed with on()
, ignoring()
, group_left()
and group_right()
modifiers.
For example, metric_awesome{instance="one"} - ignoring(instance) metric_awesom{instance="two"}
instructs to ignore instance
label during searching for time series pairs with identical labels. This may result in multiple-to-one matching, when multiple time series on one side of -
match a single time series on the another side. By default Prometheus returns error in this case. This can be fixed by adding group_left()
or group_right()
modifiers to -
operator:
metric_awesome{instance="one"}
- ignoring(instance) group_left()
metric_awesome{instance="two"}
or
metric_awesome{instance="one"}
- ignoring(instance) group_right()
metric_awesome{instance="two"}
See these docs for more details on these modifiers.
If anyone searches this wanting to do this for a one-to-many subtraction, have a look at group_right
additionally to what was written before.
metric_awesome{instance="one"} - ignoring(instance) group_right metric_awesome{job="compare-instances"}
See also Prometheus dokumentation
The issue here is that the labels don't match. What you want is:
metric_awesome{instance="one"} - ignoring(instance) metric_awesome{instance="two"}