Filter prometheus results by metric value, not by label value
If you're confused by brian's answer: The result of filtering with a comparison operator is not a boolean, but the filtered series. E.g.
min(flink_rocksdb_actual_delayed_write_rate > 0)
Will show the minimum value above 0.
In case you actually want a boolean (or rather 0 or 1), use something like
sum (flink_rocksdb_actual_delayed_write_rate >bool 0)
which will give you the greater-than-zero count.
This can be solved with subqueries:
count_over_time((metric > 0)[5m:10s])
The query above would return the number of metric
data points greater than 0 over the last 5 minutes.
This query may return inaccurate results depending on the relation between the second arg in square brackets (aka step
for the inner query) and the real interval between raw samples (aka scrape_interval
):
- If the
step
exceedsscrape_interval
, them some samples may be missing during the calculations. In this case the query will return lower than expected result. - If the
step
is smaller than thescrape_interval
, then some samples may be counted multiple times. In this case the query will return bigger than expected result.
So it is recommended setting the step
to scrape_interval
in order to get accurate results.
P.S. The issues mentioned above are solved in VictoriaMetrics - Prometheus-like monitoring system I work on. It provides count_gt_over_time() function, which ideally fits this case. For example, the following MetricsQL query returns the exact number of raw samples with values greater than 0 over the last 5 minutes:
count_gt_over_time(metric[5m], 0)
Filtering is done with the comparison operators, for example x > 0
.