Generating range vectors from return values in Prometheus queries
Yes, you need to use a recording rule for this.
Prometheus calculates the rate of client requests over the last 2 mins and returns a derivative of the resulting values over the last 5 mins.
Herein lies the problem - at what interval should Prometheus synthesise this data?
Solution
It's possible with the subquery-syntax (introduced in Prometheus version 2.7):
deriv(rate(varnish_main_client_req[2m])[5m:10s])
Warning: These subqueries are expensive, i.e. create very high load on Prometheus. Use recording-rules when you use these queries regularly (in alerts, etc.).
Subquery syntax
<instant_query>[<range>:<resolution>]
instant_query: a PromQL-function which returns an instant-vector)range: offset (back in time) to start the first subqueryresolution: the size of each of the subqueries.
It returns a range-vector.
In the example above, Prometheus runs rate() (= instant_query) 30 times (the first from 5 minutes ago to -4:50, ..., the last -0:10 to now).
The resulting range-vector is input to the deriv()-function.
Another example (mostly available on all Prometheus instances):
deriv(rate(prometheus_http_request_duration_seconds_sum{job="prometheus"}[1m])[5m:10s])
Without the subquery-range ([5m:10s]), you'll get this error-message:
parse error at char 80: expected type range vector in call to function "deriv", got instant vector