How do I delete a time series from Prometheus v2, specifically a series of alerts

Firstly the admin API is not enabled by default in Prometheus 2. This must be made active by starting the server with the option

--web.enable-admin-api

There is a new endpoint in v2 at

https://prometheus/api/v2/admin/tsdb/delete_series

This takes a POST specifying the search criteria, e.g. for a time series with the name ALERTS where the alert name is MyTestAlert, post the following application/json to the delete_series endpoint, from the tool of choice (Tested with Postman 6 on Mac)

{
    "matchers": [{
        "type": "EQ",
        "name": "__name__",
        "value": "ALERTS"
    },
    {
        "type": "EQ",
        "name": "alertname",
        "value": "MyTestAlert"
    }]
}

For completeness and to free the disk space where the alerts were persisted, POST an empty payload to 

https://prometheus/api/v2/admin/tsdb/clean_tombstones

Answer aggregated from:

  • https://prometheus.io/docs/prometheus/latest/querying/api/
  • https://github.com/prometheus/prometheus/issues/3584
  • https://groups.google.com/forum/#!msg/prometheus-users/ToMKsb9fYp8/az6afuX3CgAJ

Curl variant:

# delete alerts series
curl -g -XPOST 'http://prometheus:9090/api/v2/admin/tsdb/delete_series?match[]=ALERTS'

# delete data from disk
curl -XPOST http://prometheus:9090/api/v2/admin/tsdb/clean_tombstones

Tags:

Prometheus

Recent Posts