How to create a ServiceMonitor for prometheus-operator?
This image perfectly shows the connection between Prometheus,ServiceMonitors and Services
If any of the matches are not correct, targets won't show up.
Read more: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md#troubleshooting-servicemonitor-changes
I know this question is already answered. But I had a similar problem when Prometheus deployed in Kubernetes with Helm's stable/prometheus-operator chart couldn't find any active targets for my ServiceMonitor
.
It turned out that my Service exposed a port that I didn't explicitly named:
- protocol: TCP
port: 8080
targetPort: uwsgi
I could use it in Ingress by targeting uwsgi
port. But it seems that ServiceMonitor
needs an explicitly named port in Service
even if it has the same name as its own tagetPort:
- name: uwsgi
protocol: TCP
port: 8080
targetPort: uwsgi
I have written a blog post about this problem here
Thanks to Peter who showed me that it idea in principle wasn't entirely incorrect I've found the missing link. As a servicemonitor
does monitor services (haha), I missed the part of creating a service which isn't part of the gitlab helm chart. Finally this yaml did the trick for me and the metrics appear in Prometheus:
# Service targeting gitlab instances
apiVersion: v1
kind: Service
metadata:
name: gitlab-metrics
labels:
app: gitlab-runner-gitlab-runner
spec:
ports:
- name: metrics # expose metrics port
port: 9252 # defined in gitlab chart
targetPort: metrics
protocol: TCP
selector:
app: gitlab-runner-gitlab-runner # target gitlab pods
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: gitlab-metrics-servicemonitor
# Change this to the namespace the Prometheus instance is running in
# namespace: default
labels:
app: gitlab-runner-gitlab-runner
release: prometheus
spec:
selector:
matchLabels:
app: gitlab-runner-gitlab-runner # target gitlab service
endpoints:
- port: metrics
interval: 15s
Nice to know: the metrics
targetPort
is defined in the gitlab runner chart.