Monitoring Custom Applications with Prometheus Operator

The Prometheus Operator has recently been promoted to a stable Helm chart, making it easier to manage Prometheus instances in Kubernetes. This article will guide you through the process of adding a custom application, such as GitLab Runner, to be monitored by Prometheus.

Example: Monitoring GitLab Runner

GitLab Runner exposes metrics on port 9252 by default. To monitor this application, you will need to create a ServiceMonitor resource that tells Prometheus how to scrape the metrics.

Step 1: Create a ServiceMonitor

Below is an example YAML configuration for a ServiceMonitor that targets the GitLab Runner:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab-monitor
  namespace: default  # Ensure this matches the namespace where Prometheus is deployed
  labels:
    app: gitlab-runner
    release: prometheus
spec:
  selector:
    matchLabels:
      app: gitlab-runner
  namespaceSelector:
    any: true  # Allows monitoring across all namespaces
  endpoints:
  - port: http-metrics  # This should match the service port exposing metrics
    interval: 15s  # Scrape interval

Step 2: Verify Prometheus Configuration

To ensure that your ServiceMonitor is correctly configured, check the Prometheus instance configuration:

kubectl get prometheus -o yaml

Look for the following sections in the output:

  • serviceMonitorNamespaceSelector: This should be empty or configured to include the namespace of your ServiceMonitor.
  • serviceMonitorSelector: Ensure it matches the labels defined in your ServiceMonitor (e.g., release: prometheus).

Troubleshooting

If the metrics from GitLab Runner do not appear in the Prometheus UI, consider the following:

  • Ensure that the labels in your ServiceMonitor match the labels of the GitLab Runner service.
  • Check the Prometheus logs for any errors related to scraping.
  • Validate that the GitLab Runner is correctly exposing metrics on the specified port.

By following these steps, you should be able to successfully integrate your custom application with Prometheus using the Prometheus Operator.