Skip to content

Create alert with Prometheus

This guide shows you how to create alerts for your application.

Prerequisites

You can define alerts by using Kubernetes resources (PrometheusRule), as well as directly in Grafana (GUI based).

You will have a separate Alertmanager for each environment available at https://alertmanager.<MY-ENV>.nav.cloud.nais.io/.

Create PrometheusRule

We use native Prometheus alert rules, and let Alertmanager handle the notifications.

You can define alerts by creating a PrometheusRule resource in your teams namespace.

.nais/alert.yaml 
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: <MY-ALERT>
  namespace: <MY-TEAM>
  labels:
    team: <MY-TEAM>
spec:
  groups:
  - name: <MY-ALERT>
    rules:
    - alert: InstanceDown
      expr: kube_deployment_status_replicas_available{namespace="<namespace>", deployment="<application name>"} == 0
      for: 5m
      annotations:
        consequence: Application is unavailable
        action: "`kubectl describe pod <podname>` -> `kubectl logs <podname>`"
        summary: |-
          This is a multi-line summary with
          linebreaks and everything. Here you can give a more detailed
          summary of what this alert is about
      labels:
        namespace: <MY-TEAM> # required
        severity: critical

Activate the alert

Add the file to your application repository, alongside nais.yaml to deploy with Nais github action.

kubectl apply -f ./nais/alert.yaml

Verify your alert

You can see the alerts in the Alertmanager at https://alertmanager.<MY-ENV>.nav.cloud.nais.io/ and the defined rules in Prometheus at https://prometheus.<MY-ENV>.nav.cloud.nais.io/

Learn more

💡 Advanced alerting with Prometheus 📖 Prometheus alerts reference