Running a Monitoring Daemon Inside Your Kubernetes Cluster

May 22, 2026

The short version

We run a Python daemon as a Kubernetes Deployment that polls Graphite every 5 minutes and fires alerts when Jenkins build durations spike relative to their 7-day moving average.

1. One replica Deployment - Kubernetes handles restarts, no external process supervisor needed
2. Thresholds live in a ConfigMap - change them without rebuilding the image
3. Credentials come from GKE Secret Manager via CSI driver - no k8s Secrets, no static files
4. Runs on a preemptible node - polling monitors tolerate brief gaps, no reason to pay for guaranteed capacity

Why run a monitor inside the cluster

The obvious place to run a monitoring daemon is outside the cluster - a VM, a managed service, a Lambda. But for a daemon that only needs to reach outward (to Graphite and an alerting endpoint), running inside the cluster has real advantages.

It lives with the infrastructure it monitors. Deployed with the same manifests, visible in the same logs, restarted by the same control loop. When something is wrong with the cluster, the monitor shows up in the same place you’re already looking.

It needs no inbound firewall rules. The daemon reaches out to Graphite - nothing reaches in to it. No ingress, no load balancer, no port forwarding.

Kubernetes handles restarts automatically. If the pod crashes, it comes back. You don’t need a systemd unit or a cron to check if the process is alive.

The check: p99 ratio vs moving average

The monitor watches Jenkins build durations using this Graphite query:

aliasByNode(
  divideSeries(
    systems.jenkins.jobs.*.duration_p99,
    movingAverage(systems.jenkins.jobs.*.duration_p99, '7d')
  ),
  3
)

This produces a ratio: current p99 duration divided by the 7-day moving average of that same metric. A ratio of 1.0 means the job is as fast as usual. A ratio of 2.0 means it’s taking twice as long as normal.

The thresholds are:

• Warning: ratio > 1.3 (30% slower than the 7-day average)
• Critical: ratio > 2.0 (double the 7-day average)

Using a relative threshold instead of an absolute one matters. A job that always takes 10 minutes should not alert - that’s its normal behaviour. A job that suddenly takes 20 minutes should. Absolute thresholds require manual tuning per job and drift over time as CI infrastructure changes. The ratio approach is self-calibrating: if a job gets faster or slower over weeks, the moving average shifts with it, and the alert threshold moves too.

The query window is the last 30 minutes (from: -30min), so the check is asking: in the last half hour, did any job’s p99 spike relative to its week-long baseline?

The daemon loop

The daemon itself is simple:

def main() -> None:
    graphite_check = GraphiteCheck("configs/jenkins-p99-spike.yaml", dry_run=False)
    check_interval = 300  # 5 minutes
    while True:
        graphite_check.check()
        sleep(check_interval)

It initialises the check object once, then loops: check, sleep. GraphiteCheck handles querying Graphite, evaluating thresholds, and firing alerts. The loop handles cadence.

A 5-minute poll interval works for build duration monitoring - CI is not real-time. If a job is running slow, we’ll know within 5 minutes, which is fast enough to be useful and slow enough to not hammer the Graphite API.

Config in a ConfigMap, secrets from Secret Manager

The check configuration - the Graphite query, thresholds, and responders - lives in a ConfigMap:

data:
  jenkins-p99-spike.yaml: |
    - display_name: "Jenkins p99 build duration spike:"
      query: aliasByNode(divideSeries(systems.jenkins.jobs.*.duration_p99, movingAverage(systems.jenkins.jobs.*.duration_p99, '7d')), 3)
      warning:
        default: 1.3
      critical:
        default: 2.0
      reverse_check: false
      global_state: true
      from: -30min
      responders:
        - devops

The ConfigMap is mounted into the pod at /home/deploy/configs. The daemon reads it at startup. To change a threshold or add a new check, update the ConfigMap and restart the pod.

Credentials (Graphite API key, alerting webhook token) come from Google Secret Manager via the GKE Secret Manager CSI driver. The pod has a secret-manager-access service account with IAM permission to read specific secrets. At pod startup, the CSI driver mounts the secrets as files under /mnt/secrets. Nothing is stored in a k8s Secret or baked into the image.

volumes:
  - name: secrets-store
    csi:
      driver: secrets-store-gke.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: secret-manager-provider

This keeps credentials out of etcd and gives you IAM-level access control and audit logs on every secret read.

The deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-monitor
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: jenkins-monitor
        image: <registry>/monitoring-daemons:latest
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"
        volumeMounts:
          - name: config
            mountPath: "/home/deploy/configs"
            readOnly: true
          - name: secrets-store
            mountPath: "/mnt/secrets"
            readOnly: true
        command: ["/bin/bash", "-c", "cd /home/deploy/src/ && python3 jenkins-monitor.py"]
      nodeSelector:
        preemptible: "true"

When it breaks

Pod keeps restarting.

Check kubectl logs for the previous container (--previous). Most likely: the CSI driver failed to mount secrets (IAM misconfiguration or the SecretProviderClass doesn’t exist), or the Graphite endpoint is unreachable and the check is throwing on startup.

Alerts stopped firing but the pod is running.

The daemon might be running but the alerting endpoint credentials are stale or the webhook URL changed. Check the mounted secret at /mnt/secrets - if it’s empty or malformed, the check silently swallows the alert. Add dry_run=True temporarily to log what would have fired without actually sending.

Threshold is firing too often or not at all.

The from: -30min window may not have enough data for noisy or infrequent jobs. For jobs that run less than once per 30 minutes, widen the window. For noisy jobs with naturally high variance, raise the warning threshold or switch to a longer moving average baseline.

ConfigMap change isn’t taking effect.

The daemon reads config at startup, not on every check. After updating the ConfigMap, restart the pod: kubectl rollout restart deployment/jenkins-monitor.

No data in Graphite for a job.

The query uses wildcards (systems.jenkins.jobs.*). If a job’s metrics stopped flowing - because the job was renamed, deleted, or Jenkins lost its Graphite connection - the series disappears and the check produces no result for that job. This is a silent miss, not an alert. Monitor Graphite metric counts separately if you need to detect missing series.