Monitoring & Metrics

Job Status & Phases

Every AgentJob reports a status.phase plus rich detail: commits, changed files, PR URL, token usage, and estimated cost. The phase lifecycle:

kubectl

The CRDs define printer columns, so plain kubectl get is already informative:

# Live phase, branch, and commit count per job
kubectl get agentjobs -w

# Plans show mode and step progress
kubectl get agentplans

# Merge queues show phase, queue length, and target branch
kubectl get mergequeues

# Full status, events, and preserved logs for one job
kubectl describe agentjob update-dependencies
kubectl get agentjob update-dependencies -o jsonpath='{.status.logs}'

# Cost and token usage
kubectl get agentjob update-dependencies \
  -o jsonpath='{.status.estimatedCostDollars} {.status.tokenUsage}'

Agent logs are copied into status.logs before the pod is cleaned up, so you can debug completed jobs without hunting for dead pods.

Prometheus Metrics

The controller exposes a standard controller-runtime /metrics endpoint (default bind address :8080, configurable with --metrics-bind-address). Alongside the built-in controller-runtime and Go metrics, nominos-core registers its own series:

Scrape it like any other in-cluster target:

- job_name: nominos-controller
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names: [nominos-system]
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      regex: "8080"
      action: keep

Useful starting points: alert on a rising nominos_agentjob_retries_total rate, graph nominos_agentjob_cost_dollars by model to watch spend, and use nominos_agentjobs_total{phase="Failed"} for failure-rate SLOs.

Dashboard Views

The Nominos dashboard mirrors this data per tenant:

• Jobs list — every job with live phase, repo, and cost
• Job detail — streaming logs, checkpoints, token usage, PR link
• Pipeline & Kanban views — plans and tickets as they move through stages
• Merge queue view — queue contents, current PR, merge history and auto-reverts