Before You Debug
Almost every failure is explained by three commands:
Terminal
kubectl describe agentjob <name> # events + conditions
kubectl get agentjob <name> -o jsonpath='{.status.logs}' # preserved agent logs
kubectl get agentjob <name> -o jsonpath='{.status.warnings}'Missing provider API key
- Symptom
- The job fails within seconds of starting. Logs contain "ERROR: ANTHROPIC_API_KEY not set. Configure Anthropic credentials in the dashboard." (or the OPENAI_API_KEY / GEMINI_API_KEY equivalent).
- Cause
- The runner validates the selected provider’s API key before doing any work and exits immediately if it is missing from the credentials secret (default: nominos-credentials).
- Fix
- Add the key in the dashboard under Settings → Connections → AI Providers (or add it to the credentials secret referenced by spec.agent.credentialsSecret), then re-create the job. Make sure the key matches the provider selected in spec.agent.zeroShot.provider.
Checkpoint approval timeout
- Symptom
- A job in checkpoints mode reports "Waiting for approval..." in its logs, then fails roughly one hour later.
- Cause
- Checkpoint-based jobs pause at each configured trigger (before_commit, before_pr, …) and poll for a decision. If nobody approves or rejects within 1 hour, the checkpoint times out.
- Fix
- Approve or reject pending checkpoints promptly from the dashboard notification inbox. If the job doesn’t need human gates, switch spec.interactionMode.mode to fire_and_forget or trim the checkpointTriggers list.
Job hits its timeout
- Symptom
- The job runs for about an hour and then fails; logs stop mid-task.
- Cause
- Agents are killed when they exceed spec.timeoutSeconds (default 3600). Large refactors, slow test suites, or repeated tool retries can exhaust the budget.
- Fix
- Raise spec.timeoutSeconds (e.g. 7200 for long migrations), narrow the task with task.focusFiles, or split the work into an AgentPlan with smaller steps and per-step timeoutSeconds.
Job stuck retrying, then Failed
- Symptom
- status.retries climbs to 3, then the phase flips to Failed.
- Cause
- Every failed attempt is retried up to spec.backoffLimit (default 3). Persistent failures usually mean a deterministic problem — bad repo URL, broken auth, or a task the agent cannot complete.
- Fix
- Read the preserved output in status.logs and status.warnings to find the underlying error, fix it, and re-create the job. Raise or lower backoffLimit to control retry appetite.
Agent pod stuck in ImagePullBackOff
- Symptom
- The AgentJob sits in Queued/Running with no log output. kubectl describe pod shows ErrImagePull or ImagePullBackOff for the agent image.
- Cause
- The agent image (default ghcr.io/johnhenry/nominos-agent:latest) is pulled from a registry the node cannot reach, or the registry requires authentication.
- Fix
- Create a registry secret (kubectl create secret docker-registry ghcr-pull --docker-server=ghcr.io …) and reference it in spec.agent.imagePullSecrets. Verify custom spec.agent.image values are spelled correctly and reachable from the cluster.
RBAC: controller cannot create jobs or update status
- Symptom
- AgentJobs stay Pending forever. Controller logs show "forbidden" errors like: cannot create resource "jobs" in API group "batch".
- Cause
- The controller’s ServiceAccount is missing its RBAC rules — usually because make deploy (which applies config/rbac/) was skipped, or the controller runs in a namespace the bindings don’t cover.
- Fix
- Re-apply the operator RBAC using the manifests provided during onboarding (kubectl apply -f <path-to-rbac-manifests>). Confirm with: kubectl auth can-i create jobs.batch --as=system:serviceaccount:nominos-system:nominos-controller.
Still stuck?
Check Monitoring & Metrics for how to read job status in depth, or reach out through the dashboard — include the job name and the output of kubectl describe agentjob.