Troubleshooting production evidence

A warning in Upstate can mean a confirmed production problem, a reporting failure or not enough evidence. Start with the exact timestamp and affected component before deciding what to change.

Quick diagnostic commands

upstate doctor
upstate status
upstate observer status
upstate receipt --json
upstate kamal app details

These commands read different sources. doctor checks local mapping and setup; observer status inspects observer mode/version/freshness; receipt reads the durable deployment evidence; kamal app details inspects what Kamal sees now. Do not treat one successful command as a substitute for the others.

If no deployment receipt appears

Confirm you ran deployment inside the right repository, with the intended config and destination. Check that upstate connect installed managed lifecycle hooks, especially when an existing custom hook was present. If the wrapper was used without hooks, confirm it could identify the exact version. With a dirty checkout and no hooks, it cannot infer the randomized Kamal version.

Run upstate hook flush to retry retained lifecycle reports without redeploying the application. A failed report submission is not necessarily a failed Kamal deploy; never rerun a deployment just to create an activity entry.

If the receipt stays Incomplete

Read the attention items. Missing post-deploy completion, missing fresh host observations, or insufficient health evidence are separate causes. Check both normal Kamal hooks and the observer's last report. Incomplete is not proof that the deployed app is broken.

If a host is stale

For a systemd observer, inspect systemctl status upstate-agent and journalctl -u upstate-agent on the host. For a Kamal observer accessory, inspect its container and accessory logs. Confirm outbound HTTPS access, valid host credentials and Docker socket access. Restarting the application cannot repair a disconnected observer.

If the deployment needs attention

Inspect which role, host or public health signal is identified. Compare the target version against independently observed instances and the app's expected role count. For HTTP or framework-health failures, investigate the endpoint, pending migrations, queue health and fresh logs. Run corrective Kamal commands only after reviewing their scope and consequences.

Use Deploying with Kamal for command paths, Deployment receipts for verdicts, and App health and Rails or App health and Phoenix for framework-specific signals.

If a notification does not arrive

Open Notifications and inspect the recorded delivery result. A provider-accepted message may still be filtered or unseen by a person. Retest changed credentials, verify destination filters, and check email delivery separately from the underlying alert state. Read Alerts and notifications.

If facts disagree

Be explicit about the source and age of each fact. An observer tells you what Docker reported at its last sample; Kamal tells you what its own command saw; a receipt preserves a particular deployment's conclusion; local Production Audit evaluates source files that may not yet be deployed. No single fact guarantees the whole production environment is healthy.