App health and Phoenix

Every Kamal app gets request health (requests, server errors and response time) from the kamal-proxy log with nothing installed. See App health and Rails for how that works; it is the same for Phoenix.

For Phoenix apps, the optional upstate_phoenix package adds what request logs cannot see: whether the app booted on the expected version, pending Ecto migrations, and whether Oban jobs are moving.

Until it is installed, the project page shows Phoenix health: not collected for that environment, with the snippet below. If the framework chip does not say Phoenix, choose it in the environment settings, or let the CLI detect it on your next deploy.

Add Phoenix health

1. On the project page, open App health and choose Generate token under Add Phoenix health. Only owners and admins can do this. The token is shown once, so copy it before you leave the page.

2. Add the dependency to mix.exs and run mix deps.get:

   {:upstate_phoenix, "~> 0.1"}

3. Add the plug to your endpoint, above plug Plug.Telemetry:

   plug UpstatePhoenix.Plug, otp_app: :my_app

   Replace :my_app with your application's name. It finds your Ecto repos from that app's :ecto_repos and reads Oban through its running instance.

4. If you use force_ssl, exclude the health path, because the observer calls the container's private IP over plain HTTP:

   force_ssl: [
     rewrite_on: [:x_forwarded_proto],
     exclude: [hosts: ["localhost", "127.0.0.1"], paths: ["/up", "/upstate/health"]]
   ]

   Set force_ssl in config/prod.exs and rebuild the release; Phoenix compiles this setting into its endpoint, so a runtime-only change is insufficient.

   The paths exclusion requires Plug 1.19 or later. On older supported Plug versions, use exclude: {UpstatePhoenix, :private_host?, []} instead; this exempts private-IP hosts. See the package README for details.

5. Add the token to .kamal/secrets, and list it under env.secret in config/deploy.yml:

   UPSTATE_HEALTH_TOKEN=paste-the-token-here

   env:
     secret:
       - UPSTATE_HEALTH_TOKEN

6. Deploy with Kamal as usual.

The plug stays off (it answers 404) until UPSTATE_HEALTH_TOKEN is set, and answers 401 to a wrong token. The observer asks each container for /upstate/health from inside your server's private network and sends the answer with its next report. The first result usually appears within a couple of minutes. Do not commit the token.

To replace a token, choose Rotate token, update .kamal/secrets, and deploy. To stop collecting Phoenix health, choose Revoke token.

What is sent

• App version, Phoenix, Elixir and OTP versions, the environment, and when the app booted.
• Whether the database responds and whether migrations are pending.
• Oban job counts: waiting, oldest waiting age, completed and failed in the last five minutes, and when a job last ran.

What is never sent

Request or response bodies, parameters, headers, cookies, paths, SQL, client IP addresses, user agents, job arguments, job errors, queue and worker names, migration names, exception messages, and environment variables. The token itself is never written to Upstate's logs or audit trail.

Oban jobs

The plug reads job numbers with one aggregate query on Oban's oban_jobs table, so containers that only run jobs are covered by any web container. Oban on PostgreSQL is supported. Other job systems are shown as not read yet.

Completed and discarded jobs are counted only while they are still in the table. Oban's Pruner keeps jobs for 60 seconds by default; keep them for at least five minutes ({Oban.Plugins.Pruner, max_age: 300} or more) so the five-minute counts are complete.

Upstate opens Background jobs are stuck in the queue when jobs have waited more than five minutes and nothing has run recently, and Background jobs are failing when at least 10 jobs, and more than 10% of those processed, failed in five minutes. See Alerts and notifications.

Troubleshooting