Monitoring

Health endpoints, container checks, logs and the signals worth an alert on a Compose installation.

Run the commands on this page from the install directory: ./grovs for the installer, /opt/grovs as root on a cloud VM. For a local trial, replace --profile standalone with -f docker-compose.yml -f docker-compose.local.yml.

Health endpoints

Endpoint	Auth	Returns
`https://<API_HOST>/up`	None	`200` with `OK` once Rails has booted. The `web` healthcheck and the Caddy certificate gate use it too.
`https://<API_HOST>/api/v1/diagnostics/health_metrics`	`X-Diagnostics-Key` header	Queue depths and ClickHouse status as JSON

/up answers on every backend host, so you can point an external uptime monitor at it. It does not check the databases.

Health metrics

Call it with the DIAGNOSTICS_API_KEY from .env:

Treat DIAGNOSTICS_API_KEY like an admin password. In 2.3.1 it also opens the diagnostics test endpoints, which write data. Run checks from your own server, not a third-party monitor, and rotate the key if it leaks (Security).

Bash

curl -fsS -H "X-Diagnostics-Key: $(grep -E '^DIAGNOSTICS_API_KEY=' .env | cut -d= -f2-)" \
  "https://$(grep -E '^API_HOST=' .env | cut -d= -f2-)/api/v1/diagnostics/health_metrics"

JSON

{
  "timestamp": "2026-10-22T13:00:07Z",
  "events_pending": 0,
  "sidekiq_queues": { "events": 0, "batch_events": 0, "default": 0, "maintenance": 0, "device_updates": 0 },
  "clickhouse": { "up": true, "disk_free_pct": 81.5 }
}

Field	Meaning
`events_pending`	Events buffered in Redis, waiting for the batch processor that runs every minute
`sidekiq_queues`	Jobs waiting in each worker queue
`clickhouse.up`	`false` when ClickHouse does not answer. `error` then names the failure.
`clickhouse.disk_free_pct`	Free space on the ClickHouse data disk, in percent

The endpoint reports an outage as data instead of failing: Redis problems show as null queue values plus redis_error.

Containers

Bash

docker compose --profile standalone ps -a

Service	Healthcheck	Healthy state
`postgres`	`pg_isready`	`running (healthy)`
`redis`	`redis-cli ping`	`running (healthy)`
`clickhouse`	HTTP `/ping`	`running (healthy)`
`web`	HTTP `/up`	`running (healthy)`
`dashboard`	HTTP `/login`, built into the image	`running (healthy)`
`worker-1`, `worker-2`, `proxy`	None	`running`
`migrate`	None	`exited (0)`

Every long-running service restarts automatically (restart: unless-stopped). A container that keeps restarting shows a short Up time; read its logs.

Logs

The backend logs to stdout, and Docker stores the output.

Bash

docker compose --profile standalone logs --tail=100 -f web
docker compose --profile standalone logs --since=1h worker-1 worker-2
docker compose --profile standalone logs --tail=100 postgres redis clickhouse proxy

The backend logs at error level by default. To see more, set RAILS_LOG_LEVEL (see Configuration).

Log rotation

The Compose file sets no logging options, so containers use Docker's default json-file driver, which never rotates. Switch the host to the rotating local driver in /etc/docker/daemon.json:

JSON

{
  "log-driver": "local"
}

Restart Docker, then recreate the containers so they pick up the driver:

Bash

sudo systemctl restart docker
docker compose --profile standalone up -d --force-recreate

What to alert on

Signal	Check	Alert when
API down	`/up` from outside the server	Not `200`
Container down	`docker compose ps -a`	A service other than `migrate` is not `running`, or is `unhealthy`
ClickHouse down	`clickhouse.up`	`false`
Workers behind	`events_pending`, `sidekiq_queues`	The value keeps rising across checks instead of draining
Disk filling	`clickhouse.disk_free_pct`, `df -h /var/lib/docker`	Below the headroom you need to grow or clean up
Certificates	HTTPS on a project link host	TLS errors. See DNS and TLS.
Backups	The newest backup file	Older than your schedule. See Backups.

Example check script

Save this as check.sh in the install directory. It prints what is wrong and exits with 1, so any scheduler or monitoring agent can alert on it.

Bash

#!/usr/bin/env bash
# Print problems and exit 1 when Grovs needs attention. Run from the install directory.
set -uo pipefail
cd "$(dirname "$0")"
env_value() { grep -E "^$1=" .env | cut -d= -f2-; }
API="$(env_value SERVER_HOST_PROTOCOL)$(env_value API_HOST)"
status=0
 
curl -fsS --max-time 10 "$API/up" >/dev/null || { echo "API health check failed: $API/up"; status=1; }
 
bad="$(docker compose --profile standalone ps -a --format '{{.Service}} {{.State}} {{.Health}}' \
  | awk '$1 != "migrate" && ($2 != "running" || $3 == "unhealthy")')"
[ -z "$bad" ] || { echo "Containers not running or unhealthy:"; echo "$bad"; status=1; }
 
metrics="$(curl -fsS --max-time 10 -H "X-Diagnostics-Key: $(env_value DIAGNOSTICS_API_KEY)" \
  "$API/api/v1/diagnostics/health_metrics")" || { echo "Health metrics unavailable"; status=1; }
echo "$metrics" | grep -q '"up":true' || { echo "ClickHouse is down: $metrics"; status=1; }
echo "$metrics"
 
exit "$status"

Bash

chmod 700 check.sh
./check.sh

Run it every few minutes from cron or your monitoring agent, and alert on a non-zero exit. This root crontab entry on a cloud VM keeps a log of every run:

Bash

*/5 * * * * /opt/grovs/check.sh >> /var/log/grovs-check.log 2>&1

Last updated 2026-10-26

Was this page helpful?