# Monitoring

> Health endpoints, container checks, logs and the signals worth an alert on a Compose installation.

Source: https://docs.grovs.io/docs/self-hosting/operate/monitoring

Run the commands on this page from the install directory: `./grovs` for the installer, `/opt/grovs` as root on a cloud VM. For a local trial, replace `--profile standalone` with `-f docker-compose.yml -f docker-compose.local.yml`.

## Health endpoints

| Endpoint                                               | Auth                       | Returns                                                                                                 |
| ------------------------------------------------------ | -------------------------- | ------------------------------------------------------------------------------------------------------- |
| `https://<API_HOST>/up`                                | None                       | `200` with `OK` once Rails has booted. The `web` healthcheck and the Caddy certificate gate use it too. |
| `https://<API_HOST>/api/v1/diagnostics/health_metrics` | `X-Diagnostics-Key` header | Queue depths and ClickHouse status as JSON                                                              |

`/up` answers on every backend host, so you can point an external uptime monitor at it. It does not check the databases.

### Health metrics

Call it with the `DIAGNOSTICS_API_KEY` from `.env`:

> **Warning:** Treat `DIAGNOSTICS_API_KEY` like an admin password. In 2.3.1 it also opens the diagnostics test endpoints, which write data. Run checks from your own server, not a third-party monitor, and rotate the key if it leaks ([Security](https://docs.grovs.io/docs/self-hosting/operate/security)).

```bash
curl -fsS -H "X-Diagnostics-Key: $(grep -E '^DIAGNOSTICS_API_KEY=' .env | cut -d= -f2-)" \
  "https://$(grep -E '^API_HOST=' .env | cut -d= -f2-)/api/v1/diagnostics/health_metrics"
```

```json
{
  "timestamp": "2026-10-22T13:00:07Z",
  "events_pending": 0,
  "sidekiq_queues": { "events": 0, "batch_events": 0, "default": 0, "maintenance": 0, "device_updates": 0 },
  "clickhouse": { "up": true, "disk_free_pct": 81.5 }
}
```

| Field                      | Meaning                                                                          |
| -------------------------- | -------------------------------------------------------------------------------- |
| `events_pending`           | Events buffered in Redis, waiting for the batch processor that runs every minute |
| `sidekiq_queues`           | Jobs waiting in each worker queue                                                |
| `clickhouse.up`            | `false` when ClickHouse does not answer. `error` then names the failure.         |
| `clickhouse.disk_free_pct` | Free space on the ClickHouse data disk, in percent                               |

The endpoint reports an outage as data instead of failing: Redis problems show as `null` queue values plus `redis_error`.

## Containers

```bash
docker compose --profile standalone ps -a
```

| Service                         | Healthcheck                         | Healthy state       |
| ------------------------------- | ----------------------------------- | ------------------- |
| `postgres`                      | `pg_isready`                        | `running (healthy)` |
| `redis`                         | `redis-cli ping`                    | `running (healthy)` |
| `clickhouse`                    | HTTP `/ping`                        | `running (healthy)` |
| `web`                           | HTTP `/up`                          | `running (healthy)` |
| `dashboard`                     | HTTP `/login`, built into the image | `running (healthy)` |
| `worker-1`, `worker-2`, `proxy` | None                                | `running`           |
| `migrate`                       | None                                | `exited (0)`        |

Every long-running service restarts automatically (`restart: unless-stopped`). A container that keeps restarting shows a short `Up` time; read its logs.

## Logs

The backend logs to stdout, and Docker stores the output.

```bash
docker compose --profile standalone logs --tail=100 -f web
docker compose --profile standalone logs --since=1h worker-1 worker-2
docker compose --profile standalone logs --tail=100 postgres redis clickhouse proxy
```

The backend logs at `error` level by default. To see more, set `RAILS_LOG_LEVEL` (see [Configuration](https://docs.grovs.io/docs/self-hosting/configure/configuration#secrets-and-keys)).

### Log rotation

The Compose file sets no logging options, so containers use Docker's default `json-file` driver, which never rotates. Switch the host to the rotating [`local` driver](https://docs.docker.com/engine/logging/drivers/local/) in `/etc/docker/daemon.json`:

```json
{
  "log-driver": "local"
}
```

Restart Docker, then recreate the containers so they pick up the driver:

```bash
sudo systemctl restart docker
docker compose --profile standalone up -d --force-recreate
```

## What to alert on

| Signal          | Check                                               | Alert when                                                                                                             |
| --------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| API down        | `/up` from outside the server                       | Not `200`                                                                                                              |
| Container down  | `docker compose ps -a`                              | A service other than `migrate` is not `running`, or is `unhealthy`                                                     |
| ClickHouse down | `clickhouse.up`                                     | `false`                                                                                                                |
| Workers behind  | `events_pending`, `sidekiq_queues`                  | The value keeps rising across checks instead of draining                                                               |
| Disk filling    | `clickhouse.disk_free_pct`, `df -h /var/lib/docker` | Below the headroom you need to grow or clean up                                                                        |
| Certificates    | HTTPS on a project link host                        | TLS errors. See [DNS and TLS](https://docs.grovs.io/docs/self-hosting/get-started/deployment#tls-and-universal-links). |
| Backups         | The newest backup file                              | Older than your schedule. See [Backups](https://docs.grovs.io/docs/self-hosting/operate/backups#schedule-backups).     |

### Example check script

Save this as `check.sh` in the install directory. It prints what is wrong and exits with `1`, so any scheduler or monitoring agent can alert on it.

```bash
#!/usr/bin/env bash
# Print problems and exit 1 when Grovs needs attention. Run from the install directory.
set -uo pipefail
cd "$(dirname "$0")"
env_value() { grep -E "^$1=" .env | cut -d= -f2-; }
API="$(env_value SERVER_HOST_PROTOCOL)$(env_value API_HOST)"
status=0

curl -fsS --max-time 10 "$API/up" >/dev/null || { echo "API health check failed: $API/up"; status=1; }

bad="$(docker compose --profile standalone ps -a --format '{{.Service}} {{.State}} {{.Health}}' \
  | awk '$1 != "migrate" && ($2 != "running" || $3 == "unhealthy")')"
[ -z "$bad" ] || { echo "Containers not running or unhealthy:"; echo "$bad"; status=1; }

metrics="$(curl -fsS --max-time 10 -H "X-Diagnostics-Key: $(env_value DIAGNOSTICS_API_KEY)" \
  "$API/api/v1/diagnostics/health_metrics")" || { echo "Health metrics unavailable"; status=1; }
echo "$metrics" | grep -q '"up":true' || { echo "ClickHouse is down: $metrics"; status=1; }
echo "$metrics"

exit "$status"
```

```bash
chmod 700 check.sh
./check.sh
```

Run it every few minutes from cron or your monitoring agent, and alert on a non-zero exit. This root crontab entry on a cloud VM keeps a log of every run:

```bash
*/5 * * * * /opt/grovs/check.sh >> /var/log/grovs-check.log 2>&1
```