Observability Stack

Breeze includes an optional observability stack as a separate Docker Compose overlay (docker-compose.monitoring.yml). Enable it alongside the core stack to get full metrics, dashboards, log aggregation, and infrastructure alerting.

docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d

Components

| Component | Port | Purpose | |---|---|---| | Prometheus | 9090 (localhost) | Time-series metrics collection and alerting rules | | Grafana | 3000 (localhost) | Dashboards and visualization | | Alertmanager | 9093 (localhost) | Alert routing and notifications | | Loki | 3100 (localhost) | Log aggregation and querying | | Promtail | 9080 (localhost) | Log shipping from Docker containers to Loki | | Redis Exporter | 9121 (internal) | Exports Redis metrics for Prometheus | | Postgres Exporter | 9187 (internal) | Exports PostgreSQL metrics for Prometheus |

Note

All monitoring ports bind to 127.0.0.1 by default. Access Grafana and Prometheus via SSH tunnel or VPN — they should not be exposed to the public internet.

Accessing Grafana

# Via SSH tunnel
ssh -L 3000:127.0.0.1:3000 user@your-server

# Then open http://localhost:3000
# Username: admin
# Password: (your GRAFANA_ADMIN_PASSWORD from .env.prod)

Tip

You can override the Grafana port with the GRAFANA_PORT environment variable. Similarly, PROMETHEUS_PORT, ALERTMANAGER_PORT, and LOKI_PORT let you change the other monitoring ports.

Pre-Built Dashboards

Breeze ships with a Grafana dashboard (monitoring/grafana/dashboards/breeze-overview.json) that is automatically provisioned. It includes these panels:

| Panel | What It Shows | |---|---| | Service Status | Up/down status of API, Redis, PostgreSQL, and other services | | Request Rate | HTTP requests per second with breakdown by method | | Response Times | P50, P95, and P99 latency over time | | Error Rate | 4xx and 5xx response rates as percentages | | HTTP Status Distribution | Breakdown of responses by status code | | Top Endpoints | Most-used API endpoints by request volume | | Active Devices | Count of agents with recent heartbeats | | Organizations | Number of active tenants | | Redis Memory | Memory usage, evictions, and hit rate | | PostgreSQL Connections | Active connection count vs. max pool size |

Adding Custom Dashboards

To add your own dashboards:

1. Create or import a dashboard in the Grafana UI.
2. Export it as JSON from the Grafana dashboard settings.
3. Save the JSON file to monitoring/grafana/dashboards/.
4. The dashboard provisioner (monitoring/grafana/dashboards.yml) automatically picks up new files in that directory on the next Grafana restart.

Data Sources

Configured automatically via monitoring/grafana/datasources.yml:

| Source | Type | URL | |---|---|---| | Prometheus | Time-series | http://prometheus:9090 | | Loki | Logs | http://loki:3100 | | PostgreSQL | SQL | postgres:5432 | | Redis | Key-value | redis://redis:6379 |

Prometheus Configuration

Located at monitoring/prometheus.yml. Key settings:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s
  external_labels:
    monitor: 'breeze-rmm'
    environment: 'production'

Scrape Targets

| Job | Target | Interval | Auth | |---|---|---|---| | prometheus | localhost:9090 | 15s (default) | None | | breeze-api | api:3001 at /metrics/scrape | 10s | Bearer token | | redis | redis-exporter:9121 | 15s (default) | None | | postgres | postgres-exporter:9187 | 15s (default) | None | | node | node-exporter:9100 (optional) | 15s (default) | None |

The API metrics endpoint is protected by a bearer token. Store the token in monitoring/secrets/metrics_scrape_token and reference it in prometheus.yml:

- job_name: 'breeze-api'
  metrics_path: /metrics/scrape
  authorization:
    type: Bearer
    credentials_file: /run/secrets/metrics_scrape_token

Rule Files

Prometheus loads all .yml files from monitoring/rules/. Breeze ships with breeze-rules.yml containing API, infrastructure, and business alert rules plus recording rules for common aggregations. See Infrastructure Alerts for the full list.

Rules are evaluated every 30 seconds (API and infrastructure groups) or every 60 seconds (business alerts). Recording rules pre-compute expensive queries:

| Recording Rule | Description | |---|---| | breeze:http_requests:rate5m | Request rate by status, method, and route | | breeze:http_error_rate:ratio5m | 5xx error rate as a ratio | | breeze:http_request_duration:avg5m | Average request duration by route | | breeze:http_request_duration:p95_5m | 95th percentile request duration | | breeze:http_request_duration:p99_5m | 99th percentile request duration | | breeze:devices:active_count | Total active device count | | breeze:http_requests:rate5m_by_org | Request rate per organization | | breeze:redis_ops:rate5m | Redis operations per second | | breeze:postgres_query_duration:avg5m | Average PostgreSQL query duration |

Key Metrics

HTTP Metrics (from the API)

| Metric | Type | Description | |---|---|---| | http_requests_total | Counter | Total HTTP requests by method, path, status | | http_request_duration_seconds | Histogram | Request latency distribution | | http_requests_in_flight | Gauge | Currently processing requests |

Business Metrics

| Metric | Type | Description | |---|---|---| | breeze_active_devices | Gauge | Devices with a recent heartbeat | | breeze_active_organizations | Gauge | Organizations with active devices | | breeze_commands_total | Counter | Commands executed, labeled by type | | breeze_alerts_total | Counter | Alerts fired, labeled by severity |

Infrastructure Metrics

| Metric | Type | Description | |---|---|---| | redis_memory_used_bytes | Gauge | Redis memory consumption | | redis_commands_processed_total | Counter | Total Redis commands processed | | pg_stat_activity_count | Gauge | PostgreSQL active connections | | pg_database_size_bytes | Gauge | Database size in bytes | | pg_settings_max_connections | Gauge | PostgreSQL max allowed connections |

Log Aggregation with Loki

Promtail scrapes Docker container logs and ships them to Loki. Loki stores logs for 14 days by default (configurable via retention_period in monitoring/loki-config.yml).

Querying Logs in Grafana

Open the Explore page in Grafana, select the Loki data source, and enter LogQL queries.

Basic Queries

# All API logs
{container="breeze-api"}

# API errors only
{container="breeze-api"} |= "error"

# Structured JSON logs — filter by level
{container="breeze-api"} | json | level = "error"

# Logs from a specific container
{container="breeze-web"}

Filtering and Searching

# Search for a specific device ID
{container="breeze-api"} |= "device_id=abc123"

# Exclude health check noise
{container="breeze-api"} != "/health"

# Filter by HTTP status code in structured logs
{container="breeze-api"} | json | status >= 500

# Rate of errors over time (useful for dashboards)
rate({container="breeze-api"} |= "error" [5m])

Time-Based Queries

# Logs from the last hour containing "timeout"
{container="breeze-api"} |= "timeout"

# Count log lines per minute
sum(rate({container="breeze-api"} [1m])) by (container)

Tip

Use the Live toggle in Grafana Explore to tail logs in real time. This is useful when debugging agent connectivity or watching alert processing.

Adding Custom Prometheus Alert Rules

1. Create a new YAML file in monitoring/rules/ (e.g., monitoring/rules/custom-rules.yml).

2. Define your alert rules following the Prometheus format:

   groups:
     - name: custom-alerts
       rules:
         - alert: HighAgentChurn
           expr: rate(breeze_device_enrollments_total[1h]) > 10
           for: 30m
           labels:
             severity: warning
           annotations:
             summary: "High agent enrollment rate"
             description: "More than 10 new enrollments per hour for 30 minutes"

3. Reload the Prometheus configuration (no restart required):

   curl -X POST http://localhost:9090/-/reload

4. Verify the rule loaded successfully by checking http://localhost:9090/rules in the Prometheus UI.

Data Retention

| Component | Default Retention | Configuration | |---|---|---| | Prometheus | 15 days | --storage.tsdb.retention.time=15d in compose file | | Loki | 14 days (336h) | retention_period in monitoring/loki-config.yml | | Grafana | Unlimited (dashboards only) | N/A | | Alertmanager | Silences and notification log only | --storage.path in compose file |

To change retention, edit the relevant configuration and restart the container.

Troubleshooting

Prometheus Is Not Scraping the API

Symptom: The breeze-api target shows as DOWN in http://localhost:9090/targets.

1. Verify the API is running and healthy: curl http://localhost:3001/health
2. Check the scrape token is correct. Compare monitoring/secrets/metrics_scrape_token with the METRICS_SCRAPE_TOKEN environment variable on the API container.
3. Verify network connectivity. Both Prometheus and the API must be on the same Docker network (breeze).
4. Check Prometheus logs: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs prometheus --tail 50

Grafana Shows “No Data”

Symptom: Dashboard panels show “No data” instead of charts.

1. Confirm Prometheus is running and scraping: visit http://localhost:9090/targets and verify all targets are UP.
2. In Grafana, go to Configuration > Data Sources > Prometheus and click Test. It should say “Data source is working.”
3. Check the time range selector in Grafana. If metrics collection just started, narrow the range to “Last 15 minutes.”
4. If using a custom dashboard, verify the metric names match what Prometheus is collecting. Test a simple query like up in Grafana Explore.

Loki Queries Are Slow

Symptom: Log queries in Grafana take more than 10 seconds or time out.

1. Narrow the time range. Loki performs best with shorter ranges (last 1 hour vs. last 7 days).
2. Add label matchers. {container="breeze-api"} |= "error" is much faster than {job="varlogs"} |= "error" because the label narrows the search before the text filter runs.
3. Check Loki’s compactor. If it has fallen behind, compaction can slow queries: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs loki --tail 50
4. Increase Loki resources if needed. In the compose file, add memory limits and CPU limits that match your server capacity.

Alertmanager Is Not Sending Notifications

Symptom: Alerts fire in Prometheus but no notifications arrive.

1. Confirm Alertmanager is receiving alerts: visit http://localhost:9093/#/alerts and check for active alerts.
2. If no alerts appear, verify Prometheus is configured to send to Alertmanager. Check alerting.alertmanagers in monitoring/prometheus.yml.
3. If alerts appear but notifications are not sent, check the receiver configuration in monitoring/alertmanager.yml. Look for commented-out sections that need to be enabled.
4. Check Alertmanager logs for delivery errors: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs alertmanager --tail 50
5. Verify webhook URLs, API keys, and SMTP credentials are correct. Test Slack webhooks with curl to rule out network issues.

Containers Not Starting

Symptom: One or more monitoring containers fail to start or keep restarting.

1. Check which containers are failing: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml ps
2. Read the logs: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs <container> --tail 100
3. Common causes:
   • Grafana: GRAFANA_ADMIN_PASSWORD not set in .env.prod. The compose file requires this variable.
   • Postgres Exporter: POSTGRES_PASSWORD not set or incorrect. The exporter needs the same credentials as the database.
   • Prometheus: Invalid YAML in prometheus.yml or rule files. Validate with promtool check config monitoring/prometheus.yml.
   • Loki: Permissions issue on the data volume. Loki runs as a non-root user and needs write access to /loki.

Disk Space Growing

Monitoring data can accumulate over time, especially on busy systems.

1. Check volume sizes: docker system df -v | grep -E 'prometheus|grafana|loki'
2. Reduce Prometheus retention: lower --storage.tsdb.retention.time from 15d to 7d in the compose file.
3. Reduce Loki retention: lower retention_period in monitoring/loki-config.yml (e.g., from 336h to 168h).
4. Prune old Docker volumes if containers were previously removed without cleaning up: docker volume prune
5. Restart the affected containers after configuration changes.