Troubleshooting

Deployment Issues

Deploy script fails with “Missing required env var”

Cause: Required variables not set in .env.prod.

Fix: Ensure all required variables are set:

# Required variables:
BREEZE_DOMAIN, ACME_EMAIL, POSTGRES_PASSWORD, JWT_SECRET,
AGENT_ENROLLMENT_SECRET, METRICS_SCRAPE_TOKEN, PUBLIC_API_URL,
GRAFANA_ADMIN_PASSWORD

API refuses to boot: RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS must be set in production

Cause: This variable is the trust anchor that the API uses to verify signed release manifests before serving binaries (so a compromised GitHub release alone can’t deliver tampered binaries). It is required in production for both BINARY_SOURCE=github and BINARY_SOURCE=local. Older .env templates shipped it empty, so upgrades can surface this on first boot.

Fix: For the official Breeze releases, use the published trust anchor:

RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS=yzx8ftmcls6uBetFC5SYnZhBo+cbur3IX50TbBthTso=

This is a public key (also embedded in the agent and CI), so it is safe to commit. Only change it if you build and sign your own binaries. BREEZE_RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS is accepted as an alias.

Do not auto-rotate or overwrite this value. It is a deliberately pinned trust anchor — automatically replacing it with a “new” key defeats the protection, since the source of that key becomes the thing an attacker targets. If a real signing-key rotation ever ships, trust both keys during the overlap by comma-separating them (RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS=old,new) rather than replacing.

TLS certificate not provisioning

Cause: DNS not pointing to server, or ports 80/443 blocked.

Fix:

1. Verify DNS: dig +short breeze.yourdomain.com
2. Check ports: sudo ss -tlnp | grep -E ':(80|443)'
3. Check Caddy logs: docker logs breeze-caddy

Database migration fails

Cause: PostgreSQL not ready or connection string incorrect.

Fix:

# Check PostgreSQL is running
docker compose -f docker/docker-compose.prod.yml exec postgres pg_isready

# Check connection string
docker compose -f docker/docker-compose.prod.yml exec api env | grep DATABASE_URL

Agent Issues

Agent not appearing in dashboard

Causes:

1. Enrollment failed
2. WebSocket connection blocked
3. Agent can’t resolve server hostname

Fix:

# Check agent logs
sudo journalctl -u breeze-agent -n 50

# Test connectivity from agent
curl -v https://breeze.yourdomain.com/health

# Verify enrollment
sudo cat /etc/breeze/config.json

Agent shows “offline” despite running

Cause: WebSocket connection dropping or heartbeat not reaching API.

Fix:

1. Check agent logs for reconnection messages
2. Verify API WebSocket handling: docker logs breeze-api | grep "ws"
3. Check if a firewall is terminating long-lived connections (increase WebSocket timeout)

Terminal not connecting

Common causes:

• Browser sends resize before server onOpen completes
• WebSocket messages rejected by validation

Fix: Check API logs for WebSocket errors. The terminal uses term- prefixed command IDs that bypass database lookup.

Performance Issues

High API response times

Check:

# Prometheus metrics
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:9090/api/v1/query?query=histogram_quantile(0.95,rate(http_request_duration_seconds_bucket[5m]))

# Database slow queries
docker compose -f docker/docker-compose.prod.yml exec postgres \
  psql -U breeze -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"

Redis memory growing

Check:

docker compose -f docker/docker-compose.prod.yml exec redis redis-cli info memory

Fix: Adjust REDIS_MAXMEMORY in .env.prod. Redis uses allkeys-lru eviction by default.

Disk space running low

# Check Docker volumes
docker system df

# Prune unused images
docker image prune -a

# Check backup retention
ls -la /var/backups/breeze/

Container Issues

Container won’t start

# Check container logs
docker compose -f docker/docker-compose.prod.yml logs <service-name>

# Check resource limits
docker stats --no-stream

# Recreate container
docker compose -f docker/docker-compose.prod.yml up -d --force-recreate <service-name>

Health check failing

# Manual health check
docker compose -f docker/docker-compose.prod.yml exec api \
  wget --no-verbose --tries=1 --spider http://localhost:3001/health

# Check readiness
curl http://localhost:3001/health/ready

Installer Issues

”Generate Link” fails with Server URL not configured (set PUBLIC_API_URL or API_URL)

Cause: The API container can’t see a PUBLIC_API_URL or API_URL env var, so it has no base URL to mint the installer share link against. The bundled docker-compose.yml derives this from BREEZE_DOMAIN (since #613 / v0.65.7+), so this only bites custom or older compose files.

Fix:

1. Set PUBLIC_API_URL=https://<your-domain> in /opt/breeze/.env (or wherever your compose .env lives).

2. If you use a custom compose file, explicitly map the variable into the api service. Docker Compose only forwards env vars to a container when they appear in that service’s environment: block — having the value in .env is necessary but not sufficient.

   api:
     environment:
       PUBLIC_API_URL: ${PUBLIC_API_URL}

3. docker compose up -d api to recreate the container.

”Generate Link” fails with MSI build pipeline not reachable. Retry

Cause: The release-manifest verifier is rejecting the live MSI because the manifest’s repository field is LanternOps/breeze but the code default expects lanternops/breeze (case mismatch). Fix is on main (#867) and will ship in the next release after v0.67.0.

Fix (workaround on v0.67.0): add to .env and map into the api service.

BINARY_GITHUB_REPOSITORY=LanternOps/breeze

api:
  environment:
    BINARY_GITHUB_REPOSITORY: ${BINARY_GITHUB_REPOSITORY}

Then docker compose up -d api. Same workaround unblocks Download Installer, which hits the same code path silently.

Getting Help

• Check the GitHub Issues
• Join the community Discord
• Review API logs: docker compose -f docker/docker-compose.prod.yml logs -f api