Monitoring and alerting

This guide defines the minimum production signals, dashboards, alerts, and response links for LakeHold. It starts with conservative thresholds. Tune them from measured traffic, but retain alerts for availability, data-protection jobs, capacity, and security even on a quiet deployment.

Signal sources

Set OTEL_EXPORTER_OTLP_ENDPOINT to a collector reachable from the API. The production Compose file sets OTEL_SERVICE_NAME=lakehold-api; an unset endpoint exports nothing. Compose includes no collector or alert manager.

LakeHold emits logs, metrics, and traces through OpenTelemetry. EF instrumentation strips db.statement and db.query.text, and LakeHold spans never record submitted SQL, credentials, metadata sources, or data paths. Do not re-add those fields in collectors or processors.

Health probes

Externally through the private workbench ingress:

curl --fail --silent --show-error \
  "http://127.0.0.1:${LAKEHOLD_PORT:-8080}/health"

Internally:

docker compose -f compose.production.yaml exec api \
  curl --fail --silent --show-error http://127.0.0.1:5200/health
docker compose -f compose.production.yaml exec api \
  curl --fail --silent --show-error http://127.0.0.1:5200/alive

Both return only Healthy or Unhealthy and disable caching. Keep them unauthenticated for probes, but block both at public ingress.

• Readiness /health: includes control-plane. Remove the node from traffic when it fails.
• Liveness /alive: includes only the in-process self check. Restart the process when it fails. Do not add external dependencies; that would turn dependency failure into a restart loop.

Use a 10–15 second interval, 5 second timeout, and at least three consecutive failures before paging. The API container itself uses 15 seconds, 5 seconds, a 45-second start period, and five retries.

LakeHold metrics

The canonical OpenTelemetry instrument names are below. A Prometheus exporter may normalize dots to underscores and append unit or counter suffixes; verify the actual backend names after deployment.

Metric tags are deliberately bounded. Tenant and catalog are span attributes (lakehold.tenant, lakehold.catalog), not metric dimensions, to prevent per-tenant time-series explosion. Find an affected tenant through sampled error/slow traces after a metric identifies the class of failure.

Required dashboards

Availability and requests

• ingress and API request rate, status class, p50/p95/p99 duration;
• /health and /alive status by node;
• container status, restart count, OOM kills, and deployed image digest;
• control-plane connectivity failures;
• current incident and last deployment annotation.

Query and session behavior

• lakehold.query.duration by outcome;
• query error and truncation rate;
• p95 lakehold.session.queue.duration beside p95 query duration;
• p95 lakehold.session.start.duration;
• warm sessions against Lakehouse:MaxWarmSessions;
• pool miss rate and evictions by reason;
• host CPU, API/container memory, filesystem I/O, and object-store latency.

Data protection and maintenance

• last successful flush and backup per catalog;
• failed scheduled runs and incomplete backup generations;
• backup recovery point, age, size, manifest completeness, off-host copy, checksum, and retention;
• maintenance duration and lease outcomes by operation;
• state-volume use and forecasted exhaustion;
• last successful catalog and full-node restore drill.

The schedule endpoint alone cannot power this dashboard because it is in-memory and bounded. Export event 2000 failures and maintenance metrics, and inventory manifests from outside the API.

Security and integration

• HTTP 401/403 rate and unusual changes from baseline;
• wire authentication failures and refused connections;
• token creation, revocation, expiry, and break-glass use from the approved audit source;
• CDC subscriptions with consecutive failures, last attempt/error, and cursor lag;
• TLS/certificate and external credential expiry;
• public exposure checks for API, MCP, PostgreSQL wire, /health, and /alive.

Initial alert policy

Every page must include service, environment, severity, first observed time, current value, threshold, dashboard link, and a link to the exact incident-response section. Deduplicate by failure domain so a control-plane outage does not page separately for every failed API route.

For a low-traffic deployment, ratio alerts alone are noisy: one failed request can be 100%. Always combine a ratio with a minimum count, or use an absolute failure count and longer window.

The five-second queue and 20% pool-miss values are starting points, not product limits. Replace them after establishing a 30-day baseline and service objectives. Do not tune away backup, volume, readiness, OOM, or security alerts merely because they fired correctly.

Logs and events

The production containers use Docker's json-file driver with three 10 MB files each. This protects the host from unbounded logs but is not sufficient retention. Export logs before rotation.

Important scheduler events:

Alert on event 2000. Inventory 2001 at deployment so an unexpected cadence is detected. Treat 2002 as an alert in production unless a recorded change explicitly disabled schedules.

Log searches and exported fields must redact:

• bootstrap and bearer tokens;
• PostgreSQL and object-store credentials or connection strings;
• webhook secrets and signed headers;
• eject signing keys;
• user SQL and result data.

Backup monitoring

A successful HTTP response or scheduler log is not enough. For every writable catalog, verify:

1. a generation exists within the expected interval;
2. complete is true, which means manifest.json exists and parsed;
3. the manifest timestamp and snapshot match the intended recovery point;
4. the generation is copied outside the primary failure domain where required;
5. the off-host object/archive checksum or provider integrity check passes;
6. retention keeps enough generations without deleting the only proven recovery point;
7. a representative generation has been restored within the drill window.

The default production host currently puts backup generations in the state volume. Monitor the off-host state archive separately; a green catalog-backup job with no off-host copy still leaves whole-volume-loss RPO unbounded.

Alert validation

At installation and at least quarterly:

• stop a disposable API and prove readiness/liveness pages reach the on-call route;
• make a disposable control plane unavailable and prove readiness fails without a restart storm;
• induce a test scheduler failure and confirm event 2000 plus the backup alert;
• fill a disposable filesystem past warning and critical thresholds;
• generate rejected authentication attempts through an isolated test endpoint;
• fail a test CDC receiver and verify consecutive-failure/cursor-lag alerting;
• complete a restore drill and verify the dashboard records its RPO/RTO.

Record alert creation time, notification time, acknowledgement, runbook usability, and cleanup. Fix missing labels, links, ownership, and deduplication before calling the monitoring gate complete.