Observability · Sextant docs

Sextant exposes what it knows in three forms: metrics for dashboards and alerts, logs for debugging, and a timeline per job for support questions.

Metrics

Enable the Prometheus endpoint in the config:

sextant.tomlTOML

[metrics]
listen = "0.0.0.0:9464"

and scrape http://host:9464/metrics.

Metric	Type	Labels	Use it for
`sextant_jobs_total`	counter	`queue`, `state`	Throughput and failure rate
`sextant_job_duration_seconds`	histogram	`queue`, `handler`	Slow handlers
`sextant_queue_wait_seconds`	histogram	`queue`	Workers falling behind
`sextant_queue_depth`	gauge	`queue`, `state`	Backlog
`sextant_workers`	gauge	`queue`	Capacity
`sextant_schedule_skipped_total`	counter	`schedule`	Overlapping runs

Three alerts to start with

alerts.ymlYAML

groups:
  - name: sextant
    rules:
      - alert: SextantQueueWaitHigh
        expr: histogram_quantile(0.95, sum by (le, queue) (rate(sextant_queue_wait_seconds_bucket[5m]))) > 60
        for: 10m
        annotations:
          summary: "Jobs on {{ $labels.queue }} wait over a minute (p95)"

      - alert: SextantDeadJobs
        expr: increase(sextant_jobs_total{state="dead"}[15m]) > 0
        annotations:
          summary: "Jobs became dead on {{ $labels.queue }}"

      - alert: SextantNoWorkers
        expr: sextant_workers == 0 and sextant_queue_depth{state="queued"} > 0
        for: 5m

Logs

The server and workers write one JSON object per line to standard output:

JSON

{"ts":"2026-10-20T09:15:03.214Z","level":"warn","msg":"attempt failed","job":"job_8f3k2m9q","queue":"emails","handler":"send-receipt","attempt":2,"error":"SMTP 451: mailbox busy","next_attempt_in":"20s"}

Set level and format under [logging]. Use format = "text" on your laptop and json everywhere else.

Shell

sextant logs --follow --queue emails --level warn

The timeline of one job

sextant inspect answers “what happened to this job?” in one screen:

Shell

$ sextant inspect job_8f3k2m9q
job_8f3k2m9q  send-receipt  emails  dead
created   2026-10-20 09:14:58  by tok_71c0de (ci-deploy)
attempt 1  09:15:00  wrk_a91c7e  212 ms  error  SMTP 451: mailbox busy      then wait 5s
attempt 2  09:15:05  wrk_a91c7e  190 ms  error  SMTP 451: mailbox busy      then wait 11s
attempt 3  09:15:16  wrk_3c0d92  187 ms  error  SMTP 451: mailbox busy      then wait 19s
attempt 4  09:15:35  wrk_a91c7e  201 ms  error  SMTP 451: mailbox busy      then wait 43s
attempt 5  09:16:18  wrk_3c0d92  195 ms  error  SMTP 451: mailbox busy      dead

Tracing

Sextant propagates a W3C traceparent from the request that creates a job to the handler that runs it. Pass it on in your own outgoing calls, and one trace covers the request, the wait in the queue and the handler. Set SEXTANT_TRACE_HEADER=0 to disable.