Retries and backoff

Handlers fail: a service is down, a deploy is half rolled out, a payload is wrong. Sextant retries failed attempts according to the queue’s policy and records every attempt.

The default policy

[queues.default]
max_attempts = 5
timeout = "5m"

[queues.default.backoff]
strategy = "exponential"
base = "5s"
factor = 2
max = "1h"
jitter = 0.2

With these values the waits before attempts 2 to 5 are about 5 s, 10 s, 20 s and 40 s, each moved by up to 20 % in either direction.

Why jitter matters

When a dependency goes down for a minute, hundreds of jobs fail together. Without jitter they all return together, and the dependency falls over again. A spread of 20 % turns the burst into a ramp. Leave jitter on unless you have a reason.

Errors that should not be retried

Some failures are permanent: a payload that does not validate, a customer that was deleted. Retrying only wastes attempts. Tell Sextant to stop by throwing the SDK’s permanent error.

import { defineHandler, PermanentError } from '@sextant/node';

export default defineHandler('send-receipt', async (job) => {
  const order = await orders.find(job.payload.orderId);
  if (!order) throw new PermanentError(`order ${job.payload.orderId} does not exist`);
  await mailer.send(order.email, receipt(order));
});

from sextant import handler, PermanentError

@handler("send-receipt")
def send_receipt(job):
    order = orders.find(job.payload["order_id"])
    if order is None:
        raise PermanentError(f"order {job.payload['order_id']} does not exist")
    mailer.send(order.email, receipt(order))

func SendReceipt(ctx context.Context, job *sextant.Job) error {
	order, err := orders.Find(ctx, job.Payload["order_id"].(string))
	if errors.Is(err, orders.ErrNotFound) {
		return sextant.Permanent(fmt.Errorf("order does not exist: %w", err))
	}
	if err != nil {
		return err
	}
	return mailer.Send(ctx, order.Email, receipt(order))
}

A permanent error moves the job straight to dead. For command handlers, exit code 78 has the same meaning.

The dead state

A job that uses all its attempts becomes dead and stays there until someone looks at it. List them and retry what you have fixed:

sextant jobs list --state dead --queue emails
sextant jobs retry job_8f3k2m9q

A job.dead webhook is the right place to page a human.

Idempotency under retries

A retried job may repeat work that half succeeded. Pass the job id to the services you call (most payment and email APIs accept an idempotency key) so that attempt three does not charge the card a second time.