Retries and timeouts

Retries and timeouts bound how long a step may take and how often the runtime tries it again. A retry block sets the policy for one step, timeout and queue_timeout limit each attempt, and the workflow timeout limits the whole execution. This page lists every setting with its default, and describes exactly when the runtime retries.

Retry policy

step charge {
    worker "payments.charge"
    with { order_id: input.order_id, amount: input.amount, currency: input.currency }
    retry {
        attempts: 5
        backoff: exponential
        initial_delay: 500ms
        max_delay: 10s
        non_retryable: ["CARD_DECLINED", "INSUFFICIENT_FUNDS"]
    }
}

retry is followed directly by { (no :). Inside, each setting is name: value, separated by commas, semicolons or newlines.

A step without a retry block makes one attempt. A retry block that omits a setting uses that setting’s default, so retry { attempts: 3 } retries twice with a fixed 1 second delay.

Every value must be a literal: expressions such as attempts: input.n are rejected. The compiler reports every problem with a retry block as E0410:

Backoff

The delay before attempt n (attempt 2 is the first retry) is:

For attempts: 6, backoff: exponential, initial_delay: 500ms, max_delay: 4s:

The delay is a durable timer: a retry that is waiting survives a runtime restart, and the attempt starts when the timer fires.

When the runtime retries

After an attempt fails, the runtime schedules another attempt only if all of these hold:

1. the error is retryable,
2. the error code is not listed in non_retryable (codes are compared exactly, including case),
3. attempts remain (attempt < attempts),
4. the workflow is not being aborted.

Otherwise the step fails permanently. If the policy allowed more than one attempt and they were all used, the error message ends with (gave up after N attempts).

Whether an error is retryable is decided by whoever reports it. Workers report errors with a code and a retryable flag (see How workers work). Errors that the runtime generates have a fixed classification:

Every retry reuses the input and idempotency key that were evaluated when the step was first scheduled. Redelivery after a lost worker is not a retry and does not use up an attempt; see Delivery guarantees.

Step timeout

timeout limits a single attempt, measured from the moment the task is dispatched to a worker until its result arrives:

step analyze {
    worker "risk.analyze"
    with { customer_id: input.customer_id, amount: input.amount }
    timeout 10s
    retry { attempts: 3, backoff: exponential, initial_delay: 1s }
}

Timeouts are cooperative. The worker is told to stop, through the cancellation list it receives with heartbeat responses, but the runtime cannot force an external operation to stop. If the timed-out attempt later reports success while the step is still unresolved, the runtime accepts that result, because the work did happen.

On a workflow step, timeout limits how long the parent waits for the child. When it expires, the step fails with TIMEOUT (workflow steps cannot retry) and the child execution is cancelled.

On an await step, timeout limits how long the step waits for its signal, from the moment it starts waiting. When it expires, the step fails with SIGNAL_TIMEOUT, which is never retried. See Signals and queries.

Queue timeout

queue_timeout limits how long an attempt may wait for a worker to pick it up:

step charge {
    worker "payments.charge"
    with { order_id: input.order_id, amount: input.amount, currency: input.currency }
    queue_timeout 2m
}

Use queue_timeout to fail fast when no worker for a task (or for the using runtime) is running, instead of waiting until the workflow deadline.

Workflow timeout

The workflow-level timeout limits the whole execution, measured from its creation, across all steps, retries, sleeps and waits for signals. When it passes, the runtime stops scheduling work, waits for in-flight attempts, runs compensations, and ends the execution as TIMED_OUT. See Workflows.

Where each setting is allowed

await steps accept timeout only, and compute steps accept none of the three (E0307): a compute step is evaluated by the runtime and cannot time out or be retried.

A child workflow is durable on its own. Put retries on the child’s steps.

Retries in the history

Each attempt appears in the execution history. A step with retry { attempts: 3, backoff: exponential, initial_delay: 100ms } that fails every time produces events like these (orbit logs <id>):

+   106ms   #9  StepScheduled             step=broken attempt=1
+   201ms  #10  StepDispatched            step=broken attempt=1 worker_id=orbit-runtime
+   203ms  #11  StepAttemptFailed         step=broken attempt=1 error=HTTP_ERROR request failed: io: Connection refused (os error 111)
+   205ms  #12  RetryScheduled            step=broken delay_ms=100
+   401ms  #13  StepScheduled             step=broken attempt=2
+   403ms  #14  StepDispatched            step=broken attempt=2 worker_id=orbit-runtime
+   406ms  #15  StepAttemptFailed         step=broken attempt=2 error=HTTP_ERROR request failed: io: Connection refused (os error 111)
+   407ms  #16  RetryScheduled            step=broken delay_ms=200
+   701ms  #17  StepScheduled             step=broken attempt=3
+   702ms  #18  StepDispatched            step=broken attempt=3 worker_id=orbit-runtime
+   704ms  #19  StepAttemptFailed         step=broken attempt=3 error=HTTP_ERROR request failed: io: Connection refused (os error 111)
+   705ms  #20  StepFailed                step=broken error=HTTP_ERROR request failed: io: Connection refused (os error 111) (gave up after 3 attempts)

See also

• Steps
• Idempotency
• Delivery guarantees
• Executions and history
• How workers work