Durability

Everything in the tutorial used an in-memory DictStore, so executions vanished with the process. The point of harel is that they don’t have to: an Execution is serializable data, and the runner checkpoints it through a store at every event boundary. Swap the store and the same machine survives crashes and restarts.

The store seam

DurableRunner(store, definitions) takes any ExecutionStore. The bundled backends:

Store	For
DictStore	tests, examples (in-memory, not durable)
SqliteStore	a single machine — durable, zero infrastructure
RedisStore	distributed (atomic Lua CAS on the state-only fast path; WATCH/MULTI for complex commits)
PostgresStore	distributed (UPDATE-WHERE-version CAS)
RqliteStore	distributed SQLite over Raft (HTTP)
MongoStore	distributed document store (single-document atomic update_one CAS)
LibsqlStore	libSQL (Turso’s SQLite fork): a local file, a sqld server, or an embedded Turso replica — same CAS as SQLite. Experimental: the local-file path is tested in-process; the Turso/sqld path is wired but not yet validated against a real account
DynamoDBStore	AWS serverless (conditional-write CAS, atomic TransactWriteItems; pairs with SqsTransport)

The networked ones are optional extras (harel[redis], [postgres], [rqlite], [mongo], [libsql], [dynamodb]) and take an injected client. Selecting one is the only change — the machine and the driving loop are identical. For how each backend lays its data out — tables/keys/collections, the CAS, the outbox/dedupe/timer/trace storage — see the stores reference.

Surviving a restart

Point a SqliteStore at a file, and an execution created by one runner is picked up by another using only its id — the state lives in the store, the id is the handle that crosses the process boundary. A stand-in for “the process died and came back”:

import tempfile
from pathlib import Path

from harel import definition_from_dsl, DurableRunner, SqliteStore, Event

SOURCE = """
event Finish {}

machine order {
  initial Cart
  state Cart {}
  final Done success {}
  from Cart to Done on Finish
}
"""

defn = definition_from_dsl(SOURCE, "order")
db = str(Path(tempfile.mkdtemp()) / "stm.db")

# process #1: a runner creates the execution (persisting it) and then "crashes".
# create() returns the Execution; its `.id` is the ONLY handle that outlives the
# process — you keep it (in a URL, a queue message, another table) to refer back.
execution_id = DurableRunner(SqliteStore(db), {defn.id: defn}).create(defn.id).id

# process #2: a brand-new runner — shares nothing with the first but the file —
# reloads that execution by id from the store and drives it to completion.
exe = DurableRunner(SqliteStore(db), {defn.id: defn}).process(execution_id, Event(kind="Finish"))
print("survived restart ->", exe.active_path, "/", exe.outcome)

survived restart -> Done / success

What “durable” guarantees

The store is hardened beyond just persisting state:

• Optimistic concurrency — every record carries a version; a compare-and-set on write raises StoreConflict if another writer got there first (the single-writer backstop).

• Transactional outbox — a commit saves the execution and its emitted events in one transaction; a relay delivers them after commit, so a crash can never lose a Finished.

• Dedupe — processed event ids are recorded, so at-least-once delivery takes effect once.

• Crash-safe orthogonal fork — region children are created from a durable spawn-outbox, idempotently, so a crash mid-fork neither double-spawns nor loses a region’s result.

• Durable timers — a timeout is persisted on enter and cancelled on exit in the same commit as the transition, so a scheduled timer can’t be lost (see step 7).

• Execution trace (opt-in) — with STM_TRACE=1 (or DurableRunner(..., trace=True)) the commit also appends one timeline step (event, transition, actions, context_out) in the same transaction, so the monitor timeline is real. Off by default (zero hot-path cost); ring-capped to the last STM_TRACE_MAX steps. Per-backend storage in the stores reference.

At-least-once actions & idempotency

Delivery is at least once. The dedupe above stops a redelivered event from re-running once its prior attempt committed — but if a worker crashes after an action ran and before the commit, the event is redelivered and the action runs again. Dedupe is per event, not per action.

For a side-effecting action (charge a card, send an email — local or a remote FaaS function) the driver exposes a stable key, stm.idempotency_key = {execution_id}:{version}:{index}. It is deterministic — the pure engine reproduces the same action sequence and the version is the pre-commit value — so a redelivery hands each action the same key.

The dedupe must live in an external backend you own (Redis SET NX, a DynamoDB conditional put, a service’s native idempotency key) — not in harel’s store or context. The gap is a crash before the commit, so anything harel recorded would roll back with that failed commit; only a record the callee wrote outside harel’s transaction survives. harel.idempotency ships the opt-in helper:

# docs-test: skip
from harel import idempotent, DictIdempotency  # DictIdempotency is in-memory (tests)

backend = my_redis_backed_idempotency()         # an IdempotencyBackend you supply
actions = {"charge": idempotent(backend)(charge)}  # bind the wrapped action

idempotent runs the body at most once per stm.idempotency_key (caching the result, so a selector still routes the same way). Residual window: true exactly-once needs the effect and the claim to be atomic (an idempotency-key-native service); the helper narrows the window, it doesn’t abolish it.

Retry & backoff is a composite, not a feature

Because timers are durable and the model decides what a Timeout does, retry-with-backoff is assembled, not configured:

• a Waiting state whose delay is read from context — timeout context backoff — so an on enter can compute the next wait;

• a select that branches succeeded / retry again;

• the composite’s own timeout as the overall budget, whose Timeout exits to a failed terminal.

harel ships composable backoff actions in harel.lib — exponential_backoff, linear_backoff, reset_backoff — ordinary inputs-parameterised actions that compute the next delay. Policy stays in the model; the engine just keeps time. The repository’s retry.stm / charge_retry.stm are the complete pattern.