Penelope

Distributed pause/resume — Phase 5.A.

The thesis, cashed in

If a paused program is just a value, then "pause on machine A, resume on machine B" is a transport problem, not a runtime problem. Phase 5.A turns that observation into a concrete distributed runtime:

A coordinator process tracks jobs and worker leases.
Workers poll the coordinator, claim jobs, run them on the VM, report results.
If a worker dies mid-job, its lease expires and the coordinator hands the job to another worker. The VM is deterministic on replay (effect log), so re-execution produces the same outcome.

This is the same model Temporal/Inngest/Restate use — but where they require you to structure code around activity/step/await boundaries, Penelope lets you just write code, and any pause in the source can travel across processes.

The architecture

┌─────────────┐       submit       ┌────────────────────┐
│   client    │ ─────────────────→ │   coordinator      │
│ (pen submit)│ ←── jobId ───────  │   (single node)    │
└─────────────┘                    │                    │
                                   │  jobs (file store) │
                                   │  workers/leases    │
                                   └──┬─────────┬───────┘
                            poll/heartbeat (HTTP)
                                ┌──────┴─────┬─────────────┐
                                ▼            ▼             ▼
                          ┌─────────┐  ┌─────────┐   ┌─────────┐
                          │ worker  │  │ worker  │   │ worker  │
                          │   A     │  │   B     │   │   C     │
                          └─────────┘  └─────────┘   └─────────┘

Running it

# start the coordinator (persistent job store)
$ bin/penelope coordinator --port 7077 --store ./jobs --lease-ms 5000
pen coordinator listening on http://localhost:7077  store=./jobs  leaseMs=5000

# start a worker (in another shell)
$ bin/penelope worker --coord http://localhost:7077 --id worker-1
pen worker worker-1 → http://localhost:7077

# submit a job
$ bin/penelope submit my-workflow.pen --coord http://localhost:7077 --wait
submitted job 31e0c4f1-491d-4a7a-9ab7-050d857330c1
status: completed
bindings: {"result":{"tag":"int","v":42}}

Failure recovery

The coordinator periodically sweeps for expired leases. A worker that:

holds a job lease, and
hasn't heartbeated within leaseMs, AND
claimed the job more than leaseMs ago

…is treated as dead. Its job is marked pending again and the next poll from any worker picks it up. Since the snapshot captures complete VM state and the effect log makes replay deterministic, the second worker's execution produces the same observable behavior.

HTTP protocol

Endpoint	Body	Response
POST /workers/register	{workerId}	{ok}
POST /workers/heartbeat	{workerId}	{ok}
POST /jobs/submit	{program, state}	{jobId}
POST /jobs/next	{workerId}	{jobId, program, state} or {empty: true}
POST /jobs/:id/complete	{workerId, result}	{ok}
POST /jobs/:id/fail	{workerId, error}	{ok}
GET /jobs/:id	—	JobRecord
GET /health	—	{ok, workers, jobs}

What's out of scope

Single-coordinator: no Raft, no replication. If the coordinator dies, in-flight jobs survive (FileStore persists), but submission is blocked until the coordinator restarts. HA-ing the coordinator is genuinely a separate project (consensus, leader election) — the protocol here is correct, just not redundant.

Also out of scope: authentication (the API is unauthenticated), sharding (one coordinator owns all jobs), back-pressure on submission (the queue is unbounded).