Penelope

Phase 4 — three things you get when a paused program is just data.

What changed in Phase 4

The earlier phases established the basics: bytecode, pause/resume, snapshots, modules, a stack-based VM. Phase 4 cashes in the dividend. Because a paused program is just a value (a Snapshot: ip, value stack, frames, effects), three useful things fall out:

Live editing — pause a running program, edit the source, resume into the new code with the old state mapped over.
Time-travel debugging — step backward through a paused program from your IDE.
Self-hosting — write the lexer, parser, and compiler in Penelope itself, and verify byte-for-byte that the pen-built compiler produces the same bytecode as the TS-built one.

4.1 — Live editing (`pen edit`)

You paused at line 12. You realize line 14 has a bug. Without phase 4, you'd start over. With phase 4:

bin/penelope run buggy.pen          # hits pause, writes snapshot.penz
# … edit buggy.pen to fix line 14 …
bin/penelope edit snapshot.penz     # resume on the new source

Behind the scenes src/live-edit.ts recompiles the new source, then walks the old VMState: for each ip in the value stack, frames, return addresses, and pending effects, it looks up the corresponding source position and finds the matching opcode in the new program. If the opcode kinds don't match (e.g., old PAUSE → new LOAD_CONST), the remap is rejected so semantic drift can't silently corrupt the program. If everything lines up, the new state runs on the new bytecode.

This isn't hot-reload-the-module. It's resume-the-exact-paused-frame against new code.

4.2 — Time-travel debugger

The DAP adapter (src/dap.ts) snapshots VMState before every advance and keeps a bounded history (1000 entries). Two new requests:

stepBack — pop one entry and restore.
reverseContinue — pop entries until a breakpoint reappears.

The capability is advertised as supportsStepBack: true, so VSCode shows the reverse-step buttons automatically. Implementation cost: deep-clone VMState as JSON. Cheap because debug sessions are short.

4.3 — Self-hosting (full language)

The TS compiler is <2000 lines. Reimplementing it in Penelope is a real test of whether the language is good enough to host its own toolchain. Four files in std/ do the work:

File	Lines	Responsibility
`std/iter.pen`	~70	`list_map`, `list_filter`, `list_reduce`, etc. — the HOFs the compiler needs.
`std/lexer.pen`	~340	Tokenizes a source string. Handles template strings (`TEMPLATE_STRING` tokens with `parts`), `DOTDOTDOT`, `PIPE`, and the `\$` escape.
`std/parser.pen`	~500	Pratt-style infix parser, produces `{rootId, nodes}` AST bundle. Pattern kinds: literal / var / wildcard / unit / or / list (with `...rest`) / dict. Guard arms (`pat if cond => body`). Template-string desugaring to `+` chains.
`std/compiler.pen`	~520	AST → `{constants, code}`. Tail-call optimization (CALL → TAILCALL in tail position). Full pattern compilation including or-pattern chaining, list-pattern length+items+rest-bind, and dict-pattern `dict_has` + recursive sub-check.

The whole thing was painful in instructive ways. Penelope (as it stood) had no unary !, didn't allow else if, couldn't use a block as a primary expression in match arms. Each gap surfaced as a parse error in the pen-side code and was either worked around or fixed in the TS frontend. The result is a more honest language.

Bootstrap correctness — three-stage `pen self-test`

"Self-hosting" is easy to overclaim. pen self-test verifies three increasingly strong properties in sequence:

Stage 1 — round-trip. For a battery of source samples (including template strings, or-patterns, guards, list patterns), the pen frontend emits the same bytecode as the TS frontend.
Stage 2 — self-bootstrap. The pen frontend's own source (std/iter.pen, std/lexer.pen, std/parser.pen, std/compiler.pen) round-trips. Pen-compiling pen-frontend produces byte-identical bytecode to ts-compiling pen-frontend.
Stage 3 — acid test (fixpoint). Take the pen-built pen-frontend bytecode. Actually run it. Have it compile a source. Verify the resulting bytecode matches what TS produces directly. This proves the pen-built bytecode isn't just byte-identical — it's runnable and behaviorally correct.

$ bin/penelope self-test

Stage 1: round-trip on inline samples
  ✓ "let x = 42;"
  ✓ "let f = fn(n) { n + 1 }; print(to_str(f(41)));"
  ✓ "let r = if (1 < 2) { \"yes\" } else { \"no\" };"
  ✓ "let r = match 1 { 1 | 2 => \"small\", n if n > 100 => \"big\", _ => \"mid\" };"
  ✓ "let n = 7; print(\"sum 1..\${n} = \${n * (n + 1) / 2}\");"

Stage 2: round-trip on std/*.pen (self-bootstrap)
  ✓ std/iter.pen
  ✓ std/lexer.pen
  ✓ std/parser.pen
  ✓ std/compiler.pen

Stage 3: acid test — pen-built pen-frontend compiles a program
  ✓ "let x = 42;"
  ✓ "let f = fn(n) { n + 1 }; print(to_str(f(41)));"
  ✓ "let r = if (1 < 2) { \"yes\" } else { \"no\" };"

12/12 checks passed — Penelope self-hosts ✓

The extended test suite (test/self-host-roundtrip.test.ts + test/self-host-acid.test.ts) covers 19 cases end-to-end, including all four std/*.pen files and two real examples that use template strings and nested match.

The fixpoint diagram

             ts_compile
   source ─────────────────▶ bytecode_A
      │
      │  pen_compile
      ▼
   bytecode_B
      │
      │  pen_compile (run as bytecode)
      ▼
   bytecode_C

   self-test verifies:  bytecode_A ≡ bytecode_B ≡ bytecode_C

Stage 1 proves A ≡ B. Stage 2 proves the pen frontend's own source round-trips. Stage 3 proves B ≡ C — i.e., when the pen frontend is ITSELF compiled by the pen frontend, the resulting bytecode still computes the same answers.

Test count

557 tests across 50 files, including new suites for live editing, reverse stepping, the stdlib expansion, full pen lexer/parser/compiler comparisons (15 samples + 4 std files), and the 4 acid tests that prove pen-built bytecode is runnable.