Skip to lesson
中文
The collection/AN OPEN STUDY ROUTE

WEEK 16 / 24 · Learn, then operate

After the interruption

Recover a run: persistent state, retries, idempotency and backups

10 core hours + 2 optionalREAD · BUILD · PRACTISE
Begin the lesson ↓
After the interruptionTHE PERSONAL FDE FIELD LIBRARY

The question: What happens when a connection drops, a task triggers twice or the process restarts halfway through?

Before you begin: Have a controlled, manual, read-only runtime; locate a request and stop its process.

FIG. 16.1THIS WEEK’S REASONING PATH
  1. 01Persist pending
  2. 02Claim a run
  3. 03Check completion
  4. 04Recover or review
Teaching diagram · Derived from this week’s lab; a suggested process, not observed results.

Concepts and learning objectives

Model pending, running, succeeded, failed, awaiting-human and cancelled states. Store them durably for the chosen environment; memory and temporary disks may disappear on restart. Side effects have uncertainty windows before and after execution, so retrying cannot simply repeat everything blindly.

Idempotency means repeats do not create additional unwanted effects. Generating an ID alone does not guarantee exactly-once execution. When an external action's outcome is uncertain, inspect the target system or involve a person. Local SQLite may be sufficient; a new orchestration framework is optional.

Reading and selected sections

Priority Source and selected sections Budget
Optional R49 · LangGraph JavaScript — Persistence — Persistent state, thread or run identity and recovery concepts. 40 min
Required selections R68 · Google SRE — Effective Troubleshooting — Hypothesis-driven investigation of interruptions and retries. 35 min
Hosted option R72 · Render — Cron Jobs — Scheduling, run limits and stopping; do not assume persistent-disk support. 35 min

Studio lab

  1. Assign non-personal run IDs and persist state and minimal results. Distinguish identical triggers, repeated input and genuinely new tasks.
  2. Add bounded retries, backoff, total timeout and cancellation. Retry only suitable errors; malformed data and denied permission must not loop forever. Count every retry against the budget.
  3. Mock interruption, timeout, duplicate triggers and a result returned before its state was saved. Avoid duplicate drafts and route uncertain completion to human checking.
  4. Verify filesystem persistence and service limits. Scheduled jobs need a suitable durable data layer; check R72 and R73 independently.
  5. Restore a synthetic backup into a new location and compare counts and key fields. Recovery must not replay external actions automatically.

Deliverables and acceptance

Deliverable Evidence of completion
Durable state and deduplication State survives restart and repeated triggers have tested behaviour.
Failure drill Timeout, interruption, duplication and uncertain completion are covered.
Restorable backup An actual restore, stated recovery point and loss limitations.

Independent exercise: Trigger the same task concurrently and inspect which executions acquire permission. Explain why disabling a UI button does not solve server concurrency.

Privacy and safety: Protect backups and state tables with access and retention controls. Minimise logs; see R61. Stopping new work does not automatically cancel queued or already-sent actions.

Everyday use: If ready, schedule a low-frequency, finite read-only task. Skip inference when nothing changes and make failure notifications actionable.

Pass or revisit: Recover, deduplicate and stop, and know how to handle uncertain outcomes before attempting approved writes.

A reusable collaboration prompt

Add persistent state, deduplication, bounded retry and cancellation to a read-only task. Mock interruptions and repeats without promising exactly-once external execution. Distinguish temporary from durable storage and actually restore a synthetic backup.

Sources and verification scope

Reference IDs and scope follow the supplied manuscript, baseline 2026-09-27. External authenticated labs were not all completed; recheck versions and access before implementation.

  1. R49
    LangGraph JavaScript — Persistence ↗

    Optional persistent state; a simpler implementation may suffice.

  2. R61
    OWASP — Logging Cheat Sheet ↗

    Necessary logs and sensitive-data exclusion.

  3. R68
    Google SRE — Effective Troubleshooting ↗

    Fault diagnosis and hypothesis testing.

  4. R72
    Render — Cron Jobs ↗

    Optional scheduling; recheck runtime, billing and storage support.

  5. R73
    Render — Persistent Disks ↗

    Support varies by service type; do not treat temporary filesystems as databases.