Cryptic Developers · Engineering notes
Identify the authoritative state
Separate code artifacts, operational data and credentials. A code rollback should not silently roll back financial records. Record which state is authoritative and which copies are only caches or release outputs.
Test a consistent copy
A useful restoration exercise begins with a consistent backup, then restores to a separate location. Validate the recovered structure and representative behaviour before considering a cutover. AWS's reliability guidance treats testing as necessary evidence that a design behaves as intended.
Keep recovery independent
A copy on the same disk is convenient for an accidental file edit. It does not solve loss of that disk. Independence, access controls and retention are separate requirements and need their own verification.
Exercise the runbook
Use the actual recovery steps during a bounded drill. Record missing permissions, dependencies and timing assumptions. A procedure that has never been exercised remains an assumption, even when each configuration file looks reasonable.