Protect production before you repair it.

This runbook defines who can pause, roll back, restore, validate, and reopen AEUC Marketplace crypto services. It also separates a documented recovery plan from a real database restore drill.

Incident / rollback plan

DOCUMENTED

Primary authority: AEUC SUPER_ADMIN / AEUC Empire leadership. Production changes must be deliberate, reversible where possible, and verified before customer access is restored.

1.Confirm the incident and record the exact time, affected service, user impact, and last known good deployment.
2.Freeze risky production actions first: disable public crypto writes, swaps, token creation, or other affected write paths before making infrastructure changes.
3.Keep AEUC Chain data and Marketplace database evidence intact. Do not delete logs, transaction records, or database state while diagnosing.
4.If the Marketplace deployment caused the incident, roll back to the last known green deployment and verify health before reopening writes.
5.If AEUC Chain RPC is unhealthy, keep Marketplace crypto writes disabled while preserving read-only status pages until RPC/validator health returns.
6.If database corruption or data loss is suspected, stop write traffic and restore only into an isolated recovery target first. Never test a restore by overwriting the live production database.
7.After recovery, verify login, Marketplace health, AEUC Chain RPC, validator status, wallet balances, Explorer visibility, token balances, pools, swaps, and EAGLE intelligence before reopening normal access.
8.Record root cause, recovery steps, affected records, final verification, and any preventive code or infrastructure changes before closing the incident.

Database backup / restore drill

AUTOMATED DRILL BUILT

Railway managed backups are not available on the current plan, so AEUC now has a no-cost guarded PostgreSQL recovery-drill script. This readiness item remains READY TO TEST until the script completes successfully against an isolated recovery database.

Built-in command

npm run recovery:drill

Safety guards refuse to run if the source and recovery target resolve to the same database. The target reset also requires the explicit ALLOW_RECOVERY_TARGET_RESET=YES confirmation.

1.Create or choose an isolated non-production PostgreSQL recovery target. Never use the Marketplace production database as the restore target.
2.Set SOURCE_DATABASE_URL to the production source connection and RECOVERY_DATABASE_URL to the isolated recovery database connection.
3.Set ALLOW_RECOVERY_TARGET_RESET=YES only after confirming the recovery target is disposable and non-production.
4.Run npm run recovery:drill from an environment that has PostgreSQL pg_dump and pg_restore installed.
5.The drill creates a logical backup, restores it only into the isolated target, and compares public-table row counts between source and recovery.
6.Review the generated JSON evidence file. Every restored public table must match the source count for the drill to return PASS.
7.Record restore start time, finish time, target identity, validation results, and any errors. Never store database passwords in the evidence file.
8.Destroy or retain the isolated recovery target according to the test plan. Never merge recovery-test changes back into production.

What the automated drill proves

The script performs a real pg_dump of the source, restores with pg_restore into the isolated recovery target, reads every public table from both databases, compares row counts, writes a JSON evidence report, and exits with failure if any table is missing or mismatched. Connection passwords are never written into the evidence report.

Reopen checklist

Before reopening after any serious incident: confirm Marketplace deployment health, database connectivity, AEUC Chain RPC connection, validator block production, signatures enabled, wallet balances, Explorer records, token ledger activity, liquidity pool state, signed swap execution, and EAGLE intelligence. If any critical check is uncertain, keep affected writes disabled.