Operations: backup, restore, recovery, and upgrade

Everything on this page runs on your own ERPNext site. None of it calls Helpifyr services. An integration outage is an operations event, not a failure of the app: the recovery objectives below apply unchanged.

Daily mandatory actions

Action	Command	Expected result
Backup	bench --site SITE_NAME backup --with-files --compress	Backups in sites/SITE_NAME/private/backups, exit code 0
App status	bench --site SITE_NAME list-apps	insurance_broker_core is listed
Readiness	bench --site SITE_NAME execute insurance_broker_core.api.activation.readiness --kwargs COMPANY_KWARGS	ready: true, no findings
Outbox	python scripts/verify_outbox.py --check (run in a checkout of the source repository)	PASS outbox

Behaviour is fail-closed: a missing attestation field makes readiness return IBC-ACT-REQ-001; an outbox backlog blocks uninstall (IBC-UNINSTALL-006); restore is only done through the documented drill.

Recovery objectives

The project does not set blanket RPO and RTO figures. Every production installation needs its own Business Impact Analysis (BIA) in which the operator defines acceptable data loss (RPO) and restart target (RTO) per data class, approved by the owner. The BIA is a prerequisite of the activation gate through the attestation fields of the Company Settings (backup_approval_ref, security_baseline_ref, test_export_ref, test_import_ref). See the BIA guide.

Backup

Before every migration, upgrade, or uninstall:

cd BENCH_PATH
bench --site SITE_NAME backup --with-files --compress
bench --site SITE_NAME list-apps

Copy the database backup, the public and private file backups, site_config.json, the app, Frappe, and ERPNext revisions, and the external Disposition Ledger to your approved backup store. Verify encryption, retention, and restore permissions.

Restore drill

Never restore blindly over a production site. Restore into an isolated target site first:

bench new-site RESTORE_SITE_NAME --db-root-password DB_ROOT_PASSWORD --admin-password ADMIN_PASSWORD
bench --site RESTORE_SITE_NAME restore DATABASE_BACKUP_FILE --db-root-password DB_ROOT_PASSWORD
bench --site RESTORE_SITE_NAME migrate
bench --site RESTORE_SITE_NAME clear-cache
bench --site RESTORE_SITE_NAME list-apps

Check sample counts, links, private files, roles, deadline and privacy backlogs, and the outbox and dead-letter queue.

Note: No executable Disposition Ledger re-apply path is shipped in this version. The restore drill verifies immutable tombstones and ledger evidence. Do not invent a bench command or a manual re-apply step.

The source repository also contains an automated, disposable restore drill that runs on synthetic data in CI (bash scripts/ci/run_restore_drill.sh). It creates a synthetic dataset, takes a backup, applies reversible damage, proves that the verifier rejects the damaged site, restores database and files, and re-verifies the exact pre-backup state, including permissions and erased or anonymised subjects. Its evidence uses synthetic data only; it is not evidence about your site.

Restart after an incident

Restart order after a failure: configuration and roles, master data, contract and policy data, commissions and accounting, privacy and disposition ledger, outbox and receipts, audit and evidence bundles.

1. Check the backup and restore target from your BIA.
2. Restore database and files from the verified backup.
3. Check the restore-drill and ledger evidence. Do not alter tombstones through an invented manual step.
4. Settle the outbox and the dead-letter queue, then re-enable connectivity.
5. Check readiness and sample counters before users get access again.

For a failed installation see Installation; for a missing app or missing background jobs see Troubleshooting and FAQ.

Upgrade

The project documents no dedicated upgrade command. Combine the standard Frappe sequence with the checks above:

1. Take a verified backup (see above).
2. Update the app code in every runtime image (backend, scheduler, workers), and recreate the runtimes from the same image.
3. Run bench --site SITE_NAME migrate, then bench --site SITE_NAME clear-cache and bench --site SITE_NAME list-apps.
4. Run the registry check in every runtime (see Installation).
5. Run the readiness check, then reload the Desk once.

On existing sites, the after_migrate hook adds newly shipped workspace blocks, shortcuts, and sidebar links and keeps your site-specific additions. It touches only the current site's Desk cache and never flushes a shared Redis database.

There is no automated schema rollback. If a migration has already written a new schema, a pure image rollback may be incompatible, and a restore from the backup taken in step 1 is then a deliberate decision, not a script step.

Suspend

Stop new imports and controlled actions, drain the workers, and suspend every company:

bench --site SITE_NAME execute insurance_broker_core.api.activation.suspend \
  --kwargs COMPANY_KWARGS

Use the same COMPANY_KWARGS value as in First steps and activation.

Check open reconciliations, posting plans, outbox deliveries, the dead-letter queue, deadlines, privacy cases, and legal holds first. Suspension does not delete data.

Uninstall

Create and verify a full backup first. The uninstall guard blocks while any of the following is open, each with its own error code, and raises only the first open one per attempt:

Code	Blocked while
IBC-UNINSTALL-001	A company is active
IBC-UNINSTALL-002	A policy is active or suspended
IBC-UNINSTALL-003	A legal hold is active
IBC-UNINSTALL-004	A data-subject request case is open
IBC-UNINSTALL-005	An accounting party mapping is active
IBC-UNINSTALL-006	An outbox delivery is neither delivered nor dead-lettered
IBC-UNINSTALL-007	A document ingestion queue entry is not released or rejected
IBC-UNINSTALL-008	A retention disposition case is still running
IBC-UNINSTALL-009	An executed disposition case has no exported evidence bundle

bench --site SITE_NAME uninstall-app insurance_broker_core --yes
bench --site SITE_NAME migrate
bench --site SITE_NAME clear-cache
bench --site SITE_NAME list-apps

Remove app code from the runtime images and from sites/apps.txt only in one controlled, consistent change across backend, scheduler, and all workers. Do not prune and do not flush a shared Redis. Uninstall may remove tables and data; recovery uses the verified backup, and uninstall is not a rollback.

Health surfaces

The source repository defines a machine-readable observation contract with exactly fourteen bounded health surfaces (scheduler, outbox, consumer delivery, dead-letter queue, import queue, document quarantine, provider drift, configuration drift, privacy backlog, deadlines and renewals, accounting failures, adapter readiness, AI admission, signing-key status). A timeout, a stale observation, an unavailable source, or a malformed payload is never reported as healthy. The contract and its verifier do not make a claim about any live host.