Operations

> Release admission classification: internal_only_excluded

Start / Run / Deploy

Canonical runtime:

• infra/compose/docker-compose.yml
• .env from .env.example
• repo-owned bootstrap and alignment scripts under infra/scripts

Primary deploy/alignment helper:

• infra/scripts/align_host_runtime.py
• the helper aligns the full app runtime set (backend, queue-worker, scheduler, websocket, mcp-gateway, frontend) by default and derives the local frontend probe port from the running frontend container instead of assuming a fixed published port

Health

Use:

bash infra/scripts/healthcheck.sh

This checks:

• Compose service state
• ERPNext doctor output
• Helpifyr Spindle integration health endpoint

For four-layer runtime materialization drift checks, use:

python infra/scripts/verify_runtime_materialization.py --output artifacts/evidence/runtime-materialization.json

For shared-host live authority readback, use the host-side probes instead of local workstation TLS trust:

python infra/scripts/verify_runtime_materialization.py --check-live --ssh-target <internal-runtime-redacted><live-host> --output artifacts/evidence/runtime-materialization.live.json
python infra/scripts/align_host_runtime.py --check-only --ssh-target <internal-runtime-redacted><live-host> --output artifacts/evidence/runtime-authority.check-only.json
python infra/scripts/reconcile_agent_discoverability_runtime_main.py --check-only --output artifacts/evidence/agent-discoverability-runtime-main.check.json

When only the backend uses an owner-managed protected Compose environment file, pass its explicit absolute host path with --backend-env-file. The helper uses that path only for backend operations and backend authority readback; every other service keeps the normal .env and is started with a dependency guard so it cannot rematerialize backend. Do not copy secret values into the repository checkout.

Before site migration, the alignment helper waits up to 30 seconds for every app recorded in the site's installed-app registry to expose its Python hooks module. This prevents a newly started customer-pack container from being migrated before its mounted app is importable. A timeout fails the alignment without modifying the installed-app registry; investigate the app materialization path rather than retrying migration manually.

The same alignment path also verifies that jhf_spindle_core is installed on the target Frappe site. If the source is present but the site registration is absent, it performs the idempotent bench install-app jhf_spindle_core before the hooks gate and migration. This is the documented site-app materialization path; do not compensate with direct database or installed_apps file edits.

For branch-preview alignment evidence against a pushed remote-tracking ref, add the explicit preview flags so the script fetches and resolves the target ref before diff evaluation:

python infra/scripts/align_host_runtime.py --target-ref origin/fix/example-preview --allow-dirty-ff-origin-main --check-only --ssh-target <internal-runtime-redacted><live-host>

This verifies:

• repo-owned runtime truth from .env.example
• active host env / compose input truth
• running container env and compose labels
• app readback via integration_status and MCP /healthz
• exactly one canonical compose authority for the running jhf-spindle project

The verifier redacts secret-like values and fails when canonical runtime truth is missing, stale, or contradicted across layers.

When the running Host172 Spindle runtime is bind-mounted from the canonical root checkout but the discoverability source drifted away from merged main, use:

python infra/scripts/reconcile_agent_discoverability_runtime_main.py --output artifacts/evidence/agent-discoverability-runtime-main.apply.json

This path is intentionally narrow. It only reconciles the running apps/jhf_spindle_core/jhf_spindle_core/api/agent_registry.py file to exact origin/main, refuses to overwrite a locally dirty target file, writes a .runtime-recovery snapshot before mutation, and force-recreates only the app services that need the new source.

When the canonical Host172 checkout must move to merged main but the checkout is dirty in unrelated files, use:

python infra/scripts/align_host_runtime.py --ssh-target <internal-runtime-redacted><live-host> --allow-dirty-ff-origin-main --force-recreate

This mode fetches origin/main, compares the incoming changed paths with the currently dirty host paths, fast-forwards only when there is no overlap, and fails closed before any runtime mutation when overlap exists.

For deterministic stack upgrade truth generation (ERPNext, Zammad, MariaDB, Redis profiles), use:

python maintenance/pull_stack_oss_inventory.py --output test-results/stack-oss-inventory.workspace.json
python maintenance/generate_stack_upgrade_plan.py --inventory test-results/stack-oss-inventory.workspace.json --output test-results/stack-upgrade-plan.workspace.json

The second command fails if deterministic refs are missing or if Zammad resolves to latest.

For repo-owned OSS inventory/version/policy drift checks across compose runtime, CI actions, and Python dependency surfaces, use:

python maintenance/verify_oss_inventory_version_truth.py --output test-results/oss-version-truth.verify.json

For live runtime materialization comparison on the shared host:

python maintenance/verify_oss_inventory_version_truth.py --check-live --ssh-target <internal-runtime-redacted><live-host> --output artifacts/evidence/oss-version-truth.live.json

This check fails when inventory components are missing, version truth drifts, latest/floating runtime refs appear without classification, or live runtime images diverge from repo-owned truth.

For authenticated MCP read-only demo action verification on the live host, use:

python infra/scripts/verify_authenticated_mcp_demo_action.py --ssh-target <internal-runtime-redacted><live-host> --host-repo-path /home/administrator/jhf-spindle --output artifacts/evidence/mcp-authenticated-demo-action-live.json

The verifier is fail-closed: it requires host-side and running mcp-gateway container MCP key materialization and rejects unauthenticated access.

For spindle owner-checkout parity across Host73 and Host172, use:

python infra/scripts/verify_spindle_host_checkout_parity.py --allow-blocked --output artifacts/evidence/host-checkout-parity.live.json

This verifier is read-only. It reports local origin/main truth, Host73 and Host172 spindle checkout SHAs, stage/live parity, and drift from local origin/main.

The generated evidence now also includes a blocker_summary section so the current primary owner blocker is explicit. This distinguishes a generic stage/live mismatch from the more actionable case where Host73 is still behind origin/main while Host172 already matches it.

For a checkout-hygiene readback on the lagging host before any mutation, use:

python infra/scripts/reconcile_host_checkout_hygiene.py --ssh-target <internal-runtime-redacted><internal-runtime-redacted> --host-repo-path /home/administrator/jhf-spindle --output artifacts/evidence/host73-checkout-hygiene.inspect.json

This inspect-only path keeps the owner lane fail-closed while proving whether the Host73 checkout is simply behind origin/main, dirty, or ahead with local commits that would make an apply path unsafe.

The inspect artifact now includes a machine-readable recommendation. For the current Host73 state the expected recommendation is safe_fast_forward_to_origin_main, which means the checkout is clean, on main, and behind origin/main without local commits.

Guardrail / Recovery Checks

For dirty-checkout follow-up work, keep the repository root as classification-only state and move implementation into a clean worktree before editing, staging, or pushing.

Use the repo-only issue scanner from the target checkout root:

bash scripts/scan_open_issues_repo_only.sh

Guardrail rules:

• derives owner/repo from the checkout's origin remote and fails closed when that cannot be resolved
• limits issue discovery to this repository only; no cross-repo sweeps from a spindle checkout
• prefers GITEA_TOKEN when present but never prints token material
• local operator-only secret hints live in a bounded workspace .env; use that path only to load the bounded key needed for the current action, never to copy secret values into repo artifacts
• use the clean worktree branch for commit/push/PR once the root has been classified as dirty or blocked

Scan&Fix Runbook

Quickstart

bash scripts/scan_and_fix.sh

This selects open issues from Gitea for JaddaHelpifyr/jhf-spindle, sorts them by severity and recency, builds the embedded execution prompt, and dispatches it through the configured executor.

Dry-Run

bash scripts/scan_and_fix.sh --dry-run

Use this when you only want to inspect queue order and the exact prompt body.

Live Run

bash scripts/scan_and_fix.sh --executor-cmd "python scripts/scan_and_fix_openclaw_executor.py --host <live-host> --user administrator"

Default behavior is all matching open issues from Gitea unless narrowed with --issue, --max-issues, --since, or --labels. The canonical main OpenClaw lane on <live-host> needs an executor timeout budget of at least 90 seconds for full review prompts; lower budgets are treated as caller-side drift, not automatic proof of a host outage.

Failure Modes

• missing GITEA_TOKEN: load it from the bounded workspace .env or export it explicitly
• missing repo identity: fix the checkout origin or pass --repo-owner, --repo-name, and --base-url
• missing executor: use --executor-cmd or ensure the OpenClaw host path is reachable
• executor timeout too low: keep the default executor budget or pass python scripts/scan_and_fix_openclaw_executor.py --timeout 90 ... explicitly before classifying the host lane as blocked
• executor returns blocked review output: inspect the referenced PR/issue mismatch in Gitea and rerun after the repo-owned fix

Readiness

Current readiness-like surfaces:

• integration_status
• MCP /healthz
• targeted smoke scripts

Known gap:

• no dedicated /ready or /readiness endpoint exists today
• the current readiness view is documentary and operational, not a standalone runtime readiness contract

Version View

Current version-like sources:

• README version marker on main
• Git revision on main
• host-alignment and smoke evidence when a target runtime is being verified

Known gap:

• no dedicated runtime /version endpoint exists today

Connector Smoke

Use:

bash infra/scripts/contract-smoke.sh

This sends the checked-in sample supplier, Stripe, and Paddle contracts to the configured Helpifyr Spindle base URL.

Live Smoke Pack

Use:

python infra/scripts/live_smoke_pack.py --insecure --api-key "$JHF_SPINDLE_MCP_SMOKE_API_KEY" --output artifacts/live-smoke.json

This runs a non-destructive host and MCP smoke pack against the closest live Helpifyr Spindle runtime. It checks host-side Helpifyr Spindle containers, the integration health endpoint, and a wide read-only MCP tool surface across approvals, bank, SEPA, dunning, period close, procurement, contracts, HR, payroll, assets, VAT, reporting, and intercompany domains.

For final finance-close verification, the preferred non-destructive operator path is:

• create or list Period Close Checklist items
• create or list Annual Close Checklist items
• render structured VAT exports for XRECHNUNG, PEPPOL, or ZUGFERD
• create SEPA collection batches in preview-safe test tenants and inspect persisted pain.008 XML payloads

Logs

Primary sources:

• Compose container logs
• Frappe/ERP runtime output
• persisted Integration Event, Dispatch Job, Approval Packet, Dead Letter, and Agent Notification evidence

Artifact Governance

• artifacts/ is governed by docs/ARTIFACT_POLICY.md
• only canonical evidence files are intended for version control
• high-churn run/slice replay snapshots are local-only and should be cleaned after debugging windows

Monitoring

Important operator states:

• gateway liveness
• integration health
• dispatch backlog and callback completion
• approval backlog and stale work
• resilience/dead-letter backlog
• repo/host revision drift

Operator Alert Route (Issue #692)

Current bounded production-readiness posture for Helpifyr Spindle is dashboard-first and fail-closed:

• operator-facing channel: Fabric-owned jhf-spindle-overview Grafana dashboard for the attached jhf-spindle runtime slice
• canonical owner label: owner:solution-spindle
• owning service: jhf-spindle-mcp-gateway
• runtime signal sources that feed or justify this route:
  • MCP GET /healthz
  • GET /api/method/jhf_spindle_core.api.health.integration_status
  • repo/host drift and materialization readbacks from the owner verifiers

Canonical readback path on Host172:

python infra/scripts/verify_operator_alert_route.py --ssh-target <internal-runtime-redacted><live-host> --output artifacts/evidence/operator-alert-route.json

The verifier reads back, without mutating runtime:

1. Fabric GET /api/v1/observability/dashboard-registry
2. Fabric GET /api/v1/mcp/grafana-readonly
3. Fabric GET /api/v1/observability/grafana-provisioning
4. Fabric GET /api/v1/observability/alert-summary
5. the active Host172 dashboard file for jhf-spindle-overview
6. live Spindle runtime-health surfaces (/healthz and integration_status)

Expected bounded truth:

• the dashboard entry is active for tool_key=jhf-spindle
• owner label stays owner:solution-spindle
• owning repo/service stay JaddaHelpifyr/jhf-spindle / jhf-spindle-mcp-gateway
• active dashboard guidance still names the owner/service and references health/drift signal bindings
• live runtime-health surfaces are readable on Host172

This slice does not claim a repo-owned push-pager or Alertmanager route. It proves the current operator-addressable observability path explicitly instead of leaving it implicit.

Runtime Load Telemetry (Issue #15)

Use the repo-owned script to measure Helpifyr Spindle-scoped runtime load instead of host-global noise:

python infra/scripts/runtime_load_telemetry.py --project-prefix jhf-spindle --sample-seconds 60 --output artifacts/runtime-load-telemetry.json

Recommended before/after workflow:

1. capture baseline before a runtime change:

python infra/scripts/runtime_load_telemetry.py --project-prefix jhf-spindle --sample-seconds 60 --output artifacts/runtime-load-before.json

2. apply change
3. capture after snapshot:

python infra/scripts/runtime_load_telemetry.py --project-prefix jhf-spindle --sample-seconds 60 --output artifacts/runtime-load-after.json

4. compare:
   • host_cpu_percent
   • docker_exec_create.project_events
   • docker_exec_create.project_per_container
   • project_health

The script is Linux-host oriented (/proc/stat + Docker CLI). If /proc/stat is unavailable, CPU is returned as null while container/event telemetry remains usable.

Runtime Performance Gate (Issue #693)

Use the repo-owned bounded verifier when the Wave-2 production-readiness slice needs explicit performance evidence instead of telemetry-only snapshots:

python infra/scripts/verify_runtime_performance_gate.py --duration-seconds 60 --concurrency 50 --worker-interval-seconds 1 --output artifacts/evidence/runtime-performance-gate.json

Recommended production-readiness posture:

1. run from Host172 or another owner-bounded runner with the same live reachability as the runtime owner
2. keep the default public and runtime-health targets unless the issue explicitly narrows scope; the default one-second worker interval models 50 parallel users without silently converting the production gate into an unbounded stress test:
   • api/method
   • api/method
   • http://<internal-runtime-redacted>:18110/healthz
3. keep the default thresholds unless the bounded issue documents a stricter posture:
   • p95_latency_ms <= 2000
   • error_rate_pct <= 1.0
   • no readiness loss before/after the bounded load window
4. store the generated JSON artifact under artifacts/evidence/ and link it from the owning issue

This lane is intentionally bounded. It proves a class-A runtime can survive a short concurrent probe without claiming broad capacity-planning truth.

Callback Transport Diagnostics

Use the repo-owned transport probe when callback delivery from n8n/jhf-wire is unclear:

python infra/scripts/check_callback_transport.py --output artifacts/callback-transport.json

For certificate-trust triage from runtime peers:

python infra/scripts/check_callback_transport.py --insecure-tls --output artifacts/callback-transport-insecure.json

Interpretation:

• reachable=true and HTTP status returned: transport is available
• error_class=tls_verify_failed: route exists but peer trust store rejects the certificate
• error_class=connection_refused: ingress route/port is not accepting connections

For same-host internal loops, prefer an explicit callback route via N8N_DISPATCH_CALLBACK_URL when 443 trust from peer containers is not guaranteed.

For MCP smoke/release checks, provision an active key first:

• JHF_SPINDLE_MCP_SMOKE_API_KEY in the operator shell
• matching active dedicated smoke MCP API Key record (hash + read-only smoke permissions) in ERP
• keep JHF_SPINDLE_MCP_API_KEY for runtime/app traffic only (least-privileged path)

Weak-Host Profiles

Mode	CPU/RAM posture	Healthcheck posture	Verify posture
Standalone (default)	baseline host for ERP+workers	core DB 30s, MCP gateway 120s	fast smoke by default; full suites only on explicit runs
Standalone (low-CPU override)	constrained host with reduced headroom	DB 60s, MCP gateway 180s via infra/compose/docker-compose.low-cpu.yml	keep heavy checks/manual stacks off unless needed
Integrated (planned read-first)	baseline plus external consumer polling	same as standalone; no aggressive poll loops	read-first compatibility checks before any control-side expansion

Rules:

• prefer low-frequency lightweight probes instead of heavy script healthchecks
• MCP gateway interval is configurable via MCP_GATEWAY_HEALTHCHECK_INTERVAL (default 120s)
• start test/verify companion stacks only for explicit verification windows, then stop them
• if host pressure rises, switch to low-CPU compose override before broadening workload

Backup

Use:

bash infra/scripts/backup.sh

Artifacts:

• MariaDB dump
• sites volume archive
• bench-generated site backup files

Cross-Host Restore Masking Discovery

The generic backup helper is a raw production-data path and must not be transferred to Host73. Before a sanitized restore bundle can be designed, collect only the schema-level candidate inventory:

python maintenance/discover_sanitized_restore_export_surface.py --check-live --output <masking-candidates.json>

The command reads table names, column names, and data types from information_schema; it never reads data values, secrets, files, or runtime configuration. Every candidate remains human_review_required until a table/field masking map is approved for JaddaHelpifyr/jhf-spindle#946.

W9 Disposable Isolated Restore Masking Admission

maintenance/spindle-w9-restore-masking-map-v1.json is the current value-free contract for JaddaHelpifyr/jhf-spindle#1286. It supersedes neither the historical #946 map nor any historical bundle. It binds the read-only candidate count and inventory hash to an exact source identity hash, assigns every classified candidate through a closed class/action mapping, and fails closed for inventory, type, class, or source-identity drift.

The verifier emits only count, hashes, action totals, and disposable-target posture. It must not receive or emit database values, table/column lists, raw runtime identity, or secrets:

python maintenance/verify_spindle_w9_restore_masking_map.py --discovery <read-only-discovery.json> --runtime-identity <redacted-identity.json> --output <redacted-attestation.json>

The contract admits only the future JaddaHelpifyr/jhf-deployment#1017 / #965 disposable isolated target. It never authorizes a production restore or a Host172 business-data mutation.

Sanitized Cross-Host Restore Export

The normal backup path is not eligible for cross-host certification because it contains production data. For JaddaHelpifyr/jhf-deployment#848, package only an already-sanitized export using:

python maintenance/validate_sanitized_spindle_v16_restore_export.py \
  --bundle <sanitized-export-dir> --output <manifest.json>

The bundle must contain hashed database/files artifacts, runtime identity, and an attestation that masks personal data, removes credentials, redirects or disables external callouts, and retains finance/CRM/payment fixtures. --apply additionally requires the exact JHF_SPINDLE_CROSS_HOST_RESTORE_CHANGE_REF=JaddaHelpifyr/jhf-deployment#848 guard. The validator never creates a dump or transfers data; capture and transfer remain deployment-owned mutations.

Before creating the bundle, collect only the source identity from Host172:

python maintenance/validate_sanitized_spindle_v16_restore_export.py --check-live --output <preflight.json>

Isolated Frappe v16 Bootstrap

The owner-supported isolated restore bootstrap contract is maintenance/frappe-v16-isolated-bootstrap-contract.json. Site creation uses the compose MariaDB host, disables the local socket path, and has an inner BOOTSTRAP_NEW_SITE_TIMEOUT_SECONDS budget (default: 1200 seconds). A caller must reserve more than the inner budget so a timeout is returned as terminal exit code 124; credentials stay runtime-only and must not be written to diagnostic output.

Restore

Use:

bash infra/scripts/restore.sh /path/to/backup-dir

Run restore only against the isolated Helpifyr Spindle stack.

Restart / Recovery

• use normal Compose restart/recreate for isolated services
• if repo/host drift is suspected, prefer the repo-owned alignment path over ad hoc manual edits
• if gateway/front-end routing breaks after backend recreation, ensure frontend/gateway are recreated after backend health is restored

Operational Rules

• do not connect Helpifyr Spindle to the existing OpenClaw or n8n internal Docker networks unless there is a deliberate future design change
• keep shared secrets outside git
• treat Integration Event as append-only evidence
• do not let external systems write directly to ERPNext accounting tables
• perform new connector rollouts against staging first when available

Runtime Dependency Notes

• OpenClaw, n8n, and jhf-wire are integration counterparts, not local runtime replacements
• MariaDB, Redis, and ERPNext/Frappe availability are hard runtime dependencies
• no repo-owned metrics endpoint exists today; monitoring is evidence- and smoke-script-driven

---

License notice: AGPLv3 (GNU Affero General Public License v3.0) Website: https://helpifyr.com