Skip to main content

Operations

Start / Run / Deploy​

  • production stack contract: infra/docker/docker-compose.stack.yml
  • test stack contract: infra/docker/docker-compose.test.yml
  • workflow reconcile path: bash scripts/import_workflows.sh

The workflow reconcile path is fail-closed on auth drift: it uses N8N_API_KEY when valid and otherwise falls back to an owner login session via N8N_OWNER_EMAIL / N8N_OWNER_PASSWORD so that workflow Git truth can still be re-imported when the API key is stale.

Healthchecks​

  • n8n: ${N8N_BASE_URL}/healthz
  • ERiC: ${ERIC_HEALTHCHECK_URL}/health and require eric_available=true when the field is present

Readiness​

  • canonical readiness signal:
bash scripts/check_runtime_readiness.sh
bash scripts/check_runtime_readiness.sh --require-productive-provider

Default semantics:

  • bash scripts/check_runtime_readiness.sh validates the repo-owned sandbox baseline and may still be ready while productive provider onboarding is intentionally not active yet.
  • bash scripts/check_runtime_readiness.sh --require-productive-provider is the stricter operator path for productive provider sign-off.

For productive Storecove invoice dispatch, ready also requires:

  • STORECOVE_API_KEY

  • STORECOVE_LEGAL_ENTITY_ID

  • a Storecove tenant-side sender that is approved and carries the required sender identifiers for the active country lane

  • combined operator signal:

bash scripts/check_stack_state.sh
  • stack ownership-only signal:
bash scripts/check_stack_ownership.sh
  • stack/runtime contract signal:
bash scripts/check_stack_runtime_contract.sh
bash scripts/check_stack_runtime_contract.sh --live
python scripts/check_runtime_materialization_drift.py
python scripts/check_runtime_materialization_drift.py --live
bash scripts/materialize_eric_runtime.sh --cert-source <path-to-elster-pfx>
python scripts/check_elster_artifact_readiness.py --portal-only --portal-env-file <workspace-env-file>
python scripts/check_elster_artifact_readiness.py --live
python scripts/check_elster_artifact_readiness.py --expected-runtime-mode productive --vendor-root <local-elster-drop> --live
python scripts/export_live_port_observation.py --output artifacts/live-port-observation.json

There is still no dedicated HTTP readiness endpoint; this remains a documented gap.

Logs​

  • n8n execution logs (bounded only via bash scripts/safe_docker_logs.sh jhf-wire-n8n)
  • ERiC service logs (bounded only via bash scripts/safe_docker_logs.sh jhf-wire-eric)
  • provider responses surfaced through workflow execution details
  • policy reference: docs/HOST_DOCKER_LOG_GUARDRAILS.md

Monitoring​

Today:

  • CI status in Gitea
  • operator scripts for readiness and smoke
  • probe script for finAPI
  • repeatable runtime load snapshot script:
    • bash scripts/capture_runtime_load_snapshot.sh <output.json>

Still missing:

  • standardized metrics endpoint
  • dashboard-fed publication of the Fabric metadata export

Alerts And Dashboard Signals​

Operator-relevant alerts:

  • callback failures to Helpifyr Spindle, especially callback target reachability and TLS trust mismatches
  • readiness state drops from ready to pending
  • provider auth failures
  • workflow import drift between Git and n8n

Useful dashboard signals:

  • latest verified Git commit
  • current CI state on Gitea
  • readiness summary by dependency key
  • health surface reachability for n8n and ERiC
  • explicit ERiC binary availability signal from /health
  • explicit ERiC runtime split from /health: eric_runtime_available vs. eric_submit_available
  • last successful finAPI live probe time
  • count of callback failures by workflow

Non-HTTP Status Contracts​

  • Readiness is currently exposed only through:
    • bash scripts/check_runtime_readiness.sh
    • bash scripts/check_stack_state.sh
    • bash scripts/check_stack_runtime_contract.sh
    • python scripts/check_runtime_materialization_drift.py
  • Combined machine-readable status is currently exposed only through:
    • python scripts/export-fabric-metadata.py --pretty
  • Controlled live host-port publication for consumers is exposed through:
    • python scripts/export_live_port_observation.py --output artifacts/live-port-observation.json
  • Support-case delivery evidence and corner-case recovery posture are exposed through:
    • python scripts/validate_support_case_delivery.py --pretty
    • bash scripts/check_support_case_delivery.sh
  • Metrics have no equivalent contract yet; this is still a real gap.

Known Error Shapes​

  • provider-side domain errors from finAPI VoP / IBAN Name Check
  • Storecove sender lane incomplete (STORECOVE_LEGAL_ENTITY_ID missing, sender not approved yet, or sender identifiers missing)
  • callback endpoint mismatch or TLS trust failure (bash scripts/check_callback_target.sh)
  • pending credentials causing readiness pending
  • ELSTER sandbox runtime is an acceptable repo-owned baseline; productive ELSTER sign-off is a stricter follow-up path
  • productive ELSTER still remains blocked while ELSTER_TEST_MODE=true, a known test certificate path is mounted, or operator-owned cert material is missing; use python scripts/check_elster_artifact_readiness.py --expected-runtime-mode productive --live
  • shared workspace portal credentials can be preflight-checked without leaking values via python scripts/check_elster_artifact_readiness.py --portal-only --portal-env-file <workspace-env-file>
  • ERiC health reachable but eric_available=false
  • ERiC runtime libraries present but eric_submit_available=false
  • ERiC runtime inventory helper confirms missing runtime assets or wrong adapter paths: bash scripts/check_eric_runtime_inventory.sh
  • host restore/drift removed jhf-wire_eric_runtime; rematerialize with bash scripts/materialize_eric_runtime.sh --cert-source <path-to-elster-pfx> and rerun runtime checks
  • ELSTER vendor drop and cert-path materialization verifier: python scripts/check_elster_artifact_readiness.py --vendor-root <local-elster-drop> --live
  • support notification retry/recovery posture is validated via scripts/validate_support_case_delivery.py
  • compliance runtime issues are tracked in jhf-selvage (moved scope)

Restart / Recovery Hints​

  • repair in Git first
  • redeploy only stack-owned services
  • re-import workflows
  • re-run readiness and smoke checks

Runtime Dependencies​

  • Docker standalone runtime on the operator-managed OpenClaw host
  • n8n
  • ERiC service
  • provider credentials and network reachability

External Blockers​

  • external and operator-owned productive follow-ups are tracked in docs/EXTERNAL_BLOCKER_MATRIX.md
  • productive ELSTER is currently a deferred follow-up rather than an active sandbox blocker; repo-owned queue state must be read from live Gitea open issues instead of a hard-coded local list

Scan&Fix​

  • real queue source: live Gitea open issues for JaddaHelpifyr/jhf-wire
  • repo-local scanner: bash scripts/scan_open_issues_repo_only.sh
  • repo-local dispatcher:
bash scripts/scan_and_fix.sh --dry-run
bash scripts/scan_and_fix.sh --executor-cmd '<your bounded executor command>'

Rules:

  • repo-scoped only; never run from the shared workspace root
  • public Gitea read is allowed; token is optional for scanning and only needed for write-back lanes
  • blocked/external issues stay visible in the queue but must not be silently treated as repo-owned fixes

License: AGPLv3 Project website: https://helpifyr.com

Workspace Git/Scan Guardrails (Mandatory)​

  • Gitea is Source of Truth; local Windows workspaces are disposable working copies.
  • Never run Codex sessions on the shared workspace root; always use a concrete repo path.
  • Limit active repo sessions to 2-3 in parallel.
  • Before each run in a repo: git fetch --prune, git checkout <branch>, git pull --ff-only.
  • No background git discovery loops (git status, git ls-files, worktree scans) without explicit scoped need.
  • Automation scripts must run repo-scoped only, never global over the shared workspace root.

scan_and_fix Standard​

  • scripts/scan_and_fix.sh must enforce runner timeout + single-run lock + explicit repo-local or operator-provided .env discovery.
  • scripts/scan_open_issues_repo_only.sh must exist and query only current repo open issues via Gitea API.

Workspace Hygiene​

  • Daily cleanup: stale _worktrees/*, _tmp/*, test-results/*, large temporary artifacts.
  • Weekly cleanup: stale local branches/worktrees.
  • Never leave valuable artifacts as untracked files in workspace root.

Dirty-State Policy​

  • Dirty state is allowed while actively implementing.
  • Before new scan/automation runs: commit/stash, or use a dedicated worktree.
  • Never propagate dirty_unknown states.

Incident Playbook (git.exe storm)​

  1. Identify parent of git.exe (usually one Codex.exe).
  2. Stop only the offending process tree.
  3. Restart session on concrete repo path.
  4. Reduce parallel sessions.
  5. Verify git.exe count drops within 30-60s.