Skip to main content

Operations

Tool / Contract Summary​

This page documents how the repository is deployed, verified, restarted, and observed. It is the runtime companion to the contract and API documentation.

Business Value​

  • gives operators a deterministic run path for local, CI, and live-host verification
  • keeps runtime evidence aligned with Fabric contract truth
  • prevents host or container mutation from happening without an auditable verify path

Current Verified State​

  • reference live host: <internal-runtime-redacted>
  • host verification posture: read-first over SSH before assuming runtime failure
  • main stack and platform-plane compose files live in deploy/compose/
  • runtime evidence can be cross-checked through /api/v1/tools/host-snapshot, /api/v1/tools/runtime-status, /api/v1/tools/runtime-evidence, /api/v1/tools/runtime-contracts, and /api/v1/tools/runtime-observations

Workspace Git/Scan Guardrails (Mandatory)​

  • Gitea is Source of Truth; local Windows workspaces are disposable working copies.
  • Never run Codex sessions from an unscoped workspace root; always use a concrete repo path.
  • Limit active repo sessions to 2-3 in parallel.
  • Before each run in a repo: git fetch --prune, git checkout <branch>, git pull --ff-only.
  • No background git discovery loops (git status, git ls-files, worktree scans) without explicit scoped need.
  • Automation scripts must run repo-scoped only, never global over a workspace root.
  • CI Actions job logs are lost on rerun: rerunning a job (manual or Gitea's own automatic reassignment of a stuck task) replaces the entire log content of the prior task attempt server-side, under the same job ID, with no warning and no versioning -- GET /repos/{owner}/{repo}/actions/jobs/{id}/logs only ever returns the latest attempt. Before triggering or allowing a rerun on a job whose failure you may need to root-cause later, download and archive its log first (fabric#1893: an unmocked-live-network test failure during a Gitea outage was only recoverable because the reviewing session had downloaded the job log before the automatic rerun overwrote it).

scan_and_fix Standard​

  • scripts/scan_and_fix.sh must enforce runner timeout + single-run lock + operator credential resolution from environment or secret-managed inputs.
  • scripts/scan_open_issues_repo_only.sh must exist and query only current repo open issues via Gitea API.

Workspace Hygiene​

  • Daily cleanup: stale _worktrees/*, _tmp/*, test-results/*, large temporary artifacts.
  • Weekly cleanup: stale local branches/worktrees.
  • Never leave valuable artifacts as untracked files in workspace root.

Dirty-State Policy​

  • Dirty state is allowed while actively implementing.
  • Before new scan/automation runs: commit/stash, or use a dedicated worktree.
  • Never propagate dirty_unknown states.

Incident Playbook (git.exe storm)​

  1. Identify parent of git.exe (usually one Codex.exe).
  2. Stop only the offending process tree.
  3. Restart session on concrete repo path.
  4. Reduce parallel sessions.
  5. Verify git.exe count drops within 30-60s.

Available Now​

Runtime paths​

  • main stack: deploy/compose/jhf-fabric.stack.yml
  • low-CPU main stack: deploy/compose/jhf-fabric.stack.low-cpu.yml
  • platform plane: deploy/compose/jhf-fabric.platform-plane.yml
  • low-CPU platform plane: deploy/compose/jhf-fabric.platform-plane.low-cpu.yml
  • ephemeral suite: deploy/compose/docker-compose.test.yml
  • low-CPU ephemeral suite: deploy/compose/docker-compose.test.low-cpu.yml

Operational scripts​

  • scripts/resolve-runtime-env.sh
  • scripts/build-runtime-tool-env.sh
  • scripts/ensure-fabric-docker-resources.sh
  • scripts/redeploy-host-stack.sh
  • scripts/redeploy-platform-plane.sh
  • scripts/reconcile-platform-api-contract-runtime.sh
  • scripts/prepare-platform-plane-assets.sh
  • scripts/test-up.sh
  • scripts/test-run.sh
  • scripts/test-down.sh
  • scripts/bootstrap_wikijs_docs.py
  • scripts/safe_docker_logs.sh
  • scripts/post-deploy-guardrails.sh
  • scripts/verify-runtime-guardrails.sh
  • scripts/verify_runtime_materialization.py
  • scripts/verify_fabric_control_plane_production_readiness.py
  • scripts/materialize_stalwart_admission_host172.sh

Platform API contract reconcile​

scripts/reconcile-platform-api-contract-runtime.sh [standard|low-cpu] is the bounded Host172 reconcile path for Fabric contract changes served by jhf-fabric-platform-api. It requires exact origin/main, no tracked changes, and no unclassified untracked files; existing runtime/evidence/ artifacts are the sole allowed untracked class. It rebuilds and recreates only the api service with --no-deps, then proves the live planning-contract surface publishes the required rubric-v2 and migration references. Use scripts/redeploy-platform-plane.sh only when an intentionally broader platform-plane change requires its other services to be recreated.

Wiki.js and platform-plane assets​

  • deploy/compose/platform-plane/wiki/HELPIFYR_WIKI_HOME.md
  • deploy/compose/platform-plane/wiki/favicon.svg
  • deploy/compose/platform-plane/wiki/jadda_helpifyr_logo.svg
  • deploy/compose/platform-plane/wiki/helpifyr-wiki-theme.css
  • docs/operations/WIKIJS_PLATFORM_PLANE.md

Optional / Extended​

  • low-CPU deployment variants
  • platform-plane services such as Wiki.js, Prometheus, Grafana, and OpenTelemetry Collector
  • optional consumers such as internal docs portals and downstream runtime dashboards

Planned / Not In Current Scope​

  • any host mutation path that is not represented by a real script or guarded preview surface
  • undocumented write flows against providers or downstream tools

Public Surfaces​

Operator runtime and evidence routes:

  • GET /health
  • GET /api/v1/platform/services
  • GET /api/v1/tools/host-snapshot
  • GET /api/v1/tools/runtime-status
  • GET /api/v1/tools/runtime-evidence
  • GET /api/v1/tools/runtime-contracts
  • GET /api/v1/tools/runtime-observations
  • GET /api/v1/production-readiness/fabric/control-plane
  • GET /api/v1/production-readiness/universal-work-security-ops
  • GET /api/v1/observability/readiness
  • GET /api/v1/security/readiness
  • GET /api/v1/recovery/readiness
  • GET /api/v1/signoff/readiness

Contract Families​

Operations interact directly with:

  • runtime port contracts
  • provider instance registry
  • drift reports
  • docs and wiki governance contracts
  • shared topology and shared service baseline contracts

Producer / Consumer Zuordnung​

  • producer: Fabric publishes runtime observations and contract-shaped operational evidence
  • consumer: operators, CI, Wiki.js, and downstream repos
  • boundary rule: host observations are consumed into Fabric, but they do not override contract truth

Compatibility Window​

  • live-host posture is Linux and POSIX/bash first
  • compose and redeploy scripts are the canonical operational interface
  • direct ad-hoc mutation is not considered a compatible operator path

Lifecycle Status​

  • active deployment and verification posture
  • host and platform-plane scripts are maintained alongside the API and contract layers

Readiness / Drift / Monitoring​

Recommended health and readiness order:

  1. GET /health
  2. GET /api/v1/platform/services
  3. GET /api/v1/observability/readiness
  4. GET /api/v1/security/readiness
  5. GET /api/v1/recovery/readiness
  6. GET /api/v1/signoff/readiness
  7. subsystem-specific readiness for persistence, Dapr, events, tooling, identity, or providers as needed

Monitoring stack:

  • Prometheus for metrics collection
  • Grafana for dashboards
  • OpenTelemetry Collector for telemetry export
  • /api/v1/monitoring/metrics for tool and policy metrics

Deployment / Verify​

Validate compose​

  • bash ./scripts/resolve-runtime-env.sh /tmp/jhf-fabric-resolved.env
  • docker compose --env-file /tmp/jhf-fabric-resolved.env -f deploy/compose/jhf-fabric.stack.yml config
  • docker compose -f deploy/compose/jhf-fabric.platform-plane.yml config
  • docker compose --env-file deploy/compose/platform-plane/wiki/.env -f deploy/compose/jhf-fabric.platform-plane.yml config
  • docker compose -f deploy/compose/docker-compose.test.yml config

Deploy or redeploy​

  • bash ./scripts/redeploy-host-stack.sh
  • bash ./scripts/redeploy-platform-plane.sh
  • bash ./scripts/prepare-platform-plane-assets.sh
  • bash ./scripts/ensure-fabric-docker-resources.sh
  • bash ./scripts/verify-runtime-guardrails.sh
  • python ./scripts/verify_runtime_materialization.py --check
  • python ./scripts/verify_runtime_materialization.py --check --live-via-ssh <internal-runtime-redacted><internal-runtime-redacted>
  • python ./scripts/verify_fabric_control_plane_production_readiness.py --check
  • python ./scripts/verify_fabric_control_plane_production_readiness.py --check --live-via-ssh <internal-runtime-redacted><internal-runtime-redacted> --output runtime/evidence/issue967.fabric_control_plane_live.latest.json

Signed Stalwart admission post-merge materialization​

scripts/materialize_stalwart_admission_host172.sh is the only repo-owned Host172 materializer for the signed Fabric Stalwart admission surface. It is preview-only by default and does not open SSH or mutate the host unless --apply is explicit. This tool is not a Heddle, AdGuard, Caddy, or Stalwart materializer and never manages those services.

Before an operator uses --apply, the merged target must be the exact 40-character origin/main commit and the operator must supply a host configuration reference, not secret values:

bash scripts/materialize_stalwart_admission_host172.sh \
  --target-commit <merged-origin-main-sha> \
  --config-ref /home/administrator/.config/helpifyr/runtime-secrets/jhf-fabric-auth.env

The apply path fetches origin, rejects any target that is not fetched origin/main, snapshots git/container/compose evidence under /home/administrator/.local/state/jhf-fabric-materialization/, and archives tracked patches plus untracked work. It writes and verifies snapshot-manifest.sha256 over the recovery artifacts before any checkout or compose action; a missing or changed artifact is a hard stop. It refuses a tracked-dirty checkout rather than resetting, cleaning, stashing, or deleting it. A clean checkout is fast-forwarded to the exact target only after the snapshot. Preflight checks only the canonical auth-overlay contract in /home/administrator/.config/helpifyr/runtime-secrets/jhf-fabric-auth.env: FABRIC_STALWART_ADMISSION_PRIVATE_KEY_FILE must be a readable absolute file reference, while FABRIC_HEDDLE_STALWART_MANAGEMENT_READ_TOKEN must be a direct non-file value. The materializer never prints either secret and fails closed on duplicate, missing, empty, or ambiguous entries. See docs/contracts/HELPIFYR_STALWART_HOST172_AUTH_OVERLAY.md.

After compose configuration succeeds, only Fabric api and daprd are recreated with --no-deps. Verification requires /health and the public JWKS endpoint to return 200, while unauthenticated admission and revocation reads must return 401. An authenticated admission readback is optional and requires --authorized-check plus explicit exact scope values; it is permitted only through the supplied token-reference file and must return 200.

Rollback and break-glass​

  • Normal rollback: stop immediately on any non-zero result. The snapshot path is printed only after success or dirty-state refusal and contains the pre-change commit, patches, untracked archive when present, Fabric container inspect readback, and snapshot-manifest.sha256. Verify the manifest before using recovery artifacts. Compose configuration is validated quietly and never written out because rendered configuration can expose environment values. Restore or reconcile only through a separately reviewed, exact-commit operator procedure; this materializer intentionally has no automatic rollback or destructive restore mode.
  • Dirty checkout break-glass: do not use git reset, git clean, or this script to force through it. Preserve the emitted snapshot, have the checkout owner classify the work, then rerun only after the owner has made the checkout clean through an approved recovery path.
  • Secret/config break-glass: do not place values on the command line or in issue comments. Repair the secret-managed env-file reference and readable signing-key file, then rerun preflight. A missing/ambiguous reference is a hard stop.
  • Runtime break-glass: if Fabric health or endpoint status checks fail after the scoped recreate, stop. Do not restart Heddle, AdGuard, Caddy, or Stalwart. Use the snapshot and bounded Fabric-only diagnostics to hand off to the Fabric runtime owner.

Local API start​

  • uvicorn helpifyr_fabric.api.app:app --reload

Ephemeral verification​

  • bash ./scripts/test-up.sh
  • bash ./scripts/test-run.sh
  • bash ./scripts/test-down.sh

Docs bootstrap​

  • python scripts/bootstrap_wikijs_docs.py --wiki-url http://<internal-runtime-redacted>:33001 --site-host https://docs.helpifyr.com/
  • add --dry-run to preview output without writing pages
  • python scripts/docs/materialize_public_docs_site.py
  • python scripts/docs/export_public_docs_bundle.py
  • npm run build --prefix docs-site
  • npm run deploy:cloudflare --prefix docs-site

Wiki.js bootstrap remains internal/operator-only. The live canonical public docs entrypoint is https://docs.helpifyr.com/; https://docs.helpifyr.com/ remains only the admitted alias or handoff surface. The tracked bundle contracts/docs/public_docs_site_bundle.json remains the canonical Fabric-generated handoff artifact and must not reintroduce jhf-web or https://docs.helpifyr.com/ as the current public publisher truth.

Scan&Fix automation​

Quickstart:

  • bash scripts/scan_and_fix.sh --dry-run
  • bash scripts/scan_and_fix.sh
  • bash scripts/scan_and_fix.sh --issue 344 --dry-run
  • bash scripts/scan_and_fix.sh --max-issues 1
  • bash scripts/scan_and_fix.sh --executor-cmd 'python -c "import sys; print(sys.stdin.read())"' --issue 344

Dry-run:

  • prints the reconciled open issue set in ranked order
  • classifies the queue into really-open-repo-owned, already-covered-by-pr, blocked_external, and non_actionable_open
  • excludes open pull requests from the executable issue queue even when the backing Gitea issues API returns them in the same feed
  • falls back to the visible repo-scoped Gitea /pulls page plus per-PR detail reads when the open-pulls list API returns an empty set but open PRs still exist
  • prints the generated execution prompt with repo, issue, branch, worktree, and live-verify host/user context
  • does not invoke the runner
  • when no --issue or --max-issues is provided, the dry-run shows the full currently open ranked queue plus the actionable executable subset

Live-run:

  • prefer --executor-cmd when you want an explicit runner path, for example a local Codex wrapper
  • if --executor-cmd is omitted, the script falls back to SCAN_AND_FIX_RUNNER_CMD, then to the first verified local Codex CLI candidate it can execute, including a verified per-user installation before any inaccessible WindowsApps alias
  • the auto-discovered local Codex fallback uses the local danger-full-access execution mode by default because nested Windows --full-auto runs can fail before repo work starts with sandbox spawn errors; use --executor-cmd when you need a different runner posture explicitly
  • example:
    • bash scripts/scan_and_fix.sh --executor-cmd 'python -c "import sys; print(sys.stdin.read())"' --issue 344
  • default behavior:
  • bash scripts/scan_and_fix.sh processes the full currently open ranked issue set, then dispatches only the really-open-repo-owned subset
  • bash scripts/scan_and_fix.sh --max-issues 1 limits the run to the highest-ranked issue
  • open pull requests are skipped so the queue stays issue-driven instead of PR-driven, and issues already covered by an open PR, explicitly marked blocked_external, or already declared planned_only / planned_program_work / future_feature are reported but not dispatched by default
  • shared-host CI executes only the scanfix_fastpath pytest subset by default; the heavier scanfix_extended harness/fallback coverage is reserved for explicit local verification or workflow_dispatch so the shared runner lane stays low-pressure

Supported parameters:

  • --issue <id>
  • --dry-run
  • --max-issues <n>
  • --since <duration>
  • --labels <csv>
  • --severity-order "critical,high,medium,low"
  • --host <host>
  • --user <user>
  • --executor-cmd <cmd>

Typical failure modes:

  • missing required API credentials in the operator environment or secret-managed inputs
  • no matching open issue after filters are applied
  • --executor-cmd omitted, SCAN_AND_FIX_RUNNER_CMD unset, and no usable local Codex CLI candidate available
  • runner command returns non-zero
  • long all-open runs started without a confirming dry-run after issue churn, causing an unintended queue shape
  • operator expects mutation on main; the wrapper only prepares issue-driven branch/PR work

Known Limits​

  • direct host/port verification remains canonical when a service has not yet published its own hostname contract
  • platform-plane components are optional and may not be present on every installation
  • runtime observations can be delayed or degraded when provider dependencies are unavailable

Exceptions / Waivers​

  • legacy *.<internal-runtime-redacted> hostnames are redirect surfaces only and must not be treated as primary Fabric-owned truth
  • some runtime probes are intentionally lightweight and policy-limited to avoid pressure on host services

Logs​

For live-host log inspection, use the bounded snapshot policy in operations/HOST_DOCKER_LOG_GUARDRAILS.md (docs/operations/HOST_DOCKER_LOG_GUARDRAILS.md).

Main stack​

  • timeout 15s docker compose -f deploy/compose/jhf-fabric.stack.yml logs --since 10m --tail 80 api
  • timeout 15s docker compose -f deploy/compose/jhf-fabric.stack.yml logs --since 10m --tail 80 daprd
  • timeout 15s docker compose -f deploy/compose/jhf-fabric.stack.yml logs --since 10m --tail 80 postgres
  • timeout 15s docker compose -f deploy/compose/jhf-fabric.stack.yml logs --since 10m --tail 80 nats
  • bash ./scripts/post-deploy-guardrails.sh jhf-fabric

Platform plane​

  • timeout 15s docker compose -f deploy/compose/jhf-fabric.platform-plane.yml logs --since 10m --tail 80 api
  • timeout 15s docker compose -f deploy/compose/jhf-fabric.platform-plane.yml logs --since 10m --tail 80 prometheus
  • timeout 15s docker compose -f deploy/compose/jhf-fabric.platform-plane.yml logs --since 10m --tail 80 grafana
  • timeout 15s docker compose --env-file deploy/compose/platform-plane/wiki/.env -f deploy/compose/jhf-fabric.platform-plane.yml logs --since 10m --tail 80 wikijs
  • timeout 15s docker compose -f deploy/compose/jhf-fabric.platform-plane.yml logs --since 10m --tail 80 otel-collector
  • bash ./scripts/post-deploy-guardrails.sh jhf-fabric

CPU-Safe Runtime Guardrails​

  • Standard verify and redeploy paths must stay bounded and low-pressure.
  • Use bash ./scripts/verify-runtime-guardrails.sh before release-oriented changes to confirm repo-owned stack truth, bounded diagnostics, and low-pressure defaults.
  • Use python ./scripts/verify_runtime_materialization.py --check --live-via-ssh <internal-runtime-redacted><internal-runtime-redacted> when runtime/config changes are involved to prove repo truth, active compose labels, container env/mounts/networks, and app readback stayed aligned.
  • Post-deploy cleanup is mandatory through bash ./scripts/post-deploy-guardrails.sh jhf-fabric; the script fails closed if stale repo-owned docker logs, docker compose ... logs, or docker exec diagnostics remain beyond the configured minimum age.

Typical Failure Modes​

  • Dapr sidecar unavailable
  • PostgreSQL bootstrap or migration incomplete
  • NATS or event publication readiness drift
  • stale repository manifests or tool profiles
  • Grafana active-dashboard provisioning drift
  • provider runtime evidence delay or DNS/network mismatch on the host

Diagnosis Order​

  1. validate compose config
  2. inspect docker compose ... ps
  3. inspect bounded Fabric API log snapshots
  4. inspect bounded dependency service log snapshots
  5. run readiness endpoints in the order above
  6. use subsystem runbooks before mutating state

Restart And Recovery​

  • restart only the affected Fabric-owned service when possible
  • prefer additive rebuild or restart over manual state edits
  • use STACK_RECOVERY_RUNBOOK (docs/operations/STACK_RECOVERY_RUNBOOK.md) before direct persistence mutation
  • use recovery and signoff readiness surfaces to confirm post-restart state

Runtime Dependencies​

  • PostgreSQL
  • NATS JetStream
  • Dapr sidecar
  • Gitea for repository contract intake
  • optional platform-plane observability services
  • optional internal docs consumer through Wiki.js
  • operational history and remaining backlog items live under docs/issues/ and docs/AUTONOMOUS_BACKLOG.md

License​