Operations

Runtime Model

Primary production-like runtime today:

• Docker Compose via docker-compose.v2.yml
• backend service
• frontend service
• PostgreSQL

Primary known host:

• <internal-runtime-redacted>

Compose Services

• openclaw-pm-v2-postgres
• openclaw-pm-v2-backend
• openclaw-pm-v2-frontend

Active live container names on the primary host:

• jhf-pattern-openclaw-pm-v2-postgres-1
• jhf-pattern-openclaw-pm-v2-backend-1
• jhf-pattern-openclaw-pm-v2-frontend-1

Resource posture (mem/cpu/pids/restart/logging limits in docker-compose.v2.yml and integrations/plane/runtime/plane-app/docker-compose.yaml) is materialized from jhf-openclaw-env deploy/fleet/fleet-resource-manifest.v1.json (ref jhf-openclaw-env#2855).

Start / Run / Deploy

Local

• backend:
  • uvicorn backend.app.main:app --reload
• frontend:
  • npm run dev

Host-style deploy

• use the repo checkout plus docker-compose.v2.yml
• rebuild the changed service or full stack as needed

Common commands:

docker compose -p jhf-pattern -f docker-compose.v2.yml up -d --build
docker compose -p jhf-pattern -f docker-compose.v2.yml up -d --build openclaw-pm-v2-backend
docker compose -p jhf-pattern -f docker-compose.v2.yml -f docker-compose.backend-healthcheck.yml up -d --build openclaw-pm-v2-backend
docker compose -p jhf-pattern -f docker-compose.v2.yml ps

Low-CPU host mode

Use the override below on weaker external hosts where reducing healthcheck churn matters more than fast local feedback:

docker compose -p jhf-pattern -f docker-compose.v2.yml -f docker-compose.low-cpu.yml up -d --build

Verifier contract:

• docker-compose.low-cpu.yml is intentionally a partial override and does not need to appear on unaffected services such as the frontend container.
• runtime materialization verification should fail only when a running container points outside the allowed repo-owned compose family or loses the base docker-compose.v2.yml, not merely because a subset service omits the low-CPU overlay.

Optional backend Docker healthcheck

Keep the backend healthcheck disabled by default when external /readyz verification is enough. If operators explicitly need Docker-internal backend health state, enable the lightweight probe with:

docker compose -p jhf-pattern -f docker-compose.v2.yml -f docker-compose.backend-healthcheck.yml up -d --build openclaw-pm-v2-backend

Do not keep separate verify/test containers running permanently beside the runtime stack. Run verification commands explicitly, record the result, and stop temporary test processes afterwards.

Health and Readiness

Available runtime surfaces:

• /healthz
• /readyz
• /versionz
• /contractz
• /fabric-presencez
• /fabric-compatibilityz
• /fabric-manifest.json

These are the current minimum contract surfaces for health, readiness, version, and fabric discovery.

Verify contract classes:

• liveness:
  • /healthz
  • use for cheap HTTP reachability only
• readiness:
  • /readyz
  • use as the default rollout, restart, and recovery gate
• contract verification:
  • /versionz
  • /contractz
  • /fabric-presencez
  • /fabric-compatibilityz
  • /fabric-manifest.json
• local proof:
  • bash scripts/fabric-selfcheck.sh
  • bash scripts/runtime-port-discovery.sh

Default verify path:

1. curl or browser-check /healthz
2. curl or browser-check /readyz
3. cd frontend && npm run test:smoke:live
4. when the home shell changed, also run cd frontend && npm run test:smoke:new-ui:live
5. inspect /versionz and /fabric-manifest.json when contract drift is suspected
6. run bash scripts/fabric-selfcheck.sh
7. run bash scripts/runtime-port-discovery.sh and capture the JSON evidence
8. run bash scripts/verify_runtime_guardrails.sh when host-side CPU-safety, bounded diagnostics, restart policy, and cleanup discipline are part of the current rollout scope
9. run bash scripts/verify_runtime_materialization_drift.sh to verify repo-truth vs compose/override vs host compose .env vs container-env vs app-readback consistency
10. use deeper UI or project-specific flows only after these cheap checks are green

Live verification split:

• bounded default live smoke:
  • cd frontend && npm run test:smoke:live
  • covers the stable fast path across auth, preview/home, work, settings, repo settings, and readiness evidence
  • canonical target host is https://<internal-runtime-redacted>
• full live host sweep:
  • cd frontend && npm run test:smoke:live:full
  • broader operator and mutation coverage
  • not the default fast verify command because it is intentionally larger

Temporary fallback when the canonical ingress edge is degraded:

• cd frontend && PLAYWRIGHT_LIVE_BASE_URL=http://<internal-runtime-redacted>:15173 PLAYWRIGHT_LIVE_API_BASE=api/v1 npm run test:smoke:live
• use this only to separate app/runtime health from ingress cutover defects

Current Docker healthcheck policy:

Service	Docker healthcheck	Interval	Start period	Why
openclaw-pm-v2-postgres	pg_isready -U openclaw -d openclaw_pm_v2 -h <internal-runtime-redacted> -p 5432	90s default, 120s with low-CPU override	90s default, 120s low-CPU	critical dependency with a native lightweight probe and reduced steady-state exec churn
openclaw-pm-v2-backend	/usr/local/bin/backend-ready-probe.sh via docker-compose.backend-healthcheck.yml	90s opt-in	90s opt-in	lightweight native curl probe against /readyz when Docker-internal health state is operationally useful
openclaw-pm-v2-frontend	none	n/a	n/a	static nginx surface; external HTTP verification is cheaper and clearer

Steady-state CPU guidance:

• on shared or weak hosts, prefer 90s+ healthcheck intervals for repo-owned services
• use start_period to absorb startup and recovery time instead of keeping 30s steady-state probes
• keep backend healthchecks opt-in when external /readyz is enough

Host-side validation after rollout should include:

• docker compose ... ps
• docker inspect <container> --format '{{json .State.Health}}' for healthchecked services
• external curl against /readyz and the frontend root
• short docker events --filter event=exec_create sampling if healthcheck churn is under review
• bash scripts/runtime-load-snapshot.sh before and after runtime-affecting rollouts when weak-host load is under review

Repeatable host snapshot:

bash scripts/runtime-load-snapshot.sh
bash scripts/runtime-load-snapshot.sh 120

The helper records:

• host CPU busy percentage, load average, and memory summary
• per-container docker stats --no-stream
• current Docker health states for the compose project
• sampled exec_create event churn during the requested window

Checkout Hygiene

The host checkout at /home/administrator/control-plane/jhf-pattern must stay pull-safe and reviewable.

Use this readback before repo-owned live alignment:

cd /home/administrator/control-plane/jhf-pattern
git status --short
git rev-parse HEAD
git rev-parse origin/main

Expected:

• no tracked-file drift remains in git status --short
• runtime-generated Pattern followthrough artifacts stay ignored instead of polluting the checkout
• generated projects/openclaw-runtime-live/tasks/ and control-bundle markdown are not treated as repo-owned tracked truth
• when tracked drift exists, snapshot it first and return the checkout to canonical origin/main before post-merge live reconcile
• bash scripts/host_reconcile_pattern_checkout_main.sh is the bounded path for the classified PM-state files and .env.pre1496.<timestamp> backups; it archives them outside the checkout before reconciling and verifies the route-aware Plan Studio facade after rebuilding only Pattern services

Logs

Operational diagnosis should include:

• backend container logs
• frontend container logs
• deploy-time build output
• API route errors surfaced in backend logs

Concrete commands:

bash scripts/safe_docker_logs.sh jhf-pattern-openclaw-pm-v2-backend-1
bash scripts/safe_docker_logs.sh jhf-pattern-openclaw-pm-v2-frontend-1
bash scripts/safe_docker_logs.sh jhf-pattern-openclaw-pm-v2-postgres-1

Remote host snapshots from the operator workstation:

DOCKER_LOGS_HOST=<internal-runtime-redacted> DOCKER_LOGS_USER=administrator bash scripts/safe_docker_logs.sh jhf-pattern-openclaw-pm-v2-backend-1
DOCKER_LOGS_HOST=<internal-runtime-redacted> DOCKER_LOGS_USER=administrator bash scripts/safe_docker_logs.sh jhf-pattern-openclaw-pm-v2-frontend-1
DOCKER_LOGS_HOST=<internal-runtime-redacted> DOCKER_LOGS_USER=administrator bash scripts/safe_docker_logs.sh jhf-pattern-openclaw-pm-v2-postgres-1

Post-readback cleanup verification for shared-host diagnostics:

bash scripts/verify_host_docker_log_cleanup.sh
VERIFY_DOCKER_LOG_VMSTAT_SECONDS=3 bash scripts/verify_host_docker_log_cleanup.sh jhf-pattern-openclaw-pm-v2-backend-1

Rules:

• do not use unbounded docker logs or docker compose logs in the standard shared-host verify flow
• use bounded log snapshots only when /readyz, docker compose ps, and the runtime guardrail verify are not enough
• from the operator workstation, prefer DOCKER_LOGS_HOST=... bash scripts/safe_docker_logs.sh <container> instead of opening host-side docker logs ... | tail ... readers
• the remote helper path intentionally tails the resolved host log file instead of calling docker logs on host172, because direct daemon log reads can still reheat dockerd under shared-host pressure
• scripts/safe_docker_logs.sh is bounded-only; if you need a different window, use DOCKER_LOGS_SINCE, DOCKER_LOGS_TAIL, and DOCKER_LOGS_TIMEOUT_SECONDS instead of passing raw docker logs args
• after taking remote log snapshots on Host172, run bash scripts/verify_host_docker_log_cleanup.sh to prove no naked docker logs reader remains attached to the backend container
• after host verification, no lingering docker logs or docker events process should remain

Monitoring

Useful operator monitoring today:

• readiness checks
• provider connection status
• Plane sync status
• audit/recovery views in the product
• /metrics for sync drift and contract counters

Current gap:

• no repo-local Grafana integration

Common Failure Modes

• provider connected but no data visible
  • usually configuration, auth, or upstream-provider drift
• Plane state differs from Mission Control
  • inspect sync mode, webhook status, and mapping state
• repo-backed knowledge pages missing
  • inspect repo connection and planning/pages/sync-from-repo
• Fabric discovery endpoint mismatch
  • inspect fabric-manifest.json, /contractz, and scripts/fabric-selfcheck.sh

Restart / Recovery Hints

• prefer service-specific rebuilds rather than ad-hoc host edits
• verify runtime state after redeploy with /readyz
• for sync-related regressions, verify both endpoint behavior and operator UI state

Concrete restart commands:

docker compose -p jhf-pattern -f docker-compose.v2.yml restart openclaw-pm-v2-backend
docker compose -p jhf-pattern -f docker-compose.v2.yml restart openclaw-pm-v2-frontend

Runtime Dependencies

• PostgreSQL
• provider APIs where configured
• Plane API where configured
• n8n and runtime webhooks where configured

Known Operational Gaps

• no standardized event bus contract
• no repo-local dashboard export to Grafana or Gitea dashboard surfaces

Minimal Alert Conditions

• high
  • /readyz not returning success
  • backend container restarting repeatedly
  • PostgreSQL healthcheck failing
  • sync paths returning repeated 5xx errors
  • /metrics shows drift counters growing for Plane guarded sync or repo-page reconcile failures
• medium
  • Plane sync drift persists after a reconcile attempt
  • provider connection tests fail
  • repo-backed page sync fails
• low
  • documentation/manifests drift while runtime stays healthy

Rollout And Rollback Baseline

Before a runtime-config rollout:

• capture the current deployed commit or image identity
• record a short host snapshot with bash scripts/runtime-load-snapshot.sh
• validate the compose file with docker compose ... config

For this repo, the normal rollback path is:

1. redeploy the last known-good repo commit into the stack checkout
2. run docker compose ... up -d --build
3. verify /readyz, frontend HTTP, health state, and bash scripts/verify_runtime_guardrails.sh again

Canonical operator runbooks:

• backup and restore:
  • docs/RECOVERY_RUNBOOK.md (docs/RECOVERY_RUNBOOK.md)
• deployment promotion:
  • docs/DEPLOYMENT_PROMOTION_RUNBOOK.md (docs/DEPLOYMENT_PROMOTION_RUNBOOK.md)
• routine upgrades and rollback boundary:
  • docs/UPGRADE_RUNBOOK.md (docs/UPGRADE_RUNBOOK.md)

Prometheus-Ready Metrics Surface

Mission Control now exposes a read-only metrics surface at:

curl -sf http://<internal-runtime-redacted>:18081/metrics

Current operator-critical metric groups:

• openclaw_mc_projects_total
• openclaw_mc_integration_connections_total
• openclaw_mc_integration_connection_last_test_total
• openclaw_mc_sync_surface_total
• openclaw_mc_external_mappings_total
• openclaw_mc_guarded_sync_state_total
• openclaw_mc_plane_read_reconcile_total
• openclaw_mc_repo_page_reconcile_total

Interpretation baseline:

• openclaw_mc_sync_surface_total{sync_state="drift"} above 0
  • planning or page state needs operator review
• openclaw_mc_guarded_sync_state_total{status="conflict"} above 0
  • Plane guarded sync has unresolved conflicts
• openclaw_mc_plane_read_reconcile_total{status="error"} above 0
  • Plane auto-read reconcile failed for at least one configured bridge
• openclaw_mc_repo_page_reconcile_total{status="error"} above 0
  • repo-backed page reconcile failed for at least one configured bridge
• openclaw_mc_integration_connection_last_test_total{status="reachable"} missing for a configured provider
  • the connection has not been re-verified successfully yet

Adopt-First Rollout Boundary

For a future standalone -> integrated adoption, treat resources as follows:

• adopt:
  • repo checkout
  • product docs and manifest contracts
• reconfigure:
  • compose/runtime env wiring
  • OAuth and workspace/provider credentials
• migrate:
  • database-backed Mission Control state
• rebuild:
  • host-local webhook targets and environment-specific runtime endpoints

Do not assume that a healthy standalone host can be attached to a shared-service environment without explicit operator review of credentials, webhook targets, and database migration scope.

---

License: AGPLv3

Helpifyr: https://helpifyr.com

Workspace Git/Scan Guardrails (Mandatory)

• Gitea is Source of Truth; local Windows workspaces are disposable working copies.
• Never run Codex sessions on <local-workspace-redacted> root; always use a concrete repo path.
• Limit active repo sessions to 2-3 in parallel.
• Before each run in a repo: git fetch --prune, git checkout <branch>, git pull --ff-only.
• No background git discovery loops (git status, git ls-files, worktree scans) without explicit scoped need.
• Automation scripts must run repo-scoped only, never global over <local-workspace-redacted>.

scan_and_fix Standard

• scripts/scan_and_fix.sh must enforce runner timeout + single-run lock + .env fallback to <local-env-redacted> and <local-env-redacted>.
• scripts/scan_open_issues_repo_only.sh must exist and query only current repo open issues via Gitea API.

Scan&Fix Runbook

Canonical repo-only automation entrypoints:

• bash scripts/scan_open_issues_repo_only.sh
• bash scripts/scan_and_fix.sh

Shared local credential truth for those commands lives outside the repo in:

• <local-env-redacted>

This shared env file may contain:

• GITEA_TOKEN
• OPENCLAW_HOST
• OPENCLAW_HOST_USER
• OpenClaw host SSH/sudo credentials used for bounded live verification

Do not copy those values into repo files, issue bodies, prompts, or logs.

Quick Start

List the current repo-only queue from Gitea:

bash scripts/scan_open_issues_repo_only.sh

Preview the full prompt queue without dispatch:

bash scripts/scan_and_fix.sh --dry-run

Dispatch the bounded repo queue through a local executor command:

bash scripts/scan_and_fix.sh --executor-cmd "python local_executor.py"

Dry Run

Dry-run behavior:

• resolves repo identity from the current checkout
• loads only JaddaHelpifyr/jhf-pattern open issues from Gitea
• orders them by severity then updated_at
• prints the selected queue
• prints the final per-issue execution prompt with branch/worktree context

Use this before every live run when the workspace branch or dirty-state just changed.

Live Run

Live dispatch rules:

• run from this repo root or a repo-owned worktree only
• supply --executor-cmd unless SCAN_AND_FIX_EXECUTOR is already exported
• default queue scope is all currently open repo issues; use --max-issues only to cap an intentionally smaller run
• the wrapper holds a single-run lock and fails closed if another Scan&Fix run is already active

Failure Modes

• Missing GITEA_TOKEN:
  • verify that <local-env-redacted> is present and current
• Cannot derive owner/repo from origin:
  • fix the repo remote before retrying
• No executor configured:
  • pass --executor-cmd "<bounded command>"
• Refusing to run Scan&Fix from workspace root:
  • restart from a concrete repo path, never <local-workspace-redacted>
• executor non-zero exit:
  • treat the failing issue as the current blocker, debug boundedly, then rerun

Workspace Hygiene

• Daily cleanup: stale _worktrees/*, _tmp/*, test-results/*, large temporary artifacts.
• Weekly cleanup: stale local branches/worktrees.
• Never leave valuable artifacts as untracked files in workspace root.

Dirty-State Policy

• Dirty state is allowed while actively implementing.
• Before new scan/automation runs: commit/stash, or use a dedicated worktree.
• Never propagate dirty_unknown states.

Incident Playbook (git.exe storm)

1. Identify parent of git.exe (usually one Codex.exe).
2. Stop only the offending process tree.
3. Restart session on concrete repo path.
4. Reduce parallel sessions.
5. Verify git.exe count drops within 30-60s.