Skip to main content

Operations

Start / Run / Deploy​

Primary modes:

  • local/operator CLI via python -m n8n_expert ...
  • packaged CLI via jhf-shuttle ...
  • optional API runtime via jhf-shuttle serve
  • optional OCI image from the root Dockerfile
  • optional on-prem compose bundle via docker-compose.onprem-messaging.yml
  • optional OCI package path via scripts/oci_image.sh (explicit version and sha tags)
  • issue-driven Scan&Fix automation via scripts/scan_and_fix.sh

Canonical runtime contract source:

  • docs/RUNTIME_STACK_CONTRACT.md

Scan&Fix​

Quick start:

  • dry-run all currently matching open issues:
    • bash scripts/scan_and_fix.sh --dry-run
  • live-run all currently matching open issues through the configured executor:
    • SCAN_AND_FIX_EXECUTOR_CMD="bash scripts/scan_and_fix_openclaw_executor.sh --agent cicd-ops --timeout 900" bash scripts/scan_and_fix.sh
  • dry-run a specific issue:
    • bash scripts/scan_and_fix.sh --dry-run --issue 170
  • dry-run against a bounded local harness:
    • bash scripts/scan_and_fix.sh --dry-run --issues-file /tmp/issues.json --repo-owner JaddaHelpifyr --repo-name jhf-shuttle --base-url http://<internal-runtime-redacted>:3000
  • live-run one specific issue through the established agent/CLI path:
    • SCAN_AND_FIX_EXECUTOR_CMD="bash scripts/scan_and_fix_openclaw_executor.sh --agent cicd-ops --timeout 900" bash scripts/scan_and_fix.sh --issue 170

Parameters:

  • --issue <id>
  • --dry-run
  • --max-issues <n> (0 = all, default 0)
  • --since <duration> (7d, 12h, 2h30m, ...)
  • --labels <csv> (all listed labels must be present)
  • --exclude-labels <csv> (any listed label removes the issue from the queue; default blocked-external,quality-blocked)
  • --severity-order "critical,high,medium,low"
  • --host <host>
  • --user <user>
  • --repo-owner <owner>
  • --repo-name <name>
  • --base-url <url>
  • --issues-file <path>
  • --executor-cmd <cmd>

Runtime contract:

  • prompt source of truth: prompts/scan_and_fix_execution_standard.md
  • generated selection/prompt artifacts: artifacts/scan_and_fix/
  • generated batch result artifacts: artifacts/scan_and_fix/scan_and_fix_results_<timestamp>.jsonl
  • generated artifacts/scan_and_fix/ output is local execution evidence only and must remain untracked
  • live execution is delegated through SCAN_AND_FIX_EXECUTOR_CMD; the script does not self-merge or bypass branch/PR policy
  • repo-owned OpenClaw helper: bash scripts/scan_and_fix_openclaw_executor.sh --agent cicd-ops --timeout 900
  • live execution is blocked on branch main; create or switch to an issue branch/worktree first
  • live execution is blocked when the current worktree is dirty
  • repo identity resolution is fail-closed unless origin or explicit override flags provide a complete owner/name/base tuple
  • matching repo-owned issues are executed serially in deterministic severity/update order unless a narrower filter or --issue is supplied
  • default queue selection excludes issues labeled blocked-external or quality-blocked
  • batch live execution attempts every selected issue even if an earlier executor handoff fails; the script exits non-zero afterwards when any issue failed

Failure modes:

  • no matching open issues after filters -> non-zero exit with explicit message
  • missing prompt file -> non-zero exit
  • unresolved repo identity without explicit overrides -> non-zero exit
  • missing SCAN_AND_FIX_EXECUTOR_CMD during live run -> non-zero exit
  • live run on main -> non-zero exit
  • live run with dirty worktree -> non-zero exit
  • one or more executor failures in a batch -> non-zero exit after the queue finishes, with per-issue result evidence
  • executor command failure -> propagated non-zero exit for caller/CI visibility

Operator rule:

  • start with --dry-run
  • keep diagnostics bounded through issue filters (--issue, --max-issues, --since, --labels)
  • treat the executor as the implementation owner; the script only assembles deterministic issue context and execution standard truth

Healthchecks​

  • API health: GET /api/v1/health
  • API status/readiness-adjacent surface: GET /api/v1/status
  • mailbox adapter health: GET /healthz
  • canonical self-stack host-facing mailbox adapter health: GET http://<host>:58815/healthz (JHF_SHUTTLE_MAILBOX_HEALTH_HOST_PORT)
  • decommissioned compatibility profile only: GET http://<host>:58805/healthz (MAILBOX_ADAPTER_HEALTH_HOST_PORT); it must never be selected as canonical Shuttle presence.
  • mailbox publish payload guardrail: MAILBOX_ADAPTER_MAX_PAYLOAD_BYTES (default 131072, returns 413 payload too large when exceeded)
  • there is no separate committed /readyz endpoint today

On-prem messaging compose health policy:

  • jhf-shuttle-nats and jhf-shuttle-mailbox-adapter are the only healthchecked services in the default stack
  • default interval: 30s, timeout: 3s, retries: 5, start period: 30s
  • low-CPU override (docker-compose.onprem-messaging.lowcpu.yml) raises intervals to 90s
  • jhf-shuttle-restart-recovery and openclaw-lane-wait-observer intentionally run without healthchecks to avoid unnecessary probe churn
  • low-CPU evidence runbook: docs/LOW_CPU_24H_EVIDENCE_PLAN.md
  • low-CPU run status snapshot: docs/LOW_CPU_24H_EVIDENCE_STATUS_2026-04-02.md
  • overlap guardrail preflight: py scripts\check_shuttle_runtime_overlap.py must return 0 before cutover/deploy (mailbox/NATS/restart-recovery alias groups)

Lane-wait observer guardrails:

  • only one active lane-wait observer may poll logs for the same gateway target at a time
  • OPENCLAW_LANE_WAIT_PRIMARY_OBSERVER_NAME defines deterministic primary ownership when multiple stacks are present
  • secondary observers enter standby mode and must not continuously poll Docker logs
  • default polling is hardened for low host overhead:
    • OPENCLAW_FLOW_CONTROL_POLL_SECONDS=45
    • hard floor OPENCLAW_FLOW_CONTROL_MIN_POLL_SECONDS=30
    • idle poll OPENCLAW_FLOW_CONTROL_IDLE_POLL_SECONDS=120
    • bounded log window OPENCLAW_FLOW_CONTROL_LOG_TAIL_LINES=300
    • bounded backoff and jitter (OPENCLAW_FLOW_CONTROL_MAX_BACKOFF_SECONDS, OPENCLAW_FLOW_CONTROL_POLL_JITTER_SECONDS)

Restart-recovery guardrails:

  • restart poll defaults are low-pressure and bounded:
    • OPENCLAW_RESTART_RECOVERY_MIN_POLL_SECONDS=30
    • OPENCLAW_RESTART_RECOVERY_POLL_SECONDS=45
    • OPENCLAW_RESTART_RECOVERY_MAX_BACKOFF_SECONDS=300
    • OPENCLAW_RESTART_RECOVERY_POLL_JITTER_SECONDS=3
  • errors must not trigger tight-loop retries; restart-recovery uses bounded backoff + jitter

Safe Docker log policy for observer diagnostics:

  • use bounded log reads only (timeout + --since + --tail)
  • do not run unbounded or follow-mode reads on live hosts (docker logs -f, unlimited --tail)
  • observer use-case must always stay within bounded read limits and bounded poll intervals

Runtime port-policy verify/readiness path:

  1. python3 scripts/check_host_port_contract.py
  2. python3 scripts/verify_runtime_port_contract.py --json
  3. python3 scripts/verify_runtime_port_contract.py --json --ssh-target <internal-runtime-redacted><internal-runtime-redacted>
  4. python3 scripts/verify_cpu_safe_runtime_guardrails.py --json
  5. bash scripts/post_deploy_runtime_cleanup.sh

Stale-task escalation verify path:

  1. python3 scripts/verify_stale_task_escalation.py --ssh-target <internal-runtime-redacted><internal-runtime-redacted>
  2. confirm classification=healthy or review cancel_candidate_count / owner_review_required_count
  3. if stale tasks remain, use the emitted openclaw tasks cancel <taskId> only for explicit cancel candidates; keep all other lanes read-only and escalate with evidence

Cutover-only overlap guardrail:

  • py scripts\check_shuttle_runtime_overlap.py must return 0 before enforcing a single mailbox/NATS/restart-recovery path

Rules:

  • static-required: undeclared live port drift is a failure
  • dynamic-allowed-with-discovery: discovery source + consumer-safe publish path are mandatory
  • internal-only: host port publishes are failures
  • shared-host-exception: allowed only with explicit contract entry and discovery path

Logs And Artifacts​

  • logs/events.jsonl
  • logs/contexts/*.json
  • logs/upgrade-impact/*.json
  • logs/upgrade-automation/latest.json
  • logs/catalog-refresh/*.json
  • logs/release-hardening/latest.json
  • logs/end-to-end-regression/latest.json
  • dist/package-metadata.json (generated packaging contract output)

Monitoring-Relevant States​

  • n8n reachability
  • instance version and version gap
  • catalog freshness and baseline refresh truth
  • latest upgrade summary, alerts, and backlog
  • mailbox adapter health
  • NATS/JetStream configuration truth
  • webhook/callback contract visibility

Dashboard Fields​

Grafana should prioritize:

  • latest upstream version
  • versions behind
  • catalog freshness status
  • baseline coverage ratio
  • top upgrade alerts
  • mailbox adapter health and pressure level

Gitea dashboards should prioritize:

  • current version
  • latest successful verification
  • lifecycle stage
  • open residual risks
  • registered capabilities
  • critical dependencies

Operational Gaps​

  • no unified /metrics endpoint
  • no single committed readiness-only endpoint
  • long-running operational evidence still depends on scripts and artifacts rather than a persistent control plane

Plan Studio C8 Owner Readback Service​

The canonical Shuttle self-stack includes the bounded, bearer-authenticated read-only owner surface for the Fabric-admitted Plan Studio C8 evidence:

The service starts the API with --owner-readback-only; it does not expose the general Shuttle workflow, validation, upgrade, or catalog routes.

  • GET http://<internal-runtime-redacted>:${JHF_SHUTTLE_PLAN_STUDIO_API_HOST_PORT:-18087}/healthz
  • GET http://<internal-runtime-redacted>:${JHF_SHUTTLE_PLAN_STUDIO_API_HOST_PORT:-18087}/readyz
  • GET http://<internal-runtime-redacted>:${JHF_SHUTTLE_PLAN_STUDIO_API_HOST_PORT:-18087}/api/v1/evidence/plan-studio-review-transport

All routes require Authorization: Bearer <JHF_SHUTTLE_API_TOKEN>. The service is readback-only and must never be used as a provider, planning, approval, materialization, or execution authority. Host172 deployment and authenticated live evidence remain owner-controlled through jhf-openclaw-env#2319; missing owner token or service is fail-closed.

AGPLv3. Learn more at helpifyr.com.

Workspace Git/Scan Guardrails (Mandatory)​

  • Gitea is Source of Truth; local Windows workspaces are disposable working copies.
  • Never run Codex sessions on <local-workspace-redacted> root; always use a concrete repo path.
  • Limit active repo sessions to 2-3 in parallel.
  • Before each run in a repo: git fetch --prune, git checkout <branch>, git pull --ff-only.
  • No background git discovery loops (git status, git ls-files, worktree scans) without explicit scoped need.
  • Automation scripts must run repo-scoped only, never global over <local-workspace-redacted>.

scan_and_fix Standard​

  • scripts/scan_and_fix.sh must enforce runner timeout + single-run lock + .env fallback to <local-env-redacted> and <local-env-redacted>.
  • scripts/scan_open_issues_repo_only.sh must exist and query only current repo open issues via Gitea API.

Workspace Hygiene​

  • Daily cleanup: stale _worktrees/*, _tmp/*, test-results/*, large temporary artifacts.
  • Weekly cleanup: stale local branches/worktrees.
  • Never leave valuable artifacts as untracked files in workspace root.

Dirty-State Policy​

  • Dirty state is allowed while actively implementing.
  • Before new scan/automation runs: commit/stash, or use a dedicated worktree.
  • Never propagate dirty_unknown states.

Incident Playbook (git.exe storm)​

  1. Identify parent of git.exe (usually one Codex.exe).
  2. Stop only the offending process tree.
  3. Restart session on concrete repo path.
  4. Reduce parallel sessions.
  5. Verify git.exe count drops within 30-60s.