Operations

Operating Model

This repository is operated as an install/reapply/verify bundle for another runtime. It has no continuously running process of its own.

Start / Deploy

Linux reference path

• prepare configs/installer.env
• run bash scripts/bootstrap.sh
• complete installation with bash scripts/install_jhf_memory.sh

macOS bounded path

• use the documented macOS deployment mode in docs/MACOS.md (docs/MACOS.md)
• prefer minimal divergence from the Linux flow

Resource posture

• Resource limits (mem_limit/mem_reservation/cpus/pids_limit), restart policy, and log rotation in configs/localai-compose.yml and configs/jhf-semantic-foundation.compose.yml are materialized from jhf-openclaw-env deploy/fleet/fleet-resource-manifest.v1.json (ref jhf-openclaw-env#2855, jhf-bobbin#532/WP-D). restart: "no" only takes live effect once WP-E/E2 applies it on the host; crash-recovery until then is owned by the reconcile timer from jhf-openclaw-env#2858.

Health And Readiness

Repository-local

• bash scripts/fabric-selfcheck.sh
• python3 scripts/check_ci_verify_paths.py verifies that the manifest's defaultCi list matches the executable CI command table
• python3 scripts/run_default_ci_verifiers.py runs all declared default-CI verifiers, including tests/test_contracts.py
• python3 scripts/export-fabric-metadata.py
• python3 scripts/export_fabric_status_bundle.py

Target runtime

• LocalAI readiness endpoint
• LocalAI model listing
• LocalAI embeddings endpoint
• Qdrant collection inspection
• OpenClaw memory slot and plugin presence
• read-only runtime snapshot export and validation

The repository itself has no native /health or /ready.

TS-14 evidence boundary

• python3 scripts/check_ts14_observability_contract.py checks the repo-owned metric, audit, mandatory runbook, rebuild/restore, upgrade, and CVE/OSV contract only.
• The check performs no host, Docker, database, Cypher, TLS, credential, provider/model, restore, upgrade, or healthcheck-loop action.
• Local or synthetic fixtures are contract evidence only and cannot promote live acceptance. Missing evidence is needs_verification; an unavailable owner gate is blocked_external.
• Live dashboard, audit, runbook, rebuild, restore, upgrade, and CVE/OSV evidence remains runtime-owner materialized and must arrive as a redacted admitted receipt.

Semantic graph pilot operator posture

• Host172 graph-core shadow posture:
  • the read-only verifier reads the Compose service labels localai, bobbin-semantic-api, bobbin-control-db, bobbin-graph-store, and bobbin-reconciliation-worker; it does not compare configurable container_name prefixes with service names
  • all five services must be present for the materialized S0 shadow runtime; absence of any service fails closed
  • this readback proves shadow materialization only; the semantic activation stage remains S0_not_activated and does not promote S1-S3
• Host73 retrieval-satellite posture:
  • bobbin-retrieval-model-service-73 and bobbin-retrieval-proxy-73 are the admitted satellite surfaces from jhf-openclaw-env#2005
  • Host73 does not own graph-core truth, deletion execution, or pointer-switch authority
• repo-owned semantic ops verifier:
  • python3 scripts/check_semantic_ops_tooling_contract.py
  • python3 scripts/verify_isolation_domain_walls.py --check verifies the K1 isolation-domain walls (jhf-bobbin#517) deterministically with no live dependencies
  • live read-only posture:
    • python3 scripts/check_semantic_ops_tooling_contract.py --check-live --host172 <internal-runtime-redacted> --host73 <internal-runtime-redacted> --user administrator

Healthcheck Standards

• no interval below 20s
• standard runtime checks: 120s
• low-cpu profile checks: 180s
• timeout stays in 2-5s
• retries stay in 3-5
• start period stays in 20-60s

The same policy is exposed machine-readably via fabric-manifest.json under runtime.verificationContract.

Logs

Repository-local CI and script output are the primary local logs.

For live operation, relevant logs are on the target runtime:

• OpenClaw gateway logs
• LocalAI container logs
• Qdrant container logs

Monitoring Signals

Signal	Source	Meaning	Operator action
fabric-selfcheck ok	scripts/fabric-selfcheck.sh	Repo-local contract is internally consistent.	Continue with bounded verification or release preparation.
metadata export succeeds	scripts/export-fabric-metadata.py	Manifest and config contracts can be exported machine-readably.	Use exported metadata for Fabric/Wiki/update consumers.
status bundle export succeeds	scripts/export_fabric_status_bundle.py	Repository-only Fabric presence/status surface is valid.	Publish bundle output for Fabric read-side consumers.
runtime snapshot contract valid	scripts/export_host_runtime_snapshot.py + scripts/validate_runtime_snapshot.py	Host truth probe is available, structurally valid, and classifies blocked canonical config reads explicitly instead of hanging verification.	Inspect drift signals and execute reapply/rollback decisions if needed.
runtime materialization drift check valid	scripts/check_runtime_materialization_drift.py --check-live	Repo truth, active compose/env materialization, container truth, and app readback agree; undocumented host overrides and stale readback fail closed.	Stop rollout, classify owner, and fix repo- or runtime-owner drift before continuing.
stack/container contract valid	scripts/check_stack_contract.py	Repository stack truth is complete and naming/compose/env/health contracts are consistent.	Stop rollout and fix contract drift in repo before live mutation.
LocalAI runtime contract valid	scripts/check_live_runtime_contract.py	Active LocalAI container labels/path/project and guardrails match canonical contract, and the published LocalAI /readyz surface is reachable.	If failing, redeploy from canonical stack root and remove parallel/legacy compose launch paths.
LocalAI ready	LocalAI /readyz	Embedding runtime is available.	Proceed with runtime smoke checks.
LocalAI probe guard metrics stable	/tmp/jhf-bobbin-localai-guard.prom	Timeout bursts are below threshold and degraded mode is not active for long periods.	If timeout/degraded counters rise, stop recreate loops and investigate LocalAI runtime pressure before further mutation.
semantic ops tooling contract valid	scripts/check_semantic_ops_tooling_contract.py	Repo-owned readiness/drift, bounded metrics, backup/restore, rebuild, upgrade-shadow, CVE, and runbook hooks stay machine-readable and fail-closed.	Stop rollout and repair repo-owned ops truth before runtime-owner rollout continues.
Host172 graph-core shadow posture valid	scripts/check_semantic_ops_tooling_contract.py --check-live	Host172 exposes the required Bobbin Compose service labels for the materialized S0 shadow runtime; missing labels fail closed while S1-S3 activation remains blocked.	Stop and classify runtime-owner drift if a required service label is missing or the topology widens beyond the admitted surface.
Host73 retrieval-satellite posture valid	scripts/check_semantic_ops_tooling_contract.py --check-live	Host73 exposes the admitted retrieval satellite without becoming graph-core truth owner.	Stop and classify owner drift if the satellite disappears or widens unexpectedly.
LocalAI embeddings respond	LocalAI /v1/embeddings	Embedding path is functionally usable.	Verify model alias and semantic memory path.
Qdrant collection present	Qdrant collection inspection	Memory store is available with expected collection.	Continue recall/store checks or rerun bootstrap if missing.
OpenClaw memory slot correct	OpenClaw config/runtime	Semantic memory is routed to the intended slot.	Reapply or roll back if the slot drifted.

Bounded Metrics

Bobbin does not publish a resident metrics endpoint for the semantic graph lane in this slice.

Repo-owned bounded metrics stay file-backed only:

• LocalAI guard metrics: /tmp/jhf-bobbin-localai-guard.prom
• reconciliation-worker status snapshot: /runtime-state/reconciliation-worker-status.json

The reconciliation worker writes its status atomically through its own reconciliation-worker-status.json.tmp.* files. A bounded startup sweep removes only that exact worker-owned pattern, and signal/exit cleanup removes an in-flight temporary file before the worker terminates. Files belonging to other components are not touched.

Required bounded signal set (the names below are the names emitted by the writer or present in the named JSON snapshot; no conceptual aliases are used):

• jhf_bobbin_localai_probe_timeouts_total
• jhf_bobbin_localai_degraded_mode
• jhf_bobbin_localai_metrics_generated_at_seconds
• rebuild_required_rows
• visibility_drift_rows
• checkpoint_regression_rows

jhf_bobbin_localai_metrics_generated_at_seconds is the Unix timestamp written with every guard metrics update. Compare it with the current Unix time to classify the artifact as fresh or stale; do not use the filesystem mtime as the freshness contract. The canonical writer path is exactly /tmp/jhf-bobbin-localai-guard.prom; callers may override it explicitly with LOCALAI_PROBE_METRICS_FILE, but the default is shared by both smoke helpers.

Known Failure Modes

• OpenClaw upgrade breaks the patched plugin
• Qdrant collection shape drifts from expected dimensions/indexes
• LocalAI is reachable but configured model alias is wrong
• target host env/config values drift from repo expectations

Restart / Recovery

• use scripts/reapply_after_openclaw_update.sh after host updates
• restore memory-core via scripts/activate_memory_core.py if semantic memory becomes unstable
• rerun smoke and Qdrant checks after recovery
• runtime-owner backup/restore and tombstone reapply stay owned by jhf-deployment and jhf-openclaw-env
• namespace rebuild remains bounded to repo-owned worker/replay truth via scripts/run_reconciliation_worker.sh and configs/reconciliation-admission-episode-visibility-rebuild.sql
• upgrade shadowing remains shadow namespace -> pointer switch -> previous namespace restore, and must not be improvised outside the runtime-owner rollout

Runtime Dependency Summary

• OpenClaw host
• LocalAI
• Qdrant
• optional OpenAI-compatible LLM endpoint
• operator access to host configuration and extensions path

Packaging Operations

• build package: bash scripts/build_package.sh --version <version>
• publish package: GITEA_TOKEN=*** bash scripts/publish_package.sh --version <version>
• verify package pull + digest: GITEA_TOKEN=*** bash scripts/verify_published_package.sh --version <version>
• canonical contract: docs/ARTIFACT_CONTRACT.md (docs/ARTIFACT_CONTRACT.md)

Scan&Fix Automation

Schnellstart

• preview all matching selected issues and their composed execution prompts:
  • bash scripts/scan_and_fix.sh --dry-run
• target a specific issue:
  • bash scripts/scan_and_fix.sh --issue 93 --dry-run

Dry-Run

• --dry-run prints:
  • selected issue batch metadata
  • current branch/worktree context
  • one fully composed execution prompt per selected issue with the embedded repo-local execution standard
• default issue source is the live Gitea API for this repo
• PR coverage is derived only from currently open Gitea PRs with branch references or closing-keyword references such as Fixes #123; closed/merged PRs and generic follow-up mentions never remove an otherwise-open issue from the executable queue
• use --issues-file /path/to/issues.json or SCAN_AND_FIX_ISSUES_JSON=/path/to/issues.json for offline fixture validation in CI/tests only
• --issues-file is not part of the normal operator path; it exists to keep repo-local harnesses explicit and bounded

Live-Run

• configure a non-interactive runner first:
  • bash scripts/scan_and_fix.sh --executor-cmd 'codex exec -'
• on Windows operator workstations, auto-detection may resolve codex.exe through PowerShell when plain bash PATH lookup cannot see it
• the script auto-loads <local-env-redacted> before issue fetch and runner selection, so shared GITEA_TOKEN and SCAN_AND_FIX_RUNNER settings are picked up without manual export
• run against all matching open non-PR issues, including blocked_external handoff issues that still need owner/readback updates:
  • bash scripts/scan_and_fix.sh --executor-cmd 'codex exec -'
• run against a filtered window:
  • bash scripts/scan_and_fix.sh --executor-cmd 'codex exec -' --since 2d --labels blocked-external --max-issues 3
• host/user overrides are only prompt context for runtime-relevant issues:
  • bash scripts/scan_and_fix.sh --executor-cmd 'codex exec -' --host <internal-runtime-redacted> --user administrator
• repo-owned Windows worktrees must run without manual .git path rewriting; Scan&Fix is expected to resolve repo identity safely there
• fixture issue sources are blocked for normal runs unless SCAN_AND_FIX_ALLOW_FIXTURE_RUN=1 is set for a repo-local harness scenario
• local automation credentials are expected in <local-env-redacted>; treat that file as secret-bearing operator input and never echo values into prompts or evidence
• once the prompt has already been dispatched into Codex, the nested run must work the selected issue directly and must not relaunch scripts/scan_and_fix.sh from inside the same repo

Fehlerbilder

• no matching open issues found
  • current filters excluded all open issues; loosen --since or --labels
• requested issue #... is not open or did not match the active filters
  • the issue was closed or excluded by the active filter set
• batch stops on the first runner failure
  • inspect the failing issue prompt path and runner stderr/stdout before resuming
• no usable runner found; pass --executor-cmd ...
  • this environment has no working default codex CLI path; provide --executor-cmd or set SCAN_AND_FIX_RUNNER in <local-env-redacted> and rerun
• fatal: not a git repository: ...C:/.../.git/worktrees/...
  • this is a Windows worktree regression; the repo-local Scan&Fix contract requires a worktree-safe git command path instead of manual operator rewrites
• curl / Gitea API failure
  • verify repo visibility and base URL reachability before retrying
• fixture issue input is test-only
  • rerun against the live Gitea issue API, or set SCAN_AND_FIX_ALLOW_FIXTURE_RUN=1 only for an intentional repo-local harness run
• runner exits non-zero
  • treat as execution failure; inspect runner stderr/stdout and do not mark the issue processed

Structured Projection Worker (bobbin-graph-worker)

Plan PLAN_BOBBIN_CONTEXT-GRAPHv4.1.md §17 lists bobbin-graph-worker as its own runtime component; §23 defines the projection state machine it walks and §24 defines the structured-domain-event lane as deterministic, without an LLM, with Graphiti as materialization only. Tracked in jhf-bobbin#426.

• component: configs/jhf-semantic-foundation.compose.yml service bobbin-graph-worker, unmodified python:3.12.9-slim, standard library only
• Compose artifact-root contract: every Bobbin bind-mount source is rooted at the required absolute BOBBIN_ARTIFACT_ROOT. Operators that combine this compose file with another repository's base compose must set it to the validated Bobbin artifact directory; relative sources are forbidden because Docker Compose otherwise resolves them against the first compose file.
• entrypoint: scripts/bobbin_graph_worker.py
  • publish --event-json <path|-> — producer path, writes RECEIVED into projection_inbox
  • project [--once] — walks RECEIVED → VALIDATED → WRITE_INTENT_RECORDED → GRAPH_WRITE_STARTED → GRAPH_WRITTEN → GRAPH_WRITE_VERIFIED → ADMISSION_RECORDED → CHECKPOINTED
  • status — bounded readback of inbox states and checkpoints
• contract: configs/bobbin-structured-projection-contract.json (source authority, prescribed entity types, admitted ontology version)
• control DB access: scripts/semantic_control_db_client.py, PostgreSQL wire protocol over socket; no driver is installed and none may be added
• idempotency: the projection_inbox_episode_identity_hash_key UNIQUE constraint, not an application-side existence probe
• failure states are fail-closed: contract violation → quarantined with no graph write, transient failure → retryable_failed, graph/control-DB divergence → reconcile_required, each with a projection_attempts row
• status file: /runtime-state/graph-worker-status.json

Governed CRM legacy re-projection

For the owner decision in jhf-bobbin#546, the six checkpointed legacy spindle_customer rows are re-projected through the existing #449 graph repair lane, not by ad hoc Cypher or SQL. The command is count-gated and source-bound:

python3 /opt/jhf-bobbin/scripts/migrate_tenant_hash_and_governance_properties.py --admit-legacy-crm --expected-count 6
python3 /opt/jhf-bobbin/scripts/migrate_tenant_hash_and_governance_properties.py --admit-legacy-crm --expected-count 6 --apply

Run the dry run first. The mode validates jhf-spindle, crm.organization.linked, spindle_customer, and crm.customer_context, then admits only customer_relationship and uses the worker's existing _graph_parameters and MERGE_STATEMENT. Rows outside that exact boundary, including quarantined CRM rows, remain untouched. A count mismatch exits without writes. The name is a business relationship concept grounded in the Spindle Customer Link Registry contract; crm_account would describe an implementation artifact that is not present in the source event or entity type.

Deployment status: declared only. The component is not deployed on Host172; host materialization stays owned by jhf-openclaw-env, and activation_stage remains S0_not_activated. Repo evidence (CI job graph-worker-projection) covers the code path against throwaway Neo4j/PostgreSQL containers; it does not and cannot establish that the pilot graph contains nodes.

Current Operational Gaps

• bobbin-graph-worker is declared but not materialized on any host; the pilot graph therefore stays empty until the runtime owner deploys it
• no host-probed runtime snapshot in default CI (intentionally operator-run only)
• no centralized live runtime metrics endpoint from this repo
• compatibility matrix maintenance remains operator-evidence-driven

Canonical Ops References

• docs/STANDALONE_VERIFICATION_PROFILE.md (docs/STANDALONE_VERIFICATION_PROFILE.md)
• docs/REGRESSION_CADENCE.md (docs/REGRESSION_CADENCE.md)
• docs/OPERATOR_HANDOVER.md (docs/OPERATOR_HANDOVER.md)
• docs/STACK_CONTAINER_CONTRACT.md (docs/STACK_CONTAINER_CONTRACT.md)

AGPLv3. See ../LICENSE (LICENSE).

Learn more at helpifyr.com.

Workspace Git/Scan Guardrails (Mandatory)

• Gitea is Source of Truth; local Windows workspaces are disposable working copies.
• Never run Codex sessions on <local-workspace-redacted> root; always use a concrete repo path.
• Limit active repo sessions to 2-3 in parallel.
• Before each run in a repo: git fetch --prune, git checkout <branch>, git pull --ff-only.
• No background git discovery loops (git status, git ls-files, worktree scans) without explicit scoped need.
• Automation scripts must run repo-scoped only, never global over <local-workspace-redacted>.

scan_and_fix Standard

• scripts/scan_and_fix.sh must enforce runner timeout + single-run lock + .env fallback to <local-env-redacted> and <local-env-redacted>.
• scripts/scan_open_issues_repo_only.sh must exist and query only current repo open issues via Gitea API.

Workspace Hygiene

• Daily cleanup: stale _worktrees/*, _tmp/*, test-results/*, large temporary artifacts.
• Weekly cleanup: stale local branches/worktrees.
• Never leave valuable artifacts as untracked files in workspace root.

Dirty-State Policy

• Dirty state is allowed while actively implementing.
• Before new scan/automation runs: commit/stash, or use a dedicated worktree.
• Never propagate dirty_unknown states.

Incident Playbook (git.exe storm)

1. Identify parent of git.exe (usually one Codex.exe).
2. Stop only the offending process tree.
3. Restart session on concrete repo path.
4. Reduce parallel sessions.
5. Verify git.exe count drops within 30-60s.