Skip to main content

Operations

Tool / Contract Summary​

This page documents the real operational behavior of jhf-dobby: service modes, worker behavior, metrics, degraded semantics, and runtime limits.

Business Value​

  • gives operators an exact view of how Dobby behaves under drift and failure
  • separates normal runtime behavior from planned features
  • provides one place for monitoring and fail-closed semantics

Current Verified State​

Available now:

  • effective service modes with Fabric-driven degradation
  • persistent metrics with bounded degraded output
  • worker loop for revalidation expiry with low-pressure polling defaults and bounded backoff
  • crash recovery through compose restart policy
  • bounded degraded persistence behavior for readiness and metrics

Planned / not in current scope:

  • dedicated background queue platform
  • internal authz policy engine
  • distributed worker coordination

Available Now​

Lifecycle Status​

  • capability class: adaptive-learning
  • mode impact: stack-only
  • lifecycle stage: active
  • runtime kind: service+worker

Service Modes​

  • warmup
  • observe_only
  • proposal_only
  • promotion_enabled

Effective mode behavior:

  • requested mode comes from JHF_DOBBY_SERVICE_MODE
  • effective mode is lowered by Fabric drift
  • persistence loss degrades readiness to warmup

Worker​

  • entrypoint: python -m jhf_dobby.worker.service
  • default poll interval: sixty seconds
  • failure backoff: exponential, starts at sixty seconds, caps at three hundred seconds, adds bounded jitter
  • current responsibility: expire revalidate_required proposals that are still active
  • recovery: marks a failed cycle, backs off, and continues looping without tight retries

Readiness / Drift / Monitoring​

Readiness​

Readiness is healthy only when:

  • Fabric alignment is green
  • persistence is reachable

Degraded readiness still returns HTTP 200, but it explicitly reports degraded status and forced warmup mode.

Drift​

Drift sources include:

  • Fabric surfaces unavailable
  • JARVIS repo entry missing
  • capability class mismatch
  • mode impact mismatch
  • matrix or catalog gaps
  • admission dry-run not green

Metrics​

GET /metrics returns Prometheus text exposition with one sample per line. The route is an internal monitoring surface, not a source of Fabric governance truth and not an external business API.

Important counters and gauges:

  • dobby_signals_emitted_total_family_learning_signal_outcome_*
  • dobby_candidates_proposed_total_risk_class_*_target_type_*
  • dobby_revalidate_required_total
  • dobby_policy_denials_total_reason_code_*
  • dobby_budget_denials_total_reason_code_replay_budget_exhausted
  • dobby_budget_denials_total_reason_code_promotion_velocity_breaker_active
  • dobby_queue_depth_intake
  • dobby_queue_depth_replay
  • dobby_queue_depth_promotion
  • dobby_replay_budget_remaining
  • dobby_promotion_velocity_remaining
  • dobby_service_mode
  • dobby_persistence_ready
  • dobby_metrics_degraded

Semantics:

  • dobby_service_mode{mode="<mode>"} 1 records the effective runtime mode.
  • dobby_persistence_ready is 1 when metric persistence is reachable and 0 when Persistence unavailable handling is active.
  • dobby_metrics_degraded 1 appears only in bounded degraded metrics output after persistence read failure.
  • queue depth metrics are logical runtime indicators, not a broker queue view.
  • missing optional counters mean no observed event for that counter in the current persistence snapshot, not a hidden zero from another truth source.

Failure And Degraded Semantics​

  • Fabric failure: fail closed by lowering effective mode
  • Warp failure: approval checks fail closed
  • Shuttle failure: degraded evidence only; intake still works
  • Bobbin sink missing: degraded Bobbin publication result; proposal state still remains Dobby-owned
  • persistence failure:
    • /ready returns degraded state quickly
    • /metrics returns bounded degraded output
    • mutation routes fail closed with 503
    • worker keeps looping under bounded backoff

Runtime Guardrails​

  • shared-host defaults stay low-pressure:
    • healthchecks default to 60s
    • worker polling defaults to 60s
    • higher-sensitivity healthchecks are opt-in only
  • diagnostics must stay bounded:
    • use timeouts for host commands
    • use docker logs --since ... --tail ... instead of unbounded log streams
    • prefer one-shot snapshots such as docker stats --no-stream
  • restart and rerun behavior:
    • compose file target is restart: "no" (jhf-dobby#212); the running host172 containers still use unless-stopped today and only switch over on the next regular redeploy of this repo (live-apply is WP-E, jhf-openclaw-env#2855) -- crash recovery after that point is owned by the reconcile timer in jhf-openclaw-env#2858, not by compose restart
    • API and worker use stop_grace_period: 20s
    • Postgres uses stop_grace_period: 120s to allow clean WAL flush and shutdown on slower stops
    • repeated verify runs should not create new containers, new compose projects, or long-lived debug processes
    • mem_limit/mem_reservation/cpus/pids_limit/logging.options on all three services come from jhf-openclaw-env deploy/fleet/fleet-resource-manifest.v1.json (ref jhf-openclaw-env#2855)

Optional / Extended​

  • Bobbin sink file path through JHF_DOBBY_BOBBIN_ARTIFACT_SINK
  • Shuttle base URL through JHF_SHUTTLE_BASE_URL
  • Warp bearer token through JHF_WARP_API_TOKEN
  • configurable replay budget and promotion velocity controls

Planned / Not In Current Scope​

  • long-lived replay queue
  • autonomous retraining jobs
  • Dobby-authored governance policies

Known Limits​

  • no per-route auth implementation
  • no automatic replay retry scheduler
  • no external alerting integration in this repo
  • queue depth metrics are logical runtime indicators, not a broker queue view

Scan&Fix Runbook​

Quick start​

bash scripts/scan_and_fix.sh --dry-run

This selects all open non-PR issues by default (severity-first, then updated_at) and prints generated execution prompts that include repo/issue context and the unchanged execution-standard block from prompts/scan_and_fix_execution_standard.md.

Dry-run examples​

bash scripts/scan_and_fix.sh --dry-run --issue 63
bash scripts/scan_and_fix.sh --dry-run --max-issues 1
bash scripts/scan_and_fix.sh --dry-run --executor-cmd "cat"
bash scripts/scan_and_fix.sh --dry-run --labels contract,quality-candidate --max-issues 3
bash scripts/scan_and_fix.sh --dry-run --since 7d --severity-order "critical,high,medium,low"

Live run​

export GITEA_TOKEN=...
bash scripts/scan_and_fix.sh --executor-cmd "codex exec --input-file" --host <internal-runtime-redacted> --user administrator

The script writes one prompt per issue to test-results/scan-and-fix/generated_prompt_issue_<id>.md and executes them sequentially through --executor-cmd, then SCAN_FIX_RUNNER, then fallback codex exec --input-file.

Error patterns​

  • GITEA_TOKEN is required:
    • set token in environment before running.
  • Scan&Fix: no open non-PR issues matched filters:
    • relax --labels / --since or specify --issue.
  • No runner found:
    • set SCAN_FIX_RUNNER explicitly or install codex CLI.
  • API connectivity/auth errors:
    • validate GITEA_BASE_URL, token scope, and repo visibility.

Exceptions / Waivers​

  • Shuttle evidence is optional by design
  • Spool is treated as optional read-only evidence classification only
  • jhf-dobby#27
  • jhf-dobby#31
  • jhf-dobby#34
  • jhf-dobby#36

License: AGPLv3. See ../LICENSE (LICENSE). Learn more at helpifyr.com.

Workspace Git/Scan Guardrails (Mandatory)​

  • Gitea is Source of Truth; local Windows workspaces are disposable working copies.
  • Never run Codex sessions on <local-workspace-redacted> root; always use a concrete repo path.
  • Limit active repo sessions to 2-3 in parallel.
  • Before each run in a repo: git fetch --prune, git checkout <branch>, git pull --ff-only.
  • No background git discovery loops (git status, git ls-files, worktree scans) without explicit scoped need.
  • Automation scripts must run repo-scoped only, never global over <local-workspace-redacted>.

scan_and_fix Standard​

  • scripts/scan_and_fix.sh must enforce runner timeout + single-run lock + .env fallback to <local-env-redacted> and <local-env-redacted>.
  • repo-scoped open-issue discovery must stay inside scripts/scan_and_fix.sh and query only current repo open issues via the Gitea API.

Workspace Hygiene​

  • Daily cleanup: stale _worktrees/*, _tmp/*, test-results/*, large temporary artifacts.
  • Weekly cleanup: stale local branches/worktrees.
  • Never leave valuable artifacts as untracked files in workspace root.

Dirty-State Policy​

  • Dirty state is allowed while actively implementing.
  • Before new scan/automation runs: commit/stash, or use a dedicated worktree.
  • Never propagate dirty_unknown states.

Incident Playbook (git.exe storm)​

  1. Identify parent of git.exe (usually one Codex.exe).
  2. Stop only the offending process tree.
  3. Restart session on concrete repo path.
  4. Reduce parallel sessions.
  5. Verify git.exe count drops within 30-60s.