Skip to main content

Troubleshooting

Run This Check​

Fast repo-local checks:

python3 scripts/check_docs_contract.py
python3 scripts/check_module_features_contract.py
python3 scripts/check_runtime_materialization_drift.py
python3 scripts/check_live_runtime_contract.py

Host/runtime checks:

bash scripts/plugin_smoke_test.sh
bash scripts/probe_live_host_readonly.sh

Symptom Routing​

Use the repo-owned Bobbin checks first, then hand off by owner:

  • installer/runtime stack drift -> JaddaHelpifyr/jhf-bobbin
  • host/runtime mutation or shared runtime authority -> JaddaHelpifyr/jhf-openclaw-env
  • governance/claims truth mismatch -> JaddaHelpifyr/helpifyr-fabric
  • Dobby provenance or marked-artifact mismatch -> JaddaHelpifyr/jhf-dobby

For every handoff, attach the bounded Bobbin validator output first instead of opening with ad-hoc host guesses.

LocalAI starts but downloads too much​

The tested aio-cpu image is convenient but heavy. For leaner production packaging, swap the image later, but keep the OpenAI-compatible embedding endpoint contract stable.

Mem0 recall/capture returns Bad Request​

Check in this order:

  1. LocalAI /v1/embeddings returns 384 values
  2. Qdrant collection uses 384 + Cosine
  3. The LocalAI compatibility patch is present in vendor/mem0-oss.mjs
  4. Qdrant collection is plain, not native-tenant
  5. Qdrant client in the fork is new enough for your server version

If LocalAI starts returning 500s for oversized embedding inputs, update the override to trim recall queries before search and keep auto-capture to the last 6 messages with per-message compaction. That reduces token burn and keeps small LocalAI hosts from tripping over large recall/capture payloads.

OpenAI SDK embeddings from LocalAI look wrong​

In the validated stack, raw HTTP to LocalAI returned correct 384-d vectors while the SDK path returned an invalid 96-value zero vector. That is why this kit patches the fork to use raw HTTP for LocalAI embeddings.

AIO LocalAI returns 500 for /v1/embeddings​

Symptom:

  • POST /v1/embeddings fails with 500
  • LocalAI logs mention a broken model path under /models/huggingface:/...
  • text-embedding-ada-002 appears present, but embeddings still fail

Cause:

  • the stock aio-cpu image can fall back to an internal huggingface://... embedding reference
  • that fallback can drift away from the working all-MiniLM-L6-v2 definition used by this kit

Fix in this repository:

  1. force MODELS to include /models/text-embedding-ada-002.yaml
  2. bind-mount the known-good localai-model.text-embedding-ada-002.yaml to /models/text-embedding-ada-002.yaml
  3. restart the LocalAI container

Verify:

curl -s http://HOST:8088/v1/models
curl -s http://HOST:8088/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"text-embedding-ada-002","input":"semantic memory check"}'

Expected:

  • /v1/models lists text-embedding-ada-002
  • /v1/embeddings returns a vector with 384 values

Operator guidance:

  • for production stability, slim remains the safer LocalAI profile
  • aio is fine when you want convenience, but only with the explicit model override now built into this kit

LocalAI says backend not found: transformers or sentencetransformers​

Symptom:

  • /v1/embeddings fails even though the model file exists
  • LocalAI logs mention backend not found: transformers or backend not found: sentencetransformers
  • this often appears on latest-aio-cpu when operators expect all-MiniLM-L6-v2 to work through the sentence-transformer backend

Cause:

  • the runtime image does not provide the sentence-transformer backend for that deployment
  • the previous fallback path expected a GGUF artifact that was not guaranteed to exist

Deterministic fix path:

  1. keep the canonical YAML contract:
    • localai-model.text-embedding-ada-002.yaml (configs/localai-model.text-embedding-ada-002.yaml)
  2. ensure the canonical GGUF artifact exists:
    • all-minilm-l6-v2_f16.gguf
  3. ensure installer vars are set:
    • LOCALAI_EMBEDDING_MODEL_FILE=all-minilm-l6-v2_f16.gguf
    • LOCALAI_EMBEDDING_MODEL_URL=https://huggingface.co/LLukas22/all-MiniLM-L6-v2-GGUF/resolve/main/all-minilm-l6-v2_f16.gguf
  4. recreate LocalAI and verify POST /v1/embeddings returns length 384

Why this matters:

  • this path removes backend ambiguity and artifact-missing drift from the critical embedding runtime
  • the smoke path now fails closed when embeddings are not functional

AIO LocalAI keeps downloading unrelated models and Mem0 stays down too long​

Symptom:

  • jhf-bobbin-localai remains in health: starting
  • logs show downloads for speech, image, vision, or other side models
  • OpenClaw logs show transient fetch failed during that time

Fix:

  • reduce MODELS to /models/text-embedding-ada-002.yaml during recovery
  • let the stack expose only the embedding path needed by Mem0 first
  • reintroduce extra AIO models later only if you actually need them

Validated effect:

  • this removed the long startup drag on the live host and allowed Mem0 recall/capture to come back promptly after recreate

OpenClaw update breaks semantic memory​

Run:

bash scripts/reapply_after_openclaw_update.sh

Likely owner:

  • JaddaHelpifyr/jhf-bobbin

Evidence to collect:

  • bash scripts/fabric-selfcheck.sh
  • python3 scripts/check_runtime_materialization_drift.py

Escalation path:

  • if the repro shows host-owned runtime drift instead of Bobbin script/config drift, hand off to jhf-openclaw-env with the bounded evidence bundle

If still broken, roll back to memory-core:

python3 scripts/activate_memory_core.py
cd /root/openclaw
docker compose up -d --force-recreate openclaw-gateway

apply_patch warning still appears​

That warning is unrelated to the semantic-memory stack. It comes from the OpenClaw runtime/profile tool allowlist and does not by itself mean Mem0 is broken.

plugins.entries.openclaw-mem0 warns about compiled runtime output​

The installed Mem0 fork must not remain as a source-only index.ts package inside OpenClaw's managed extensions root.

If the gateway runtime moves to a newer Node ABI than the one that last wrote better_sqlite3.node, Mem0 recall/capture fails closed until the native module is reconciled. Bobbin's install and reapply lanes now normalize the runtime plugin tree to node:node and rebuild better-sqlite3 plus sqlite3 inside openclaw-gateway before restart. When this still fails, inspect bounded openclaw-gateway logs for fresh ABI mismatch evidence and verify the runtime plugin tree under /home/node/.openclaw/extensions/openclaw-mem0 is writable by node:node.

Run:

bash scripts/reapply_after_openclaw_update.sh

That re-materializes dist/index.js and writes openclaw.runtimeExtensions so OpenClaw can resolve the built runtime peer instead of warning about a missing compiled output.

Qdrant external host vs local host​

If you already have a Qdrant server elsewhere:

  • set USE_LOCAL_QDRANT=0
  • set QDRANT_URL=http://HOST:6333
  • rerun the installer

Likely owner:

  • JaddaHelpifyr/jhf-bobbin for contract/config posture
  • runtime owner for the external Qdrant endpoint itself

Retired model references​

Do not configure or revive a retired local model through this kit. The model used for extraction is supplied through the configured MEM0_LLM_MODEL binding and must be admitted by the active model policy. Use the current provider-neutral model profile, its recorded revision, and its capability receipt when diagnosing memory quality or model selection.

Want to disable Mem0 but keep the stack ready​

Run:

python3 scripts/activate_memory_core.py
cd /root/openclaw
docker compose up -d --force-recreate openclaw-gateway

LocalAI and Qdrant can remain running for later re-enable.

Evidence Before Escalation​

Before escalating a Bobbin incident, collect:

python3 scripts/check_docs_contract.py
python3 scripts/check_module_features_contract.py
python3 scripts/check_runtime_materialization_drift.py
python3 scripts/check_live_runtime_contract.py

If the issue is runtime-only and host readback is needed after these bounded checks, continue with bash scripts/probe_live_host_readonly.sh and attach the output to the owner issue.

Agents hit session file locked and suddenly switch provider/model​

Symptom:

  • logs show session file locked (timeout 10000ms)
  • the same agent then shows model fallback decisions through older provider aliases before falling through to backups
  • cicd-ops, main, or hocksie are affected most often

Important diagnosis:

  • this is usually not a Mem0 recall bug
  • the provider switch is often a downstream effect of the lock/timeout
  • in the validated live stack, the biggest trigger was the older agent-native post-compaction-recovery cron jobs before they were converted to the lightweight file-only variant

Why this happens:

  • those recovery jobs run with isolated cron session keys
  • but they still inspect the agent's recent state
  • that can contend with the hot interactive agent:*:main session for the same agent
  • once the session path stalls, the model request times out and OpenClaw falls through its fallback chain
  • a separate validated failure mode was using host-only paths like /root/openclaw/... inside the gateway container; the lightweight jobs must read the mounted container paths under /home/node/.openclaw/...

Current fix boundary:

  1. treat legacy post-compaction recovery jobs as jhf-openclaw-env runtime-owner drift, not as a Bobbin-owned helper path
  2. reconcile the live runtime against the admitted OpenClaw model/recovery posture instead of reintroducing the retired Bobbin-local legacy recovery migration helper
  3. keep Bobbin focused on Mem0 / LocalAI / Qdrant memory enablement, not agent-cron model policy

Verify:

openclaw cron list --json
timeout 20s docker logs --since 30m --tail 400 openclaw-gateway 2>&1 | grep -E 'session file locked|model fallback decision'

Expected:

  • no Bobbin-owned recovery helper script is required to reconcile the host
  • any remaining recovery exception is justified by the runtime owner posture, not by this repository
  • legacy recovery jobs no longer depend on a Bobbin-local hardcoded legacy local-Ollama migration path
  • lock spikes and surprise provider switches drop sharply

Log-read safety note:

  • on live hosts, use bounded log probes only (timeout + --since + --tail)
  • avoid unbounded docker logs calls in operational verification paths

Repeated LocalAI timeout bursts and recreate churn​

Symptom:

  • frequent timeout/grpc-style errors around LocalAI probes
  • repeated recreate attempts increase host CPU pressure

Guarded recovery in this repo:

  1. use scripts/localai_probe_guard.py for /readyz and /v1/models probes
  2. allow the guard to enter temporary degraded mode when timeout bursts cross threshold
  3. do not run tight recreate loops while degraded mode is active

Useful checks:

python3 scripts/localai_probe_guard.py --endpoint models --base-url http://<internal-runtime-redacted>:8088/v1
cat /tmp/jhf-bobbin-localai-guard.prom

Tuning knobs (installer env):

  • LOCALAI_PROBE_DEGRADE_THRESHOLD
  • LOCALAI_PROBE_TIMEOUT_WINDOW_SECONDS
  • LOCALAI_PROBE_DEGRADE_COOLDOWN_SECONDS
  • LOCALAI_MODELS_MIN_INTERVAL_SECONDS

AGPLv3. See ../LICENSE (LICENSE).

Learn more at helpifyr.com.