openclaw

Files

Vishal Doshi e91a5b0216 fix: release stale session locks and add watchdog for hung API calls (#18060 )

When a model API call hangs indefinitely (e.g. Anthropic quota exceeded
mid-call), the gateway acquires a session .jsonl.lock but the promise
never resolves, so the try/finally block never reaches release(). Since
the owning PID is the gateway itself, stale detection cannot help —
isPidAlive() always returns true.

This commit adds four layers of defense:

1. **In-process lock watchdog** (session-write-lock.ts)
   - Track acquiredAt timestamp on each held lock
   - 60-second interval timer checks all held locks
   - Auto-releases any lock held longer than maxHoldMs (default 5 min)
   - Catches the hung-API-call case that try/finally cannot

2. **Gateway startup cleanup** (server-startup.ts)
   - On boot, scan all agent session directories for *.jsonl.lock files
   - Remove locks with dead PIDs or older than staleMs (30 min)
   - Log each cleaned lock for diagnostics

3. **openclaw doctor stale lock detection** (doctor-session-locks.ts)
   - New health check scans for .jsonl.lock files
   - Reports PID status and age of each lock found
   - In --fix mode, removes stale locks automatically

4. **Transcript error entry on API failure** (attempt.ts)
   - When promptError is set, write an error marker to the session
     transcript before releasing the lock
   - Preserves conversation history even on model API failures

Closes #18060

2026-02-16 23:59:22 +01:00

protocol

Add mesh auto-planning with chat command UX and hardened auth/session behavior

2026-02-16 23:58:23 +01:00

server

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server-methods

Add mesh auto-planning with chat command UX and hardened auth/session behavior

2026-02-16 23:58:23 +01:00

agent-event-assistant-text.ts

refactor(gateway): dedupe assistant delta parsing

2026-02-15 19:08:47 +00:00

agent-prompt.e2e.test.ts

refactor(gateway): share agent prompt builder

2026-02-14 15:39:45 +00:00

agent-prompt.ts

refactor(gateway): share agent prompt builder

2026-02-14 15:39:45 +00:00

assistant-identity.test.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

assistant-identity.ts

feat(ui): add Agents dashboard

2026-02-02 21:31:17 -05:00

auth-rate-limit.test.ts

feat(gateway): add auth rate-limiting & brute-force protection (#15035 )

2026-02-13 15:32:38 +01:00

auth-rate-limit.ts

feat(gateway): add auth rate-limiting & brute-force protection (#15035 )

2026-02-13 15:32:38 +01:00

auth.test.ts

refactor(test): dedupe trusted-proxy auth test setup

2026-02-16 18:31:37 +00:00

auth.ts

refactor(gateway): share host header parsing

2026-02-15 16:15:53 +00:00

boot.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

boot.ts

fix(gateway): keep boot sessions ephemeral without remapping main

2026-02-16 00:03:21 +00:00

call.test.ts

refactor(test): reuse env snapshots in gateway call tests

2026-02-15 23:22:58 +00:00

call.ts

fix: context overflow compaction and subagent announce improvements (#11664 ) (thanks @tyler6204)

2026-02-07 20:02:32 -08:00

channel-health-monitor.test.ts

feat(gateway): add channel health monitor with auto-restart

2026-02-16 23:53:35 +01:00

channel-health-monitor.ts

feat(gateway): add channel health monitor with auto-restart

2026-02-16 23:53:35 +01:00

chat-abort.test.ts

Gateway/Control UI: preserve partial output on abort (#15026 )

2026-02-15 16:55:28 -08:00

chat-abort.ts

Gateway/Control UI: preserve partial output on abort (#15026 )

2026-02-15 16:55:28 -08:00

chat-attachments.test.ts

fix(security): reject oversized base64 before decode

2026-02-14 15:45:41 +01:00

chat-attachments.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

chat-sanitize.test.ts

fix: hide message_id hints in web chat

2026-01-24 13:52:31 +00:00

chat-sanitize.ts

refactor: dedupe chat envelope + daemon output + skills UI

2026-02-15 03:41:11 +00:00

client.e2e.test.ts

perf: speed up gateway missing-tick e2e watchdog

2026-02-13 19:52:45 +00:00

client.ts

fix: clear stale device-auth token on token mismatch

2026-02-16 23:54:23 +01:00

config-reload.test.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

config-reload.ts

fix: ignore meta field changes in config file watcher (#13460 )

2026-02-12 07:55:26 -06:00

control-ui-contract.ts

refactor(gateway): share Control UI bootstrap contract and CSP

2026-02-16 03:50:39 +01:00

control-ui-csp.test.ts

refactor(gateway): share Control UI bootstrap contract and CSP

2026-02-16 03:50:39 +01:00

control-ui-csp.ts

refactor(gateway): share Control UI bootstrap contract and CSP

2026-02-16 03:50:39 +01:00

control-ui-shared.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

control-ui.http.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

control-ui.ts

refactor(gateway): share Control UI bootstrap contract and CSP

2026-02-16 03:50:39 +01:00

device-auth.ts

feat: enforce device-bound connect challenge

2026-01-20 13:04:19 +00:00

exec-approval-manager.ts

fix(gateway): bind system.run approvals to exec approvals

2026-02-14 13:27:45 +01:00

gateway-cli-backend.live.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

gateway-config-prompts.shared.ts

refactor(gateway): share tailscale prompt constants

2026-02-15 18:06:48 +00:00

gateway-misc.test.ts

test(gateway): move Control UI http coverage

2026-02-16 03:50:39 +01:00

gateway-models.profiles.live.test.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

gateway.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

hooks-mapping.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

hooks-mapping.ts

fix(security): restrict hook transform module loading

2026-02-14 13:46:09 +01:00

hooks.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

hooks.ts

fix(security): enforce bounded webhook body handling

2026-02-13 19:14:54 +01:00

http-auth-helpers.ts

refactor(gateway): share bearer auth helper

2026-02-15 04:40:04 +00:00

http-common.ts

fix(security): enforce bounded webhook body handling

2026-02-13 19:14:54 +01:00

http-endpoint-helpers.test.ts

refactor(gateway): dedupe json endpoint prelude

2026-02-15 13:24:37 +00:00

http-endpoint-helpers.ts

refactor(gateway): dedupe json endpoint prelude

2026-02-15 13:24:37 +00:00

http-utils.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

live-image-probe.ts

refactor: consolidate PNG encoder and safeParseJson utilities (#12457 )

2026-02-09 00:21:54 -08:00

net.test.ts

feat(gateway): add trusted-proxy auth mode (#15940 )

2026-02-14 12:32:17 +01:00

net.ts

refactor(gateway): share host header parsing

2026-02-15 16:15:53 +00:00

node-command-policy.ts

fix(gateway): block node.invoke exec approvals

2026-02-14 19:22:37 +01:00

node-invoke-sanitize.ts

refactor(gateway): centralize node.invoke param sanitization

2026-02-14 13:27:45 +01:00

node-invoke-system-run-approval.ts

fix(node-host): enforce system.run rawCommand/argv consistency

2026-02-14 18:53:23 +01:00

node-registry.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

open-responses.schema.ts

feat(gateway): implement OpenResponses /v1/responses endpoint phase 2

2026-01-20 07:37:01 +00:00

openai-http.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

openai-http.ts

refactor(gateway): dedupe assistant delta parsing

2026-02-15 19:08:47 +00:00

openresponses-http.e2e.test.ts

refactor(test): dedupe openresponses server setup

2026-02-15 23:34:52 +00:00

openresponses-http.ts

refactor(gateway): dedupe assistant delta parsing

2026-02-15 19:08:47 +00:00

openresponses-parity.e2e.test.ts

test(gateway): add OpenResponses parity E2E tests

2026-01-20 07:37:01 +00:00

origin-check.test.ts

fix: harden control ui framing + ws origin

2026-02-03 16:00:57 -08:00

origin-check.ts

refactor(gateway): share host header parsing

2026-02-15 16:15:53 +00:00

probe-auth.ts

refactor(gateway): dedupe probe auth resolution

2026-02-15 06:40:04 +00:00

probe.ts

refactor: consolidate duplicate utility functions (#12439 )

2026-02-08 23:59:43 -08:00

server-broadcast.ts

refactor(gateway): share broadcast function types

2026-02-15 13:39:59 +00:00

server-browser.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

server-channels.ts

gateway: expose isManuallyStopped and resetRestartAttempts on ChannelManager

2026-02-16 23:53:35 +01:00

server-chat.agent-events.test.ts

refactor(test): share server chat event harness

2026-02-15 15:44:14 +00:00

server-chat.ts

Gateway: bound agent run sequence tracking

2026-02-14 17:50:49 -08:00

server-close.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

server-constants.ts

fix(canvas): port remaining iOS branch stability fixes (#18228 )

2026-02-16 16:42:28 +00:00

server-cron.ts

cron: log model+token usage per run + add usage report script

2026-02-16 23:58:38 +01:00

server-discovery-runtime.ts

refactor: rename to openclaw

2026-01-30 03:16:21 +01:00

server-discovery.test.ts

refactor: rename to openclaw

2026-01-30 03:16:21 +01:00

server-discovery.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

server-http.hooks-request-timeout.test.ts

fix(gateway): prune expired entries instead of clearing all hook auth failure state (#15848 )

2026-02-14 01:46:12 +01:00

server-http.ts

fix(gateway): prune expired entries instead of clearing all hook auth failure state (#15848 )

2026-02-14 01:46:12 +01:00

server-lanes.ts

refactor: use command lane enum

2026-01-20 10:51:25 +00:00

server-maintenance.ts

Gateway: bound agent run sequence tracking

2026-02-14 17:50:49 -08:00

server-methods-list.ts

Add mesh auto-planning with chat command UX and hardened auth/session behavior

2026-02-16 23:58:23 +01:00

server-methods.ts

Add mesh auto-planning with chat command UX and hardened auth/session behavior

2026-02-16 23:58:23 +01:00

server-mobile-nodes.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

server-model-catalog.ts

refactor(gateway): split server runtime

2026-01-14 09:11:21 +00:00

server-node-events-types.ts

refactor: remove bridge protocol

2026-01-19 10:08:29 +00:00

server-node-events.test.ts

agents: reduce prompt token bloat from exec and context (#16539 )

2026-02-14 18:32:45 -05:00

server-node-events.ts

refactor(gateway): share node event sessionKey parsing

2026-02-15 18:02:55 +00:00

server-node-subscriptions.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

server-plugins.test.ts

refactor: rename to openclaw

2026-01-30 03:16:21 +01:00

server-plugins.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

server-reload-handlers.ts

refactor: unify gateway restart deferral and dispatcher cleanup

2026-02-14 00:38:18 +01:00

server-restart-deferral.test.ts

perf(test): cut gateway unit suite overhead

2026-02-16 00:18:26 +00:00

server-restart-sentinel.ts

fix: allow agent workspace directories in media local roots (#17136 )

2026-02-15 10:53:45 -05:00

server-runtime-config.test.ts

feat(gateway): add trusted-proxy auth mode (#15940 )

2026-02-14 12:32:17 +01:00

server-runtime-config.ts

feat(gateway): add trusted-proxy auth mode (#15940 )

2026-02-14 12:32:17 +01:00

server-runtime-state.ts

refactor(gateway): share broadcast function types

2026-02-15 13:39:59 +00:00

server-session-key.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

server-shared.ts

refactor(gateway): split server helpers

2026-01-03 19:37:09 +01:00

server-startup-log.ts

chore: Enable "experimentalSortImports" in Oxfmt and reformat all imorts.

2026-02-01 10:03:47 +09:00

server-startup-memory.test.ts

fix (gateway/memory): start qmd onBoot for all agents

2026-02-16 10:35:26 -08:00

server-startup-memory.ts

fix (gateway/memory): start qmd onBoot for all agents

2026-02-16 10:35:26 -08:00

server-startup.ts

fix: release stale session locks and add watchdog for hung API calls (#18060 )

2026-02-16 23:59:22 +01:00

server-tailscale.ts

chore: migrate to oxlint and oxfmt

2026-01-14 15:02:19 +00:00

server-utils.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

server-wizard-sessions.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

server-ws-runtime.ts

feat(gateway): add auth rate-limiting & brute-force protection (#15035 )

2026-02-13 15:32:38 +01:00

server.agent.gateway-server-agent-a.e2e.test.ts

refactor(test): dedupe agent and memory cli test setup

2026-02-16 17:57:45 +00:00

server.agent.gateway-server-agent-b.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.agent.gateway-server-agent.mocks.ts

fix(test): complete gateway plugin registry mock

2026-02-14 23:51:41 +00:00

server.auth.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.canvas-auth.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.channels.e2e.test.ts

refactor(test): share gateway e2e registry helper

2026-02-15 21:41:18 +00:00

server.chat.gateway-server-chat-b.e2e.test.ts

gateway: hard-cap chat.history oversized payloads

2026-02-16 23:56:05 +01:00

server.chat.gateway-server-chat.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.config-apply.e2e.test.ts

perf(test): stabilize e2e harness and reduce flaky gateway coverage

2026-02-13 17:32:14 +00:00

server.config-patch.e2e.test.ts

test(gateway): cover mixed-id config.patch rollback

2026-02-16 08:24:55 -05:00

server.cron.e2e.test.ts

cron: separate webhook POST delivery from announce (#17901 )

2026-02-16 02:36:00 -08:00

server.e2e-registry-helpers.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.e2e-ws-harness.ts

refactor(test): reuse env snapshot in gateway ws harness

2026-02-15 23:56:57 +00:00

server.health.e2e.test.ts

refactor(test): share gateway ws e2e harness

2026-02-15 22:19:08 +00:00

server.hooks.e2e.test.ts

refactor(test): share gateway server start helper

2026-02-15 23:02:27 +00:00

server.impl.ts

gateway: wire channel health monitor into startup with configurable interval

2026-02-16 23:53:35 +01:00

server.ios-client-id.e2e.test.ts

refactor(test): dedupe ios/android gateway client id tests

2026-02-15 23:07:50 +00:00

server.models-voicewake-misc.e2e.test.ts

refactor(test): simplify voicewake env cleanup

2026-02-15 23:34:02 +00:00

server.node-invoke-approval-bypass.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.plugin-http-auth.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.reload.e2e.test.ts

refactor(test): share gateway server start helper

2026-02-15 23:02:27 +00:00

server.roles-allowlist-update.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.sessions-send.e2e.test.ts

fix(security): scope session tools and webhook secret fallback

2026-02-16 03:47:10 +01:00

server.sessions.gateway-server-sessions-a.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.skills-status.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.talk-config.e2e.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

server.ts

chore: migrate to oxlint and oxfmt

2026-01-14 15:02:19 +00:00

session-utils.fs.test.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

session-utils.fs.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

session-utils.test.ts

fix(sessions): resolve transcript paths with explicit agent context (#16288 )

2026-02-14 13:44:51 -05:00

session-utils.ts

perf(gateway): optimize sessions/ws/routing

2026-02-14 12:21:44 +00:00

session-utils.types.ts

fix: /status shows incorrect context percentage — totalTokens clamped to contextTokens (#15114 ) (#15133 )

2026-02-12 23:52:19 -05:00

sessions-patch.test.ts

Agents: add nested subagent orchestration controls and reduce subagent token waste (#14447 )

2026-02-14 22:03:45 -08:00

sessions-patch.ts

Agents: add nested subagent orchestration controls and reduce subagent token waste (#14447 )

2026-02-14 22:03:45 -08:00

sessions-resolve.ts

fix(gateway): normalize session key casing to prevent ghost sessions (#12846 )

2026-02-13 20:42:24 +01:00

test-helpers.e2e.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

test-helpers.mocks.ts

security: redact credentials from config.get gateway responses (#9858 )

2026-02-05 16:34:48 -08:00

test-helpers.openai-mock.ts

chore: Enable "curly" rule to avoid single-statement if confusion/errors.

2026-01-31 16:19:20 +09:00

test-helpers.server.ts

refactor(test): dedupe gateway env restores

2026-02-15 23:18:16 +00:00

test-helpers.ts

refactor(src): split oversized modules

2026-01-14 01:17:56 +00:00

test-http-response.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

test-openai-responses-model.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

test-temp-config.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

test-with-server.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

tools-invoke-http.test.ts

perf(test): mock config stack in tools invoke http tests

2026-02-14 18:46:24 +00:00

tools-invoke-http.ts

refactor(tools): dedupe alsoAllow merge

2026-02-15 16:52:14 +00:00

ws-log.test.ts

fix: add agent context to ws logs

2026-01-17 20:37:36 +00:00

ws-log.ts

refactor(channels): dedupe transport and gateway test scaffolds

2026-02-16 14:59:31 +00:00

ws-logging.ts

Gateway: optimize ws logs in normal mode

2025-12-18 13:27:52 +00:00