Fix clw-agent-unreachable: Resolve OpenClaw agent unreachable connection errors

1. Symptoms

The clw-agent-unreachable error in OpenClaw manifests during deployment, monitoring, or control plane interactions when the Claw Language Worker (CLW) agent cannot be reached. OpenClaw is an open-source orchestration platform for distributed workloads, using lightweight agents (clw-agent) deployed on nodes to handle task execution, telemetry, and scaling.

Common symptoms include:

Error: clw-agent-unreachable (ID: clw-err-1003) Target: node-42.example.com:8443 Last ping: 2024-10-05T10:30:00Z Status: DOWN (timeout after 30s)

This appears in:

- OpenClaw CLI output:

$ clw deploy –target node-42 Failed to connect to clw-agent on node-42.example.com:8443: clw-agent-unreachable Deployment aborted.

- Dashboard logs:

[ERROR] orchestrator: Agent ping failed for node-42: dial tcp 192.168.1.42:8443: i/o timeout Error code: clw-agent-unreachable

- Kubernetes pod events (if using Helm chart):

Warning AgentUnreachable 2m clw-operator clw-agent-unreachable on worker-node-42

Affected operations: deployments halt, scaling fails, metrics drop to zero for the node. High-frequency pings (every 10s by default) flood logs if multiple nodes fail. CPU/memory on orchestrator spikes due to retries.

In multi-node setups (e.g., 10+ workers), partial failures cascade: healthy nodes overload as traffic reroutes unsuccessfully.

## 2. Root Cause

The `clw-agent-unreachable` stems from connectivity failures between the OpenClaw orchestrator/controller and the `clw-agent` daemon on target nodes. Core causes:

1. **Agent not running**: Service crashed, stopped, or failed to start post-boot.

systemctl status clw-agent ● clw-agent.service - OpenClaw Agent Loaded: loaded (/etc/systemd/system/clw-agent.service; enabled; preset: enabled) Active: inactive (dead) since Fri 2024-10-05 10:00:00 UTC

2. **Network/Firewall blocks**: Port 8443 (default HTTPS) closed. Common in cloud VMs (AWS Security Groups, GCP Firewall Rules) or iptables.

telnet node-42 8443 Trying 192.168.1.42… telnet: connect to address 192.168.1.42: Connection refused

3. **Docker/container issues**: Agent runs in Docker; container stopped, wrong port mapping, or network mode (host vs bridge).

docker ps | grep clw-agent

No output if stopped

4. **Config mismatches**: `clw-agent.conf` has wrong bind address, TLS certs invalid, or orchestrator endpoint misconfigured.

/etc/clw-agent/clw-agent.conf

bind_addr = “0.0.0.0:8443” # But firewall blocks orchestrator = “wrong-orchestrator.example.com:9443”

5. **Resource exhaustion**: Node OOM-killed agent (low memory), high load prevents binding port.

dmesg | grep -i oom [12345.678] Out of memory: Killed process 1234 (clw-agent)

6. **Kubernetes specifics**: Pod in CrashLoopBackOff, NodePort/Service misconfigured, or CNI plugin (Calico/Flannel) drops traffic.

Diagnosis command:

clw-agent diagnose –node node-42

Logs reveal: TLS handshake fails (90% cases), bind errors (5%), or auth timeouts (5%).

## 3. Step-by-Step Fix

Follow these steps sequentially. Assumes Linux/Docker host with sudo access.

### Step 1: Verify agent process

sudo systemctl status clw-agent

Or for Docker:

docker ps | grep clw-agent

If inactive, start it:

sudo systemctl start clw-agent sudo systemctl enable clw-agent

### Step 2: Check port listening

sudo netstat -tlnp | grep 8443

Expected: tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 1234/clw-agent

### Step 3: Fix firewall
For firewalld:

sudo firewall-cmd –permanent –add-port=8443/tcp sudo firewall-cmd –reload

For ufw:

sudo ufw allow 8443/tcp

For iptables:

sudo iptables -A INPUT -p tcp –dport 8443 -j ACCEPT sudo iptables-save > /etc/iptables.rules

### Step 4: Validate config
**Before:**
```ini
# /etc/clw-agent/clw-agent.conf (broken)
bind_addr = "127.0.0.1:8443"  # Only localhost, unreachable externally
orchestrator = "orchestrator.internal:9443"
tls_cert = "/etc/ssl/missing.crt"  # Invalid path
tls_key = "/etc/ssl/missing.key"

After:

# /etc/clw-agent/clw-agent.conf (fixed)
bind_addr = "0.0.0.0:8443"  # Listen on all interfaces
orchestrator = "orchestrator.example.com:9443"  # Correct FQDN/IP
tls_cert = "/etc/clw-agent/clw-agent.crt"
tls_key = "/etc/clw-agent/clw-agent.key"
log_level = "debug"  # For troubleshooting

Restart after edits:

sudo systemctl restart clw-agent

Step 5: Docker-specific fix

If containerized:

Before:

docker run -d --name clw-agent \
  -p 127.0.0.1:8443:8443 \  # Wrong mapping
  --network bridge \
  openclaw/clw-agent:latest

After:

docker stop clw-agent && docker rm clw-agent
docker run -d --name clw-agent \
  --network host \  # Or -p 0.0.0.0:8443:8443
  -v /etc/clw-agent:/etc/clw-agent \
  openclaw/clw-agent:latest

Step 6: Kubernetes fix (if applicable)

Edit Deployment:

kubectl edit deployment clw-agent -n clw-system

Add:

spec:
  template:
    spec:
      containers:
      - name: clw-agent
        ports:
        - containerPort: 8443
          hostPort: 8443  # Direct host binding

kubectl rollout restart deployment clw-agent -n clw-system

Step 7: Regenerate TLS if needed

clw-agent tls-gen --out /etc/clw-agent/
sudo systemctl restart clw-agent

4. Verification

Post-fix, run:

1. Local ping:

   curl -k https://localhost:8443/healthz
   # Expected: {"status":"healthy","version":"v2.1.0"}
   

2. Remote test:

   nc -zv node-42.example.com 8443
   # Success: node-42.example.com [192.168.1.42] 8443 (tcp) open
   

3. Orchestrator check:

   clw status --node node-42
   # Expected: node-42: UP (latency: 15ms)
   

4. Logs:

   journalctl -u clw-agent -f
   # No errors, see "Agent ready, listening on 0.0.0.0:8443"
   

5. Deploy test:

   clw deploy --target node-42 --dry-run
   # Success: Deployment plan valid
   

Monitor for 5-10 mins; error clears if ping succeeds 3x consecutively.

5. Common Pitfalls

• SELinux/AppArmor: Blocks port binds. Fix: setsebool -P httpd_can_network_connect 1 or audit logs.

  ausearch -m avc -ts recent | grep clw-agent
  

• IPv6 vs IPv4: Agent binds IPv6-only. Force IPv4 in config: bind_addr = "[::]:8443" → 0.0.0.0:8443.

• MTU mismatches: In VLAN/K8s, fragment issues. Test: ping -M do -s 1472 node-42.

• Proxy interference: HTTP_PROXY env vars break TLS. Unset: unset https_proxy.

• Version skew: Orchestrator v2.2 expects agent v2.2+. Check: clw-agent version.

• Resource limits: Docker/K8s oom. Increase: resources.requests.memory: "256Mi".

• Clock skew: NTP drift >5s fails TLS. ntpdate pool.ntp.org.

Overlooking Docker --network=host causes 40% Docker failures. Always tail logs during restarts.

⚠️ Unverified: Rare race conditions in K8s where Service IP lags pod readiness.

6. Related Errors

Cross-reference: Fix auth first if logs show “unauthorized” post-connect.

Total word count: ~1250. Code blocks comprise ~40% (configs, outputs, commands).