Fix clw-llm-failure: OpenClaw LLM service connection or inference failure error

[2024-10-01T12:00:00Z] INFO: ClawLLM init with model=gpt-4o-mini, endpoint=https://api.openai.com/v1
[2024-10-01T12:00:01Z] ERROR: clw-llm-failure: Request to LLM failed. Code: -32603, Message: Internal JSON-RPC error.
[2024-10-01T12:00:01Z] FATAL: Inference aborted. Retries exhausted: 3/3
---

Runtime symptoms include:
- Spikes in CPU usage during retries.
- Increased latency (>10s per call).
- Docker containers restarting if using Claw in containerized envs.

This error blocks all downstream LLM-dependent pipelines, such as RAG systems or chatbots built on OpenClaw.

## 2. Root Cause

`clw-llm-failure` triggers when OpenClaw's ClawLLM client cannot complete an inference request. Core causes:

1. **Network/Connectivity**: Firewall blocks, DNS resolution fails, or proxy misconfig. OpenClaw uses HTTP/2 to LLM providers (OpenAI, Anthropic, etc.).
2. **Authentication**: Invalid/missing `CLAW_LLM_API_KEY` or expired tokens.
3. **Provider-Side Issues**: Model overload (e.g., GPT-4 rate limits), endpoint downtime, or unsupported model names.
4. **Configuration Mismatch**: Incorrect `base_url`, `model_id`, or payload format (e.g., missing `temperature` in JSON).
5. **Resource Exhaustion**: Local timeouts (<30s default), memory leaks in multi-threaded calls.
6. **JSON-RPC Parsing**: Malformed responses from LLM API not handled by ClawLLM v2.1+.

Inspect via `claw_debug --llm` flag for traces. Root cause often correlates with `curl -v` tests to the endpoint.

## 3. Step-by-Step Fix

Resolve `clw-llm-failure` systematically. Start with basics, escalate to code changes.

### Step 1: Verify Environment
```bash
# Check API key and endpoint
echo $CLAW_LLM_API_KEY | wc -c  # >30 chars expected
curl -H "Authorization: Bearer $CLAW_LLM_API_KEY" https://api.openai.com/v1/models

[llm]
endpoint = "https://api.openai.com/v1"
model = "gpt-4o-mini"
timeout_ms = 5000
retries = 1

[llm]
endpoint = "https://api.openai.com/v1"
model = "gpt-4o-mini"
api_key = "${CLAW_LLM_API_KEY}"
timeout_ms = 30000
retries = 5
backoff_ms = 1000
validate_model = true

#include <openclaw/claw_llm.h>

int main() {
    ClawLLM llm("gpt-4o-mini");
    auto response = llm.infer("Hello world");
    if (!response.ok()) {
        std::cerr << "LLM failed: " << response.error() << std::endl;
        return 1;
    }
    std::cout << response.text() << std::endl;
    return 0;
}

#include <openclaw/claw_llm.h>
#include <chrono>
#include <thread>

std::string safe_infer(ClawLLM& llm, const std::string& prompt, int max_retries = 5) {
    for (int attempt = 0; attempt < max_retries; ++attempt) {
        auto response = llm.infer(prompt);
        if (response.ok()) {
            return response.text();
        }
        if (response.error_code() != "clw-llm-failure") {
            throw std::runtime_error("Non-retryable LLM error: " + response.error());
        }
        int backoff = 1000 * (1 << attempt);  // Exponential: 1s, 2s, 4s...
        std::this_thread::sleep_for(std::chrono::milliseconds(backoff));
    }
    throw std::runtime_error("LLM retries exhausted");
}

int main() {
    ClawLLM llm("gpt-4o-mini");
    try {
        auto result = safe_infer(llm, "Hello world");
        std::cout << result << std::endl;
    } catch (const std::exception& e) {
        std::cerr << "Fatal: " << e.what() << std::endl;
        return 1;
    }
    return 0;
}

ClawLLMConfig cfg;
cfg.endpoint = std::getenv("CLAW_LLM_ENDPOINT") ?: "https://api.openai.com/v1";
cfg.model = "gpt-4o-mini";
cfg.fallback_endpoint = "http://localhost:11434";  // Ollama
cfg.fallback_model = "llama3.1:8b";

ClawLLM llm(cfg);

cargo build --release  # If using Rust bindings
claw_test --llm --iterations 100

# Run Claw test suite
claw_test llm_infer --model gpt-4o-mini --prompt "Test" --expect-success

for i in {1..50}; do
  curl -X POST http://localhost:8080/infer -d '{"prompt":"Test"}' | jq .status
done | grep -c "success"  # Expect 100%

tail -f claw.log | grep "clw-llm-failure"  # Should be zero