Fix clw-worker-denied: Worker Access Denied by Cluster Security Policy

1. Symptoms

When the clw-worker-denied error occurs, users typically observe the following symptoms:

In the OpenClaw Dashboard:

Worker status shows as REJECTED instead of READY
Task queue displays workers but marks them with a red deny icon
Jobs remain in PENDING state without worker assignment

In the Worker Logs:

[2025-01-15 10:42:18] INFO Connecting to cluster at https://cluster.openclaw.io [2025-01-15 10:42:19] ERROR clw-worker-denied: Worker registration rejected by cluster security policy [2025-01-15 10:42:19] ERROR Reason: Worker identity does not match allowed patterns for namespace ‘production’ [2025-01-15 10:42:19] INFO Retrying in 30 seconds…


**In the Cluster Controller Logs:**

WARN Worker registration denied for worker-id=wrk_8x7k2m3n: Pod service account ‘default’ does not have required cluster role ‘openclaw-worker’ role=worker-binding namespace=production violation=security-policy-check


**Additional Symptoms:**
- Worker pod starts successfully but immediately terminates
- `kubectl get pods` shows worker pods in `CrashLoopBackOff` or `Error` state
- API server returns HTTP 403 when worker attempts to register
- No tasks are being pulled from the queue despite available workers

---

## 2. Root Cause

The `clw-worker-denied` error occurs when the OpenClaw cluster's security policy rejects a worker registration attempt. This rejection can happen for several reasons:

**Primary Causes:**

1. **RBAC Permission Mismatch**: The worker pod's service account lacks the necessary cluster role bindings required by OpenClaw. The worker needs specific permissions to read jobs, update status, and communicate with the cluster API.

2. **Worker Identity Validation Failure**: OpenClaw's security policy validates worker identities against allowed patterns (regex-based). If the worker's generated identity doesn't match the configured allowed patterns, registration is denied.

3. **Namespace Isolation Violation**: Workers in one namespace attempting to register to a cluster configured to only accept workers from specific namespaces will be rejected.

4. **Expired or Invalid Authentication Token**: Workers authenticate using tokens that may have expired or been revoked through the cluster's token rotation policy.

5. **Mutual TLS (mTLS) Certificate Issues**: In clusters with strict mTLS policies, if the worker's certificate doesn't pass the cluster's CA validation, access is denied.

The error essentially means: "The cluster received a valid registration request, but the worker's credentials or identity failed the security policy checks, so access was denied."

---

## 3. Step-by-Step Fix

### Step 1: Verify Worker Service Account Configuration

Check if your worker deployment has the correct service account and RBAC bindings:

```bash
# Check current service account for worker pods
kubectl get pod <worker-pod-name> -n <namespace> -o jsonpath='{.spec.serviceAccountName}'

# List existing cluster role bindings
kubectl get clusterrolebinding -A | grep -i openclaw

# Check if openclaw-worker cluster role exists
kubectl get clusterrole openclaw-worker

Step 2: Create or Update RBAC Configuration

Before:

# worker-deployment.yaml (incomplete - missing RBAC)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openclaw-worker
  namespace: production
spec:
  replicas: 3
  template:
    spec:
      serviceAccountName: default  # ❌ Wrong service account
      containers:
      - name: worker
        image: openclaw/worker:latest

After:

# worker-deployment.yaml (with proper RBAC)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: openclaw-worker-sa
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: openclaw-worker
rules:
- apiGroups: ["openclaw.io"]
  resources: ["jobs", "tasks"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["openclaw.io"]
  resources: ["workers"]
  verbs: ["get", "update", "patch"]
- apiGroups: [""]
  resources: ["pods", "configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: openclaw-worker-binding
subjects:
- kind: ServiceAccount
  name: openclaw-worker-sa
  namespace: production
roleRef:
  kind: ClusterRole
  name: openclaw-worker
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openclaw-worker
  namespace: production
spec:
  replicas: 3
  template:
    spec:
      serviceAccountName: openclaw-worker-sa  # ✅ Correct service account
      containers:
      - name: worker
        image: openclaw/worker:latest
        env:
        - name: WORKER_IDENTITY_PATTERN
          value: "^worker-[a-z0-9]{8}-[a-z0-9]{4}$"

Step 3: Update Cluster Security Policy (If Needed)

If the cluster uses custom identity validation patterns, ensure your worker identity matches:

# Check cluster security policy configuration
kubectl get configmap openclaw-security-policy -n openclaw-system -o yaml

# If pattern mismatch, update worker's identity generation
# Edit the worker config to match allowed patterns
kubectl edit configmap worker-config -n production

Before:

# worker-config.yaml
worker:
  identity_generation: auto
  cluster:
    endpoint: https://cluster.openclaw.io
    namespace: production
    # identity_pattern not set - uses default

After:

# worker-config.yaml
worker:
  identity_generation: auto
  identity_pattern: "worker-[a-z0-9]{8}-[a-z0-9]{4}"  # Must match cluster policy
  cluster:
    endpoint: https://cluster.openclaw.io
    namespace: production
    tls_verify: true
    ca_bundle: "/etc/openclaw/certs/ca.crt"

Step 4: Apply Changes and Restart Workers

# Apply RBAC and deployment changes
kubectl apply -f worker-deployment.yaml

# Delete existing worker pods to force restart
kubectl delete pod -l app=openclaw-worker -n production

# Verify new pods are running
kubectl get pods -n production -l app=openclaw-worker

# Check logs for successful registration
kubectl logs -l app=openclaw-worker -n production | grep -E "(registered|READY|clw-worker-denied)"

Step 5: Verify Worker Registration

# Check worker status via OpenClaw CLI
openclaw worker list --namespace production

# Expected output:
# WORKER ID          STATUS    NAMESPACE    LAST SEEN
# worker-8x7k2m3n    READY     production   2025-01-15T10:45:00Z
# worker-9y2n5p7q    READY     production   2025-01-15T10:45:01Z

4. Verification

After applying the fix, verify the worker is properly registered and tasks are being processed:

Method 1: Check Worker Status

# List all workers and their status
openclaw worker list --all-namespaces

# Check for READY status
openclaw worker list --namespace production --format json | jq '.workers[] | select(.status=="READY")'

Method 2: Monitor Worker Logs

# Stream worker logs for successful registration message
kubectl logs -f -l app=openclaw-worker -n production | grep -E "Successfully registered|Worker status: READY"

# Expected output after fix:
# [2025-01-15 10:45:00] INFO  Successfully registered with cluster
# [2025-01-15 10:45:00] INFO  Worker status: READY
# [2025-01-15 10:45:01] INFO  Pulling task from queue...

Method 3: Submit Test Job

# Create a simple test job
cat <<EOF | kubectl apply -f -
apiVersion: openclaw.io/v1
kind: Job
metadata:
  name: test-job-verification
  namespace: production
spec:
  task:
    image: busybox
    command: ["echo", "Worker verification successful"]
  replicas: 1
EOF

# Monitor job execution
openclaw job get test-job-verification --namespace production

# Check for completion
openclaw job list --namespace production --status completed

Method 4: Verify RBAC Permissions

# Test if worker service account has correct permissions
kubectl auth can-i get jobs --namespace production --as=system:serviceaccount:production:openclaw-worker-sa
# Expected: yes

kubectl auth can-i update tasks --namespace production --as=system:serviceaccount:production:openclaw-worker-sa
# Expected: yes

# Verify cluster role exists
kubectl get clusterrole openclaw-worker -o jsonpath='{.rules}' | jq length
# Expected: 3 (one rule for each resource group)

5. Common Pitfalls

When fixing clw-worker-denied, developers frequently encounter these issues:

1. Namespace Mismatch Between Worker and Cluster Configuration Always ensure the worker’s configured namespace matches the actual deployment namespace. A worker deployed in staging but configured to connect to production cluster will be rejected by namespace isolation policies.

# Verify namespace alignment
kubectl get deployment openclaw-worker -n staging -o jsonpath='{.spec.template.spec.serviceAccountName}'
# Check deployment environment variable
kubectl get deployment openclaw-worker -n staging -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name=="CLAW_NAMESPACE")'

2. Using Default Service Account Instead of Dedicated One The default service account rarely has the necessary permissions. Create a dedicated service account with specific RBAC bindings.

# Don't do this (security risk + won't work)
serviceAccountName: default

# Do this instead
serviceAccountName: openclaw-worker-sa

3. Forgetting to Apply RBAC Changes Before Worker Deployment Workers start immediately upon pod creation. If RBAC bindings aren’t in place, the first registration attempt will fail, potentially triggering CrashLoopBackOff.

4. Identity Pattern Regex Syntax Errors When configuring WORKER_IDENTITY_PATTERN, ensure the regex is valid:

# Invalid pattern (unclosed bracket) - will cause registration denial
value: "^worker-[a-z0-9]{8}-[a-z0-9]{4}$"  # This is correct
# value: "^worker-[a-z0-9]{8"  # This is invalid

# Test your regex before deploying
echo "worker-8x7k2m3n-a2b3" | grep -E "^worker-[a-z0-9]{8}-[a-z0-9]{4}$"
# Should return the matching line

5. Certificate Rotation Without Updating Workers If your cluster rotates CA certificates, worker pods using old certificates will be denied. Configure certificate reloading:

# Add to worker deployment
env:
- name: OPENCLAW_TLS_RELOAD
  value: "true"
- name: OPENCLAW_CA_BUNDLE
  value: "/etc/openclaw/certs/ca.crt"

6. Not Checking Cluster Security Policy Updates Cluster administrators may update security policies without notifying workers. Monitor for configuration drift:

# Add to monitoring script
kubectl get configmap openclaw-security-policy -n openclaw-system -o yaml > /tmp/current-policy.yaml
diff /tmp/previous-policy.yaml /tmp/current-policy.yaml

7. Multiple Workers with Identical Identities If multiple workers generate the same identity, the cluster will deny duplicates. Ensure identity generation includes sufficient entropy:

# Check for identity collisions in logs
kubectl logs -l app=openclaw-worker -n production | grep "identity already registered"
# If found, increase identity entropy in worker config

The following errors are commonly encountered alongside or instead of clw-worker-denied:

clw-auth-invalid

Symptom: Worker sends valid credentials but they’re rejected as malformed or expired.

[2025-01-15 10:42:18] ERROR clw-auth-invalid: Token signature verification failed

Relationship: Often precedes clw-worker-denied when authentication tokens are invalid.

clw-cluster-unreachable

Symptom: Worker cannot establish network connection to cluster API.

[2025-01-15 10:42:18] ERROR clw-cluster-unreachable: Connection refused on port 443

Relationship: May appear if security policy temporarily blocks access, causing registration attempts to fail.

clw-task-timeout

Symptom: Worker successfully registers but tasks exceed configured timeout.

[2025-01-15 10:50:00] WARN  clw-task-timeout: Task task_abc123 exceeded 300s timeout

Relationship: Can occur after successful registration if worker runs out of quota.

clw-pod-evicted

Symptom: Worker pod is evicted due to resource pressure or node termination.

[2025-01-15 11:00:00] ERROR clw-pod-evicted: Pod terminated due to node pressure

Relationship: May cause workers to re-register, triggering additional clw-worker-denied if RBAC state is inconsistent during rapid restarts.

clw-worker-notfound

Symptom: Cluster has no record of the worker attempting to register.

[2025-01-15 10:42:18] ERROR clw-worker-notfound: Worker wrk_8x7k2m3n not registered in cluster registry

Relationship: May indicate that security policy change deleted worker records before re-registration.

For persistent clw-worker-denied issues that remain unresolved after following this guide, contact your cluster administrator to verify that no recent security policy changes were applied to the OpenClaw cluster configuration.