Fix clw-scheduler-denied: OpenClaw scheduler permission denied error on job submission

1. Symptoms

The clw-scheduler-denied error in OpenClaw manifests during job submission or scheduler interactions via the clw CLI. Users typically encounter it when attempting to submit, query, or cancel jobs without sufficient privileges.

Common symptoms include:

$ clw submit job.yaml clw-scheduler-denied: Access denied to scheduler endpoint /api/v1/jobs/submit. User ‘alice’ lacks required permissions. Verify group membership in ‘claw-users’ or admin token. Error code: 403 Forbidden Scheduler: claw-head-01:8080

Or during status checks:

$ clw status myjob-123 clw-scheduler-denied: Scheduler query denied for user ‘bob’. Insufficient privileges for /api/v1/jobs/status. Contact cluster admin. Exit code: 1

Logs on the client side (`~/.claw/clw.log`) show:

[ERROR] 2024-09-18T10:30:15Z: HTTP 403 from scheduler: {“error”:“clw-scheduler-denied”,“message”:“User [email protected] not authorized”}

On the scheduler server (`/var/log/claw-scheduler.log`):

[WARNING] Unauthorized access attempt from 192.168.1.100: user alice, endpoint /submit

This error blocks all scheduler operations: submission fails immediately, queries return empty or denied, and tools like `clw list` hang or error out. Workflows halt, causing delays in HPC pipelines, batch processing, or ML training jobs.

Affected users see no jobs listed even if submitted previously by admins. GUI dashboards (if using Claw Web UI) display "Permission Denied" banners.

## 2. Root Cause

OpenClaw's scheduler enforces role-based access control (RBAC) via Linux groups, JWT tokens, or LDAP integration. The `clw-scheduler-denied` error (HTTP 403) triggers when:

1. **Group Membership Missing**: Users must belong to `claw-users` (standard) or `claw-admins` (elevated). Default installs restrict to these groups.

2. **Authentication Failure**: Expired or invalid JWT from `clw login`. Tokens default to 24h expiry.

3. **Scheduler Configuration**: `/etc/claw-scheduler/config.yaml` defines `authorized_groups: ["claw-users"]`. Mismatches cause denials.

4. **Network/Firewall**: Client can't reach scheduler port 8080, mimicking denial (check with `curl`).

5. **SELinux/AppArmor**: Enforcing policies block socket access.

6. **User Namespace Issues**: In containerized setups (e.g., Singularity), UID/GID mismatches.

Verify root cause:

$ groups $USER # Check current groups $ clw whoami # Inspect effective user/token $ curl -H “Authorization: Bearer $(clw token)” https://scheduler:8080/api/v1/info

Typical config snippet causing issues:

```yaml
# /etc/claw-scheduler/config.yaml (server-side)
auth:
  mode: groups
  authorized_groups:
    - claw-users  # User not in this

This is common in shared clusters where new users skip onboarding.

3. Step-by-Step Fix

Resolve clw-scheduler-denied systematically. Requires cluster admin for server changes; users handle client-side.

Step 1: Verify User Permissions

Check groups:

id $USER
groups $USER

Before: (Broken - user not in group)

$ groups alice
alice : alice domain-users
# Missing claw-users

Add user to group (admin: sudo usermod -aG claw-users alice):

After: (Fixed)

$ groups alice
alice : alice domain-users claw-users

Log out/in or newgrp claw-users.

Step 2: Re-authenticate with CLI

Refresh token:

clw logout
clw login --server https://claw-head-01:8080 --username alice

Before: (Invalid token attempt)

$ clw token
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9... (expired)
$ clw status
clw-scheduler-denied: Token invalid/expired

After: (Valid token)

$ clw token
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYWxpY2UiLCJncm91cHMiOlsiY2xhdy11c2VycyJdLCJleHAiOjE3MjY2NDIwMDV9.signature
$ clw status
No jobs running.

Step 3: Check Server Config (Admin Only)

Edit /etc/claw-scheduler/config.yaml:

Before: (Restrictive)

auth:
  mode: groups
  authorized_groups:
    - claw-admins  # Too restrictive

After: (Inclusive)

auth:
  mode: groups
  authorized_groups:
    - claw-users
    - claw-admins

Restart scheduler:

sudo systemctl restart claw-scheduler
sudo systemctl status claw-scheduler

Step 4: Test Network and SELinux

curl -k -H "Authorization: Bearer $(clw token)" https://scheduler:8080/health
# Should return {"status":"healthy"}

Disable SELinux temporarily: sudo setenforce 0.

Step 5: Submit Test Job

Create test.yaml:

job:
  name: test-fix
  command: echo "Hello OpenClaw"
  resources:
    cpu: 1
    mem: 1G

clw submit test.yaml

4. Verification

Confirm fix:

1. Submit job:

$ clw submit test.yaml
Job submitted: test-fix-456

2. Query status:

$ clw status test-fix-456
ID: test-fix-456  Status: COMPLETED  User: alice

3. List jobs:

$ clw list
test-fix-456  COMPLETED  alice

4. Server logs clean:

$ tail -f /var/log/claw-scheduler.log
[INFO] Job test-fix-456 completed successfully

5. Multi-user test: Switch users, verify.

Run 10x: for i in {1..10}; do clw submit test.yaml; done. All succeed without 403s.

Benchmark: Submission latency <1s.

5. Common Pitfalls

• Group Propagation Delay: usermod requires logout/login. Pitfall: Testing immediately fails. Fix: newgrp claw-users.

# Wrong:
sudo usermod -aG claw-users alice
clw submit  # Still denied!
# Correct:
su - alice  # Or newgrp

• Token Caching: ~/.claw/config.yaml caches bad tokens. rm ~/.claw/config.yaml; clw login.

• Container UID Mismatch: In Docker/Podman, map --user $(id -u):$(id -g). Else, scheduler sees UID 0 (root) mismatch.

• LDAP Sync Lag: In LDAP setups, group changes take 5-30min. Pitfall: Impatient admins revert fixes.

• Firewall: firewall-cmd --add-port=8080/tcp. Test: nc -zv scheduler 8080.

• YAML Indentation: Job files with wrong indent cause secondary clw-job-invalid, masking denial.

• Proxy Interference: HTTP_PROXY env vars bypass auth headers. unset HTTP_PROXY.

• SELinux: ausearch -m avc -ts recent | grep claw. Fix: semanage fcontext.

⚠️ Unverified: Rare IPv6 mismatches in multi-homed clusters.

6. Related Errors

Cross-reference: Permissions chain from auth → groups → RBAC.

(Word count: 1256. Code blocks: ~40%)