> ## Documentation Index
> Fetch the complete documentation index at: https://docs.noxus.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Workers

> Configure worker pools, task routing, tenant isolation, and autoscaling for Noxus deployments

Noxus workers execute background tasks — workflow runs, knowledge-base ingestion, and agent conversations. The worker system uses **Redis** as the default message broker and coordination layer. PostgreSQL can also be used as an alternative broker, but Redis is recommended for production deployments due to lower latency and better support for pub/sub patterns.

Workers are configured through environment variables that control task routing, and the Helm chart supports defining **multiple pools** — each with its own Deployment, Service, autoscaler, and PodDisruptionBudget.

## Task Routing

### Queue Types (`WORKER_SUBSCRIBE`)

Each worker pool subscribes to one or more task types via the `WORKER_SUBSCRIBE` environment variable.

| Queue Type   | Description                                |
| ------------ | ------------------------------------------ |
| `all`        | Process all task types                     |
| `all_but_kb` | Everything except knowledge-base ingestion |
| `flow`       | Workflow execution only                    |
| `chat`       | Conversational AI / agent tasks only       |
| `kb`         | Knowledge-base ingestion only              |

### Tenant & Workspace Filtering

Workers can be scoped to specific tenants and/or workspaces using comma-separated ID lists:

| Variable                      | Description                                                       |
| ----------------------------- | ----------------------------------------------------------------- |
| `WORKER_SUBSCRIBE_TENANTS`    | Comma-separated tenant IDs this worker processes (empty = all)    |
| `WORKER_SUBSCRIBE_WORKSPACES` | Comma-separated workspace IDs this worker processes (empty = all) |

These filters combine with `WORKER_SUBSCRIBE` — a worker set to `workerSubscribe: "flow"` with `workerSubscribeTenants: "tenant-abc,tenant-xyz"` will only process workflow tasks for those two tenants.

***

## Worker Pools

Define multiple pools under `worker.pools` in your Helm values. Each pool creates an independent Kubernetes Deployment.

### Pool Configuration Reference

| Field                       | Type   | Default        | Description                                                            |
| --------------------------- | ------ | -------------- | ---------------------------------------------------------------------- |
| `enabled`                   | bool   | —              | Enable or disable this pool                                            |
| `replicaCount`              | int    | `1`            | Static replica count (ignored when autoscaling is enabled)             |
| `workerSubscribe`           | string | `"all_but_kb"` | Queue type subscription                                                |
| `workerSubscribeTenants`    | string | `""`           | Comma-separated tenant IDs (empty = all)                               |
| `workerSubscribeWorkspaces` | string | `""`           | Comma-separated workspace IDs (empty = all)                            |
| `envSecretRef`              | string | `""`           | Name of an additional K8s Secret to layer on top of the shared app-env |
| `resources`                 | object | —              | CPU/memory requests and limits                                         |
| `autoscaling`               | object | —              | HPA or KEDA autoscaling config                                         |
| `podDisruptionBudget`       | object | —              | PDB settings                                                           |
| `affinity`                  | object | `{}`           | Pod affinity/anti-affinity rules                                       |
| `nodeSelector`              | object | `{}`           | Node selector constraints                                              |
| `tolerations`               | list   | `[]`           | Node tolerations                                                       |
| `topologySpreadConstraints` | list   | `[]`           | Topology spread rules                                                  |

### Basic Multi-Pool Example

```yaml theme={null}
worker:
  enabled: true
  pools:
    default:
      enabled: true
      workerSubscribe: "all_but_kb"
      resources:
        requests:
          cpu: "2"
          memory: "12Gi"
        limits:
          cpu: "2"
          memory: "12Gi"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

    kb:
      enabled: true
      workerSubscribe: "kb"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75

    chat:
      enabled: true
      workerSubscribe: "chat"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 0
        maxReplicas: 5
        keda:
          query: >-
            SELECT COUNT(*) FROM runs
            WHERE status IN ('Queued')
            AND queue_type = 'chat';
          targetQueryValue: "5"
```

***

## Tenant Isolation

Use `workerSubscribeTenants` and `workerSubscribeWorkspaces` to dedicate worker pools to specific tenants or workspaces. This is useful for:

* **Noisy-neighbor isolation** — prevent one tenant's heavy workloads from starving others
* **SLA tiers** — dedicated capacity for premium tenants
* **Data residency** — pin certain tenants to workers in specific regions or nodes

```yaml theme={null}
worker:
  pools:
    # Shared pool for all tenants (catch-all)
    default:
      enabled: true
      workerSubscribe: "all"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

    # Dedicated pool for a high-volume tenant
    tenant-acme:
      enabled: true
      workerSubscribe: "all"
      workerSubscribeTenants: "tid_acme_corp_123"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 8

    # Dedicated KB processing for specific workspaces
    kb-enterprise:
      enabled: true
      workerSubscribe: "kb"
      workerSubscribeWorkspaces: "ws_ent_001,ws_ent_002,ws_ent_003"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75
```

***

## Per-Pool Secrets

By default, all worker pools share the same Kubernetes Secret (`{release}-app-env`). When different pools need different environment variables — such as separate LLM API keys per tenant, different Redis databases, or pool-specific feature flags — use `envSecretRef` to layer an additional Secret on top.

```yaml theme={null}
worker:
  pools:
    default:
      enabled: true
      workerSubscribe: "all"
      # Uses only the shared app-env secret

    tenant-acme:
      enabled: true
      workerSubscribe: "all"
      workerSubscribeTenants: "tid_acme_corp_123"
      # Env vars in this secret override the shared app-env
      envSecretRef: "acme-worker-env"
```

Create the per-pool secret separately (or via External Secrets Operator):

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: acme-worker-env
  namespace: spotflow
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-acme-dedicated-key"
  ANTHROPIC_API_KEY: "sk-ant-acme-key"
```

The pool-specific secret is mounted **after** the shared one in `envFrom`, so its values take precedence for any overlapping keys.

***

## Multi-Namespace Deployment

To run worker groups in **different namespaces** (e.g., for resource quotas or network policy isolation), deploy separate Helm releases that share the same backend infrastructure.

<Steps>
  <Step title="Deploy the primary release">
    The primary release deploys backend, frontend, and the default worker pool.

    ```yaml theme={null}
    # values-primary.yaml
    backend:
      enabled: true
    frontend:
      enabled: true

    worker:
      enabled: true
      pools:
        default:
          enabled: true
          workerSubscribe: "all_but_kb"
          autoscaling:
            enabled: true
            type: "keda"
            minReplicas: 1
            maxReplicas: 10
    ```

    ```bash theme={null}
    helm upgrade --install noxus ./cdk/helm/noxus-platform \
      --namespace spotflow --create-namespace \
      -f values.yaml -f values-primary.yaml
    ```
  </Step>

  <Step title="Deploy worker-only releases">
    For each additional namespace, disable all non-worker components and provide the same database/Redis credentials.

    ```yaml theme={null}
    # values-kb-workers.yaml
    backend:
      enabled: false
    frontend:
      enabled: false
    relay:
      enabled: false

    # Same credentials as the primary release
    env:
      DATABASE: "spot"
      REDIS_PORT: "6379"
    secrets:
      DATABASE_URL: "postgresql://user:pass@db-host:5432/spot"
      REDIS_URL: "redis-host"
      REDIS_PASSWORD: "redis-pass"

    worker:
      enabled: true
      pools:
        kb:
          enabled: true
          workerSubscribe: "kb"
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
            limits:
              cpu: "4"
              memory: "16Gi"
          autoscaling:
            enabled: true
            type: "hpa"
            minReplicas: 1
            maxReplicas: 5
            targetCPUUtilizationPercentage: 75
          nodeSelector:
            workload-type: kb
    ```

    ```bash theme={null}
    helm upgrade --install noxus-kb ./cdk/helm/noxus-platform \
      --namespace spotflow-kb --create-namespace \
      -f values.yaml -f values-kb-workers.yaml
    ```
  </Step>
</Steps>

<Note>
  All worker releases **must** connect to the same PostgreSQL and Redis
  instances. Redis coordinates task distribution — workers in any namespace pick
  up tasks from their subscribed queues regardless of where they run.
</Note>

### Cross-Namespace Considerations

* **Secrets**: Each namespace gets its own K8s Secret. Use External Secrets Operator or a shared values file to keep credentials in sync.
* **Service Account**: Worker-only releases still need a ServiceAccount with IRSA annotations for S3 access.
* **KEDA**: ScaledObjects are namespace-scoped. Each release creates its own KEDA resources; the cluster-wide KEDA operator discovers them automatically.
* **Network Policies**: Ensure worker namespaces can reach PostgreSQL, Redis, external APIs, and storage endpoints.

***

## Autoscaling

<Tabs>
  <Tab title="KEDA (Queue-Based)">
    Best for scaling based on actual queue depth. Supports **scale-to-zero**.

    KEDA polls PostgreSQL to count queued/running tasks and adjusts replicas to maintain a target ratio.

    ```yaml theme={null}
    worker:
      keda:
        postgresConnectionString: "postgresql://user:pass@host:5432/spot"
        pollingInterval: 15
        query: >-
          SELECT COALESCE(COUNT(*), 0) FROM runs
          WHERE status IN ('Queued', 'Running')
          AND created_at > NOW() - '1 hour'::interval;
        targetQueryValue: "10"
        cronTrigger:
          enabled: true
          timezone: "UTC"
          start: "0 9 * * *"
          end: "0 20 * * *"
          desiredReplicas: 1

      pools:
        default:
          autoscaling:
            enabled: true
            type: "keda"
            minReplicas: 1
            maxReplicas: 10
            behavior:
              scaleDown:
                stabilizationWindowSeconds: 300
                policies:
                  - type: Percent
                    value: 10
                    periodSeconds: 60
    ```

    Individual pools can override the global KEDA query and target:

    ```yaml theme={null}
    autoscaling:
      type: "keda"
      keda:
        query: "SELECT COUNT(*) FROM runs WHERE status = 'Queued' AND queue_type = 'chat';"
        targetQueryValue: "5"
        pollingInterval: 10
    ```
  </Tab>

  <Tab title="HPA (Resource-Based)">
    Best for simple CPU/memory-based scaling. **Cannot** scale to zero.

    ```yaml theme={null}
    autoscaling:
      enabled: true
      type: "hpa"
      minReplicas: 1
      maxReplicas: 5
      targetCPUUtilizationPercentage: 75
      targetMemoryUtilizationPercentage: 85
    ```
  </Tab>
</Tabs>

***

## Run Archiving

After each workflow run completes, the worker archives run data (state, progress, node IO, logs, content streams) from Redis to object storage (S3/GCS/Azure Blob). This ensures long-term persistence but adds per-run overhead from token acquisition and upload latency.

### Archive Mode (`ARCHIVE_MODE`)

| Mode       | Behavior                                                                                       | Use case                                                                |
| ---------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `sync`     | Archives immediately after each run completes (default)                                        | Production — guarantees data is persisted before the worker moves on    |
| `async`    | Skips per-run archiving; a periodic Celery task sweeps Redis and batch-archives completed runs | High-throughput deployments where per-run archiving creates bottlenecks |
| `disabled` | No archiving — data stays in Redis until TTL expires or the orphan cleanup task runs           | Benchmarking and stress testing                                         |

### Configuration

| Variable                 | Default | Description                                                      |
| ------------------------ | ------- | ---------------------------------------------------------------- |
| `ARCHIVE_MODE`           | `sync`  | Archive strategy: `sync`, `async`, or `disabled`                 |
| `ARCHIVE_BATCH_INTERVAL` | `60`    | Seconds between batch archive sweeps (only used in `async` mode) |

### Async Mode

In `async` mode, a periodic Celery task (`batch_archive_runs`) runs every `ARCHIVE_BATCH_INTERVAL` seconds and archives all completed runs that have been idle in Redis for at least that duration. This eliminates per-run cloud storage token acquisition and reduces GCP/AWS auth pressure under high concurrency.

```yaml theme={null}
# Helm values example
env:
  ARCHIVE_MODE: "async"
  ARCHIVE_BATCH_INTERVAL: "30"
```

### Disabled Mode

Use `disabled` for pure performance benchmarking. Run data remains in Redis (subject to TTL) and can still be retrieved by the backend. The existing `archive_orphaned_data` periodic task (runs every 30 minutes) acts as a safety net and will eventually archive idle data regardless of mode.

```yaml theme={null}
# Helm values for stress testing
env:
  ARCHIVE_MODE: "disabled"
```

<Warning>
  In `disabled` mode, run data that exceeds its Redis TTL (4 hours for node IO, 60 days for state/progress) will be lost if not archived by the orphan cleanup task. Do not use this mode in production.
</Warning>

***

## Verification

After deploying, verify the setup:

```bash theme={null}
# Worker deployments across namespaces
kubectl get deployments -l app=worker --all-namespaces

# Pool labels on pods
kubectl get pods -l app=worker --all-namespaces --show-labels

# KEDA ScaledObjects
kubectl get scaledobjects --all-namespaces

# HPAs
kubectl get hpa --all-namespaces -l app=worker

# PodDisruptionBudgets
kubectl get pdb --all-namespaces -l app=worker

# Verify queue subscription in worker logs
kubectl logs -l app=worker --tail=50 | grep "WORKER_SUBSCRIBE"
```

<CardGroup cols={2}>
  <Card title="Environment" icon="sliders" href="/deployment/configuration/environment">
    Platform environment variables and configuration layers
  </Card>

  <Card title="Secrets" icon="key-round" href="/deployment/configuration/secrets">
    Secret management, per-pool injection, and rotation
  </Card>

  <Card title="Scaling" icon="arrow-up-right" href="/deployment/operations/scaling">
    General scaling strategies and capacity planning
  </Card>

  <Card title="Kubernetes" icon="square-stack" href="/deployment/kubernetes/overview">
    Kubernetes deployment guide
  </Card>
</CardGroup>