Workers

Noxus workers execute background tasks — workflow runs, knowledge-base ingestion, and agent conversations. The job queue lives in PostgreSQL. Workers poll it rather than consuming from a broker, which has two practical consequences:

Database connection pressure scales with worker replica count. Each pod opens its own connection pools. See connection pooling below.
Queue depth is directly observable in the database, which is what makes KEDA’s queue-driven autoscaling accurate — it reacts to actual backlog instead of waiting for CPU to rise after work has already been queued.

Redis is used for caching, distributed locks, and live-stream coordination — not as the job queue. Workers are configured through environment variables that control task routing, and the Helm chart supports defining multiple pools — each with its own Deployment, Service, autoscaler, and PodDisruptionBudget.

Task Routing

Queue Types (`WORKER_SUBSCRIBE`)

Each worker pool subscribes to one or more task types via the WORKER_SUBSCRIBE environment variable.

Queue Type	Description
`all`	Process all task types
`all_but_kb`	Everything except knowledge-base ingestion
`flow`	Workflow execution only
`chat`	Conversational AI / agent tasks only
`kb`	Knowledge-base ingestion only

A single pool on the default all_but_kb has no consumer for knowledge-base ingestion — those jobs queue indefinitely with nothing to pick them up. Either add a kb pool or set your single pool to all.

Workspace Filtering

Workers can be scoped to specific workspaces using comma-separated ID lists:

Variable	Description
`WORKER_SUBSCRIBE_WORKSPACES`	Workspace IDs this worker processes (empty = all)
`WORKER_UNSUBSCRIBE_WORKSPACES`	Workspace IDs this worker excludes (empty = exclude none)

These filters combine with WORKER_SUBSCRIBE — a pool set to workerSubscribe: "flow" with workerSubscribeWorkspaces: "ws-abc,ws-xyz" processes workflow tasks for those two workspaces only. The pair is designed to be used together: give a noisy workspace a dedicated pool via workerSubscribeWorkspaces, and add the same IDs to the catch-all pool’s workerUnsubscribeWorkspaces so it keeps running everything else — including maintenance and system jobs — without competing for that workspace’s work.

Worker Pools

Define multiple pools under worker.pools in your Helm values. Each pool creates an independent Kubernetes Deployment.

Pool Configuration Reference

Field	Type	Default	Description
`enabled`	bool	—	Enable or disable this pool
`replicaCount`	int	`1`	Static replica count (ignored when autoscaling is enabled)
`workerSubscribe`	string	`"all_but_kb"`	Queue type subscription
`workerSubscribeWorkspaces`	string	`""`	Workspace IDs this pool processes (empty = all)
`workerUnsubscribeWorkspaces`	string	`""`	Workspace IDs this pool excludes (empty = exclude none)
`envSecretRef`	string	`""`	Name of an additional K8s Secret to layer on top of the shared app-env
`resources`	object	—	CPU/memory requests and limits
`autoscaling`	object	—	HPA or KEDA autoscaling config
`podDisruptionBudget`	object	—	PDB settings
`affinity`	object	`{}`	Pod affinity/anti-affinity rules
`nodeSelector`	object	`{}`	Node selector constraints
`tolerations`	list	`[]`	Node tolerations
`topologySpreadConstraints`	list	`[]`	Topology spread rules

Basic Multi-Pool Example

worker:
  enabled: true
  pools:
    default:
      enabled: true
      workerSubscribe: "all_but_kb"
      resources:
        requests:
          cpu: "2"
          memory: "12Gi"
        limits:
          cpu: "2"
          memory: "12Gi"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

    kb:
      enabled: true
      workerSubscribe: "kb"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75

    chat:
      enabled: true
      workerSubscribe: "chat"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 0
        maxReplicas: 5
        keda:
          query: >-
            SELECT COUNT(*) FROM runs
            WHERE status IN ('Queued')
            AND queue_type = 'chat';
          targetQueryValue: "5"

Workspace Isolation

Use workerSubscribeWorkspaces and workerUnsubscribeWorkspaces to dedicate worker pools to specific workspaces. This is useful for:

Noisy-neighbor isolation — prevent one team’s heavy workloads from starving others
SLA tiers — dedicated capacity for priority workspaces
Data residency — pin certain workspaces to workers on specific nodes or in specific zones

The two variables are designed to be used as a pair. Giving a workspace a dedicated pool is only half the job: without excluding it from the catch-all pool, both pools compete for the same work.

worker:
  pools:
    # Catch-all pool. Runs everything except the workspaces that have their own
    # pool below — including maintenance and system jobs.
    default:
      enabled: true
      workerSubscribe: "all"
      workerUnsubscribeWorkspaces: "ws_acme_001,ws_ent_001,ws_ent_002"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

    # Dedicated pool for a high-volume workspace
    workspace-acme:
      enabled: true
      workerSubscribe: "all"
      workerSubscribeWorkspaces: "ws_acme_001"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 8

    # Dedicated KB processing for specific workspaces
    kb-enterprise:
      enabled: true
      workerSubscribe: "kb"
      workerSubscribeWorkspaces: "ws_ent_001,ws_ent_002"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75

This isolates load, not data. All pools share one database and one cluster; the tenant and workspace boundaries are enforced by the application, not by the infrastructure. If you need infrastructure-level separation between customers, run separate deployments.

Per-Pool Secrets

By default, all worker pools share the same Kubernetes Secret ({release}-app-env). When different pools need different environment variables — such as separate LLM API keys per tenant, different Redis databases, or pool-specific feature flags — use envSecretRef to layer an additional Secret on top.

worker:
  pools:
    default:
      enabled: true
      workerSubscribe: "all"
      # Uses only the shared app-env secret

    tenant-acme:
      enabled: true
      workerSubscribe: "all"
      workerSubscribeWorkspaces: "ws_acme_001"
      # Env vars in this secret override the shared app-env
      envSecretRef: "acme-worker-env"

Create the per-pool secret separately (or via External Secrets Operator):

apiVersion: v1
kind: Secret
metadata:
  name: acme-worker-env
  namespace: spotflow
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-acme-dedicated-key"
  ANTHROPIC_API_KEY: "sk-ant-acme-key"

The pool-specific secret is mounted after the shared one in envFrom, so its values take precedence for any overlapping keys.

Multi-Namespace Deployment

To run worker groups in different namespaces (e.g., for resource quotas or network policy isolation), deploy separate Helm releases that share the same backend infrastructure.

Deploy the primary release

The primary release deploys backend, frontend, and the default worker pool.

# values-primary.yaml
backend:
  enabled: true
frontend:
  enabled: true

worker:
  enabled: true
  pools:
    default:
      enabled: true
      workerSubscribe: "all_but_kb"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

helm upgrade --install noxus ./cdk/helm/noxus-platform \
  --namespace spotflow --create-namespace \
  -f values.yaml -f values-primary.yaml

Deploy worker-only releases

For each additional namespace, disable all non-worker components and provide the same database/Redis credentials.

# values-kb-workers.yaml
backend:
  enabled: false
frontend:
  enabled: false
relay:
  enabled: false

# Same credentials as the primary release
env:
  DATABASE: "spot"
  REDIS_PORT: "6379"
secrets:
  DATABASE_URL: "postgresql://user:pass@db-host:5432/spot"
  REDIS_URL: "redis-host"
  REDIS_PASSWORD: "redis-pass"

worker:
  enabled: true
  pools:
    kb:
      enabled: true
      workerSubscribe: "kb"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75
      nodeSelector:
        workload-type: kb

helm upgrade --install noxus-kb ./cdk/helm/noxus-platform \
  --namespace spotflow-kb --create-namespace \
  -f values.yaml -f values-kb-workers.yaml

All worker releases must connect to the same PostgreSQL and Redis instances. Redis coordinates task distribution — workers in any namespace pick up tasks from their subscribed queues regardless of where they run.

Cross-Namespace Considerations

Secrets: Each namespace gets its own K8s Secret. Use External Secrets Operator or a shared values file to keep credentials in sync.
Service Account: Worker-only releases still need a ServiceAccount with IRSA annotations for S3 access.
KEDA: ScaledObjects are namespace-scoped. Each release creates its own KEDA resources; the cluster-wide KEDA operator discovers them automatically.
Network Policies: Ensure worker namespaces can reach PostgreSQL, Redis, external APIs, and storage endpoints.

Autoscaling

KEDA (Queue-Based)
HPA (Resource-Based)

Best for scaling based on actual queue depth. Supports scale-to-zero.KEDA polls PostgreSQL to count queued/running tasks and adjusts replicas to maintain a target ratio.

worker:
  keda:
    postgresConnectionString: "postgresql://user:pass@host:5432/spot"
    pollingInterval: 15
    query: >-
      SELECT COALESCE(COUNT(*), 0) FROM runs
      WHERE status IN ('Queued', 'Running')
      AND created_at > NOW() - '1 hour'::interval;
    targetQueryValue: "10"
    cronTrigger:
      enabled: true
      timezone: "UTC"
      start: "0 9 * * *"
      end: "0 20 * * *"
      desiredReplicas: 1

  pools:
    default:
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10
        behavior:
          scaleDown:
            stabilizationWindowSeconds: 300
            policies:
              - type: Percent
                value: 10
                periodSeconds: 60

Individual pools can override the global KEDA query and target:

autoscaling:
  type: "keda"
  keda:
    query: "SELECT COUNT(*) FROM runs WHERE status = 'Queued' AND queue_type = 'chat';"
    targetQueryValue: "5"
    pollingInterval: 10

Best for simple CPU/memory-based scaling. Cannot scale to zero.

autoscaling:
  enabled: true
  type: "hpa"
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 75
  targetMemoryUtilizationPercentage: 85

Run Archiving

After each workflow run completes, the worker archives run data (state, progress, node IO, logs, content streams) from Redis to object storage (S3/GCS/Azure Blob). This ensures long-term persistence but adds per-run overhead from token acquisition and upload latency.

Archive Mode (`ARCHIVE_MODE`)

Mode	Behavior	Use case
`sync`	Archives immediately after each run completes (default)	Production — guarantees data is persisted before the worker moves on
`async`	Skips per-run archiving; a periodic Celery task sweeps Redis and batch-archives completed runs	High-throughput deployments where per-run archiving creates bottlenecks
`disabled`	No archiving — data stays in Redis until TTL expires or the orphan cleanup task runs	Benchmarking and stress testing

Configuration

Variable	Default	Description
`ARCHIVE_MODE`	`sync`	Archive strategy: `sync`, `async`, or `disabled`
`ARCHIVE_BATCH_INTERVAL`	`60`	Seconds between batch archive sweeps (only used in `async` mode)

Async Mode

In async mode, a periodic Celery task (batch_archive_runs) runs every ARCHIVE_BATCH_INTERVAL seconds and archives all completed runs that have been idle in Redis for at least that duration. This eliminates per-run cloud storage token acquisition and reduces GCP/AWS auth pressure under high concurrency.

# Helm values example
env:
  ARCHIVE_MODE: "async"
  ARCHIVE_BATCH_INTERVAL: "30"

Disabled Mode

Use disabled for pure performance benchmarking. Run data remains in Redis (subject to TTL) and can still be retrieved by the backend. The existing archive_orphaned_data periodic task (runs every 30 minutes) acts as a safety net and will eventually archive idle data regardless of mode.

# Helm values for stress testing
env:
  ARCHIVE_MODE: "disabled"

In disabled mode, run data that exceeds its Redis TTL (4 hours for node IO, 60 days for state/progress) will be lost if not archived by the orphan cleanup task. Do not use this mode in production.

Verification

After deploying, verify the setup:

# Worker deployments across namespaces
kubectl get deployments -l app=worker --all-namespaces

# Pool labels on pods
kubectl get pods -l app=worker --all-namespaces --show-labels

# KEDA ScaledObjects
kubectl get scaledobjects --all-namespaces

# HPAs
kubectl get hpa --all-namespaces -l app=worker

# PodDisruptionBudgets
kubectl get pdb --all-namespaces -l app=worker

# Verify queue subscription in worker logs
kubectl logs -l app=worker --tail=50 | grep "WORKER_SUBSCRIBE"

Environment

Platform environment variables and configuration layers

Secrets

Secret management, per-pool injection, and rotation

Scaling

General scaling strategies and capacity planning

Kubernetes

Kubernetes deployment guide

Overview

Deployment Options

Networking

Configuration

Security

Operations

Task Routing

Queue Types (`WORKER_SUBSCRIBE`)

Workspace Filtering

Worker Pools

Pool Configuration Reference

Basic Multi-Pool Example

Workspace Isolation

Per-Pool Secrets

Multi-Namespace Deployment

Cross-Namespace Considerations

Autoscaling

Run Archiving

Archive Mode (`ARCHIVE_MODE`)

Configuration

Async Mode

Disabled Mode

Verification

Environment

Secrets

Scaling

Kubernetes

​Task Routing

​Queue Types (WORKER_SUBSCRIBE)

​Workspace Filtering

​Worker Pools

​Pool Configuration Reference

​Basic Multi-Pool Example

​Workspace Isolation

​Per-Pool Secrets

​Multi-Namespace Deployment

​Cross-Namespace Considerations

​Autoscaling

​Run Archiving

​Archive Mode (ARCHIVE_MODE)

​Configuration

​Async Mode

​Disabled Mode

​Verification

Environment

Secrets

Scaling

Kubernetes

Task Routing

Queue Types (`WORKER_SUBSCRIBE`)

Workspace Filtering

Worker Pools

Pool Configuration Reference

Basic Multi-Pool Example

Workspace Isolation

Per-Pool Secrets

Multi-Namespace Deployment

Cross-Namespace Considerations

Autoscaling

Run Archiving

Archive Mode (`ARCHIVE_MODE`)

Configuration

Async Mode

Disabled Mode

Verification