Workers

Noxus workers execute background tasks — workflow runs, knowledge-base ingestion, and agent conversations. The worker system uses Redis as the default message broker and coordination layer. PostgreSQL can also be used as an alternative broker, but Redis is recommended for production deployments due to lower latency and better support for pub/sub patterns. Workers are configured through environment variables that control task routing, and the Helm chart supports defining multiple pools — each with its own Deployment, Service, autoscaler, and PodDisruptionBudget.

Task Routing

Queue Types (`WORKER_SUBSCRIBE`)

Each worker pool subscribes to one or more task types via the WORKER_SUBSCRIBE environment variable.

Queue Type	Description
`all`	Process all task types
`all_but_kb`	Everything except knowledge-base ingestion
`flow`	Workflow execution only
`chat`	Conversational AI / agent tasks only
`kb`	Knowledge-base ingestion only

Tenant & Workspace Filtering

Workers can be scoped to specific tenants and/or workspaces using comma-separated ID lists:

Variable	Description
`WORKER_SUBSCRIBE_TENANTS`	Comma-separated tenant IDs this worker processes (empty = all)
`WORKER_SUBSCRIBE_WORKSPACES`	Comma-separated workspace IDs this worker processes (empty = all)

These filters combine with WORKER_SUBSCRIBE — a worker set to workerSubscribe: "flow" with workerSubscribeTenants: "tenant-abc,tenant-xyz" will only process workflow tasks for those two tenants.

Worker Pools

Define multiple pools under worker.pools in your Helm values. Each pool creates an independent Kubernetes Deployment.

Pool Configuration Reference

Field	Type	Default	Description
`enabled`	bool	—	Enable or disable this pool
`replicaCount`	int	`1`	Static replica count (ignored when autoscaling is enabled)
`workerSubscribe`	string	`"all_but_kb"`	Queue type subscription
`workerSubscribeTenants`	string	`""`	Comma-separated tenant IDs (empty = all)
`workerSubscribeWorkspaces`	string	`""`	Comma-separated workspace IDs (empty = all)
`envSecretRef`	string	`""`	Name of an additional K8s Secret to layer on top of the shared app-env
`resources`	object	—	CPU/memory requests and limits
`autoscaling`	object	—	HPA or KEDA autoscaling config
`podDisruptionBudget`	object	—	PDB settings
`affinity`	object	`{}`	Pod affinity/anti-affinity rules
`nodeSelector`	object	`{}`	Node selector constraints
`tolerations`	list	`[]`	Node tolerations
`topologySpreadConstraints`	list	`[]`	Topology spread rules

Basic Multi-Pool Example

worker:
  enabled: true
  pools:
    default:
      enabled: true
      workerSubscribe: "all_but_kb"
      resources:
        requests:
          cpu: "2"
          memory: "12Gi"
        limits:
          cpu: "2"
          memory: "12Gi"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

    kb:
      enabled: true
      workerSubscribe: "kb"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75

    chat:
      enabled: true
      workerSubscribe: "chat"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 0
        maxReplicas: 5
        keda:
          query: >-
            SELECT COUNT(*) FROM runs
            WHERE status IN ('Queued')
            AND queue_type = 'chat';
          targetQueryValue: "5"

Tenant Isolation

Use workerSubscribeTenants and workerSubscribeWorkspaces to dedicate worker pools to specific tenants or workspaces. This is useful for:

Noisy-neighbor isolation — prevent one tenant’s heavy workloads from starving others
SLA tiers — dedicated capacity for premium tenants
Data residency — pin certain tenants to workers in specific regions or nodes

worker:
  pools:
    # Shared pool for all tenants (catch-all)
    default:
      enabled: true
      workerSubscribe: "all"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

    # Dedicated pool for a high-volume tenant
    tenant-acme:
      enabled: true
      workerSubscribe: "all"
      workerSubscribeTenants: "tid_acme_corp_123"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 8

    # Dedicated KB processing for specific workspaces
    kb-enterprise:
      enabled: true
      workerSubscribe: "kb"
      workerSubscribeWorkspaces: "ws_ent_001,ws_ent_002,ws_ent_003"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75

Per-Pool Secrets

By default, all worker pools share the same Kubernetes Secret ({release}-app-env). When different pools need different environment variables — such as separate LLM API keys per tenant, different Redis databases, or pool-specific feature flags — use envSecretRef to layer an additional Secret on top.

worker:
  pools:
    default:
      enabled: true
      workerSubscribe: "all"
      # Uses only the shared app-env secret

    tenant-acme:
      enabled: true
      workerSubscribe: "all"
      workerSubscribeTenants: "tid_acme_corp_123"
      # Env vars in this secret override the shared app-env
      envSecretRef: "acme-worker-env"

Create the per-pool secret separately (or via External Secrets Operator):

apiVersion: v1
kind: Secret
metadata:
  name: acme-worker-env
  namespace: spotflow
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-acme-dedicated-key"
  ANTHROPIC_API_KEY: "sk-ant-acme-key"

The pool-specific secret is mounted after the shared one in envFrom, so its values take precedence for any overlapping keys.

Multi-Namespace Deployment

To run worker groups in different namespaces (e.g., for resource quotas or network policy isolation), deploy separate Helm releases that share the same backend infrastructure.

Deploy the primary release

The primary release deploys backend, frontend, and the default worker pool.

# values-primary.yaml
backend:
  enabled: true
frontend:
  enabled: true

worker:
  enabled: true
  pools:
    default:
      enabled: true
      workerSubscribe: "all_but_kb"
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10

helm upgrade --install noxus ./cdk/helm/noxus-platform \
  --namespace spotflow --create-namespace \
  -f values.yaml -f values-primary.yaml

Deploy worker-only releases

For each additional namespace, disable all non-worker components and provide the same database/Redis credentials.

# values-kb-workers.yaml
backend:
  enabled: false
frontend:
  enabled: false
relay:
  enabled: false

# Same credentials as the primary release
env:
  DATABASE: "spot"
  REDIS_PORT: "6379"
secrets:
  DATABASE_URL: "postgresql://user:pass@db-host:5432/spot"
  REDIS_URL: "redis-host"
  REDIS_PASSWORD: "redis-pass"

worker:
  enabled: true
  pools:
    kb:
      enabled: true
      workerSubscribe: "kb"
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"
      autoscaling:
        enabled: true
        type: "hpa"
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilizationPercentage: 75
      nodeSelector:
        workload-type: kb

helm upgrade --install noxus-kb ./cdk/helm/noxus-platform \
  --namespace spotflow-kb --create-namespace \
  -f values.yaml -f values-kb-workers.yaml

All worker releases must connect to the same PostgreSQL and Redis instances. Redis coordinates task distribution — workers in any namespace pick up tasks from their subscribed queues regardless of where they run.

Cross-Namespace Considerations

Secrets: Each namespace gets its own K8s Secret. Use External Secrets Operator or a shared values file to keep credentials in sync.
Service Account: Worker-only releases still need a ServiceAccount with IRSA annotations for S3 access.
KEDA: ScaledObjects are namespace-scoped. Each release creates its own KEDA resources; the cluster-wide KEDA operator discovers them automatically.
Network Policies: Ensure worker namespaces can reach PostgreSQL, Redis, external APIs, and storage endpoints.

Autoscaling

KEDA (Queue-Based)
HPA (Resource-Based)

Best for scaling based on actual queue depth. Supports scale-to-zero.KEDA polls PostgreSQL to count queued/running tasks and adjusts replicas to maintain a target ratio.

worker:
  keda:
    postgresConnectionString: "postgresql://user:pass@host:5432/spot"
    pollingInterval: 15
    query: >-
      SELECT COALESCE(COUNT(*), 0) FROM runs
      WHERE status IN ('Queued', 'Running')
      AND created_at > NOW() - '1 hour'::interval;
    targetQueryValue: "10"
    cronTrigger:
      enabled: true
      timezone: "UTC"
      start: "0 9 * * *"
      end: "0 20 * * *"
      desiredReplicas: 1

  pools:
    default:
      autoscaling:
        enabled: true
        type: "keda"
        minReplicas: 1
        maxReplicas: 10
        behavior:
          scaleDown:
            stabilizationWindowSeconds: 300
            policies:
              - type: Percent
                value: 10
                periodSeconds: 60

Individual pools can override the global KEDA query and target:

autoscaling:
  type: "keda"
  keda:
    query: "SELECT COUNT(*) FROM runs WHERE status = 'Queued' AND queue_type = 'chat';"
    targetQueryValue: "5"
    pollingInterval: 10

Best for simple CPU/memory-based scaling. Cannot scale to zero.

autoscaling:
  enabled: true
  type: "hpa"
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 75
  targetMemoryUtilizationPercentage: 85

Run Archiving

After each workflow run completes, the worker archives run data (state, progress, node IO, logs, content streams) from Redis to object storage (S3/GCS/Azure Blob). This ensures long-term persistence but adds per-run overhead from token acquisition and upload latency.

Archive Mode (`ARCHIVE_MODE`)

Mode	Behavior	Use case
`sync`	Archives immediately after each run completes (default)	Production — guarantees data is persisted before the worker moves on
`async`	Skips per-run archiving; a periodic Celery task sweeps Redis and batch-archives completed runs	High-throughput deployments where per-run archiving creates bottlenecks
`disabled`	No archiving — data stays in Redis until TTL expires or the orphan cleanup task runs	Benchmarking and stress testing

Configuration

Variable	Default	Description
`ARCHIVE_MODE`	`sync`	Archive strategy: `sync`, `async`, or `disabled`
`ARCHIVE_BATCH_INTERVAL`	`60`	Seconds between batch archive sweeps (only used in `async` mode)

Async Mode

In async mode, a periodic Celery task (batch_archive_runs) runs every ARCHIVE_BATCH_INTERVAL seconds and archives all completed runs that have been idle in Redis for at least that duration. This eliminates per-run cloud storage token acquisition and reduces GCP/AWS auth pressure under high concurrency.

# Helm values example
env:
  ARCHIVE_MODE: "async"
  ARCHIVE_BATCH_INTERVAL: "30"

Disabled Mode

Use disabled for pure performance benchmarking. Run data remains in Redis (subject to TTL) and can still be retrieved by the backend. The existing archive_orphaned_data periodic task (runs every 30 minutes) acts as a safety net and will eventually archive idle data regardless of mode.

# Helm values for stress testing
env:
  ARCHIVE_MODE: "disabled"

In disabled mode, run data that exceeds its Redis TTL (4 hours for node IO, 60 days for state/progress) will be lost if not archived by the orphan cleanup task. Do not use this mode in production.

Verification

After deploying, verify the setup:

# Worker deployments across namespaces
kubectl get deployments -l app=worker --all-namespaces

# Pool labels on pods
kubectl get pods -l app=worker --all-namespaces --show-labels

# KEDA ScaledObjects
kubectl get scaledobjects --all-namespaces

# HPAs
kubectl get hpa --all-namespaces -l app=worker

# PodDisruptionBudgets
kubectl get pdb --all-namespaces -l app=worker

# Verify queue subscription in worker logs
kubectl logs -l app=worker --tail=50 | grep "WORKER_SUBSCRIBE"

Environment

Platform environment variables and configuration layers

Secrets

Secret management, per-pool injection, and rotation

Scaling

General scaling strategies and capacity planning

Kubernetes

Kubernetes deployment guide

Overview

Deployment Options

Networking

Configuration

Security

Operations

Task Routing

Queue Types (`WORKER_SUBSCRIBE`)

Tenant & Workspace Filtering

Worker Pools

Pool Configuration Reference

Basic Multi-Pool Example

Tenant Isolation

Per-Pool Secrets

Multi-Namespace Deployment

Cross-Namespace Considerations

Autoscaling

Run Archiving

Archive Mode (`ARCHIVE_MODE`)

Configuration

Async Mode

Disabled Mode

Verification

Environment

Secrets

Scaling

Kubernetes

​Task Routing

​Queue Types (WORKER_SUBSCRIBE)

​Tenant & Workspace Filtering

​Worker Pools

​Pool Configuration Reference

​Basic Multi-Pool Example

​Tenant Isolation

​Per-Pool Secrets

​Multi-Namespace Deployment

​Cross-Namespace Considerations

​Autoscaling

​Run Archiving

​Archive Mode (ARCHIVE_MODE)

​Configuration

​Async Mode

​Disabled Mode

​Verification

Environment

Secrets

Scaling

Kubernetes

Task Routing

Queue Types (`WORKER_SUBSCRIBE`)

Tenant & Workspace Filtering

Worker Pools

Pool Configuration Reference

Basic Multi-Pool Example

Tenant Isolation

Per-Pool Secrets

Multi-Namespace Deployment

Cross-Namespace Considerations

Autoscaling

Run Archiving

Archive Mode (`ARCHIVE_MODE`)

Configuration

Async Mode

Disabled Mode

Verification