> ## Documentation Index
> Fetch the complete documentation index at: https://docs.noxus.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Platform ops events

> Enable and consume the platform-owned OTLP logs stream

Ops events give deployment operators a structured record of lifecycle events and
classified failures. They are disabled by default and are separate from tenant
Phoenix/Langfuse tracing. Prometheus remains the metrics backend; enabling ops
events does not create an OTel metrics pipeline.

## Enable the exporter

Set these values on backend, worker, and relay processes, then restart them:

```bash theme={null}
OPS_TELEMETRY_ENABLED=true
OPS_TELEMETRY_ENDPOINT=http://ops-collector:4318
OPS_TELEMETRY_EXPORT_INTERVAL_MS=5000
```

The endpoint is a base URL. The exporter appends `/v1/logs` and sends OTLP/HTTP
protobuf. For an authenticated collector, set `OPS_TELEMETRY_HEADERS` through the
existing secret configuration, for example `Authorization=Bearer <collector-token>`.
Do not put credentials in the endpoint URL. Use TLS for traffic outside the trusted
cluster network.

Both platform Helm charts accept these settings in `extraEnv`; put authentication
headers in `secrets`. With `SECRETS_SOURCE=FILE`, add the values to the encrypted
internal configuration instead. In an environment-backed Docker Compose deployment,
pass the same variables through each service's `environment` section.

```yaml theme={null}
extraEnv:
  OPS_TELEMETRY_ENABLED: "true"
  OPS_TELEMETRY_ENDPOINT: "http://ops-collector:4318"
  OPS_TELEMETRY_EXPORT_INTERVAL_MS: "5000"
```

There is no collector or storage backend enabled by default. Setting the flag
without a nonempty endpoint leaves the stream disabled. Configuration is loaded
once per process; changing it requires restarting all producers.

## Minimal collector for staging verification

Run an OpenTelemetry Collector reachable only inside the test namespace. A logs
pipeline with the debug exporter is sufficient to inspect synthetic test events:

```yaml theme={null}
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
```

Expose port 4318 through a ClusterIP Service named `ops-collector`, apply the Helm
settings above, and read the collector pod logs. For ongoing operation, replace
`debug` with an exporter to your logs store and configure retention and restricted
operator access. Event attributes include tenant, workspace, user, and resource
identifiers; they must not be exposed through a public receiver or dashboard.

## Event catalogue

Every event name supported by this implementation is listed below. All records use
instrumentation scope `noxus.ops`, severity `INFO`, a UTC event timestamp, and an
empty body. Resource metadata includes `service.name` (the deployment name),
`deployment.environment.name`, and OpenTelemetry SDK metadata. Optional attributes
are omitted when their source identity is unavailable.

### Billing lifecycle

These events fire when `BillingManager.register_event()` accepts the registration
attempt. They carry these attributes:

| Attribute                     | Presence       | Meaning                                            |
| ----------------------------- | -------------- | -------------------------------------------------- |
| `noxus.event.id`              | Always         | Stable billing event ID for consumer deduplication |
| `noxus.event.type`            | Always         | Event name without the `noxus.` prefix             |
| `noxus.tenant_id`             | Always         | Tenant ID                                          |
| `noxus.group_id`              | Always         | Workspace ID                                       |
| `deployment.environment.name` | Always         | Billing event environment                          |
| `noxus.user.id`               | When available | Actor ID                                           |
| `noxus.api_key.id`            | When available | Calling API-key ID, never its value                |

| Event                     | Registration                          |
| ------------------------- | ------------------------------------- |
| `noxus.workflow_created`  | Workflow registered                   |
| `noxus.assistant_created` | Agent registered                      |
| `noxus.kb_created`        | Knowledge base registered             |
| `noxus.kb_files_uploaded` | Knowledge-base file upload registered |
| `noxus.trigger_created`   | Trigger registered                    |
| `noxus.trigger_activated` | Trigger activation registered         |
| `noxus.preset_created`    | Preset creation registered            |
| `noxus.preset_updated`    | Preset update registered              |
| `noxus.preset_deleted`    | Preset deletion registered            |

### Case activity

A newly inserted `CaseActivity` row emits after its outer database transaction
commits. Every event carries `noxus.case.id` and `noxus.case.activity_type`; it also
carries `noxus.user.id` when the activity has an actor. Titles, comments, email
contents, field values, and activity payloads are excluded.

| Event                              | `noxus.case.activity_type` |
| ---------------------------------- | -------------------------- |
| `noxus.case.case_created`          | `case_created`             |
| `noxus.case.field_update`          | `field_update`             |
| `noxus.case.status_change`         | `status_change`            |
| `noxus.case.priority_change`       | `priority_change`          |
| `noxus.case.assigned`              | `assigned`                 |
| `noxus.case.comment_added`         | `comment_added`            |
| `noxus.case.email_received`        | `email_received`           |
| `noxus.case.email_sent`            | `email_sent`               |
| `noxus.case.node_execution`        | `node_execution`           |
| `noxus.case.lookup`                | `lookup`                   |
| `noxus.case.agent_action`          | `agent_action`             |
| `noxus.case.case_closed`           | `case_closed`              |
| `noxus.case.extraction_complete`   | `extraction_complete`      |
| `noxus.case.extraction_failed`     | `extraction_failed`        |
| `noxus.case.process_failed`        | `process_failed`           |
| `noxus.case.custom`                | `custom`                   |
| `noxus.case.case_moved`            | `case_moved`               |
| `noxus.case.task_created`          | `task_created`             |
| `noxus.case.task_completed`        | `task_completed`           |
| `noxus.case.task_failed`           | `task_failed`              |
| `noxus.case.automation_failed`     | `automation_failed`        |
| `noxus.case.task_prefilled`        | `task_prefilled`           |
| `noxus.case.task_assigned`         | `task_assigned`            |
| `noxus.case.task_priority_changed` | `task_priority_changed`    |

`task_failed` is a legacy accepted activity value. Current task writers use
`automation_failed`; old stored rows are not replayed into the ops stream.
The `custom` classification stays fixed and never becomes a tenant-defined event name.

### Agent insights

A newly inserted `InsightEventRow` emits after its outer database transaction
commits. Attributes are `noxus.tenant_id`, `noxus.group_id`, and
`noxus.insight.type`, plus `noxus.conversation.id` and `noxus.agent.id` when present.
Insight keys, labels, scores, analysis text, and payloads are excluded.

| Event                                  | `noxus.insight.type`     |
| -------------------------------------- | ------------------------ |
| `noxus.insight.conversation_sentiment` | `conversation_sentiment` |
| `noxus.insight.topic_detected`         | `topic_detected`         |
| `noxus.insight.sub_topic_detected`     | `sub_topic_detected`     |
| `noxus.insight.conversation_outcome`   | `conversation_outcome`   |
| `noxus.insight.custom_insight`         | `custom_insight`         |
| `noxus.insight.rating_driver`          | `rating_driver`          |
| `noxus.insight.noticed_insight`        | `noticed_insight`        |
| `noxus.insight.user_feedback`          | `user_feedback`          |
| `noxus.insight.csat_response`          | `csat_response`          |

### Classified failures and API keys

| Event                              | Trigger                                                  | Attributes                                                                                                                           |
| ---------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `noxus.credential.refresh_failed`  | Automatic, on-demand, or explicit UI OAuth refresh fails | `noxus.credential.id`, `noxus.integration.type`, `noxus.credential.permanent_failure`; `noxus.integration.provider_key` when present |
| `noxus.api_key.created`            | Workspace API-key creation commits                       | `noxus.api_key.id`, `noxus.tenant_id`; `noxus.user.id` when available                                                                |
| `noxus.api_key.revoked`            | Workspace API-key revocation commits                     | `noxus.api_key.id`, `noxus.tenant_id`; `noxus.user.id` when available                                                                |
| `noxus.table.sql_guard_denied`     | The instrumented table SQL guard rejects execution       | `noxus.table.denial_reason`                                                                                                          |
| `noxus.billing.constraint_blocked` | A billing constraint rejects an operation                | `noxus.billing.constraint_type`, `noxus.billing.limit_type`; `noxus.tenant_id` and `noxus.group_id` when available                   |

Credential permanence is a Boolean classification: `true` means reconnection is
required; `false` means the failure was not classified as permanent. Integration
and provider keys identify registered integration types, not customer credential
names. Provider error text and tokens are excluded.

The SQL denial reason is one of `parse_error`, `multi_statement`, `not_select`,
`not_write`, `schema_denied`, `table_denied`, `function_denied`, or `write_denied`.
SQL text and database error messages are excluded.

Billing limit type is `internal` or `license`. Constraint type uses the enum member
name: `AI_OPERATIONS`, `MESSAGES_SENT`, `KB_FILES_UPLOADED`, `TOKENS`, `WORKFLOWS`,
`TRIGGER_RUNS`, `NCS_SEARCH`, or `NCS_SCRAPE`. Usage amounts and limits are excluded.

### Coverage and delivery semantics

Run, node, model, and token usage remain in the existing Prometheus instruments and
billing accounting. Successful table queries produce operation/row/duration metrics;
only instrumented guard denials produce an ops event. Plugin installation failures,
sandbox OOM, and model-health suspension/recovery are follow-up coverage; this
implementation does not emit dedicated events for them.

These are best-effort diagnostics, not a durable audit or accounting log. A crash or
full exporter queue can lose events. Billing registration events describe the
registration attempt, not successful database persistence; `noxus.event.id` supports
consumer deduplication. Case/insight events are emitted only after the outer commit;
rolled-back savepoints and transactions do not emit. Existing database history is
not backfilled.

## Verify delivery

1. Enable the exporter and start the collector as described above.
2. Create a temporary API key through the workspace's API keys page.
3. Read the collector pod logs and find `noxus.api_key.created`. Confirm the record
   has a current timestamp, an empty body, and the expected key and tenant IDs.
   The secret key value must be absent.
4. Revoke the temporary key through the same page and confirm
   `noxus.api_key.revoked` arrives with the same key ID.
5. Allow for the configured export interval and the collector's batch delay before
   treating an event as missing.

## Troubleshooting

* **No events:** verify the flag and endpoint in every producer, restart after
  changes, and confirm the collector has a logs pipeline on port 4318.
* **Exporter errors:** check collector health, network policy, HTTP path, TLS, and
  authentication. A trace-only receiver is insufficient.
* **Missing run traces:** inspect the run span capture separately. It uses
  `SimpleSpanProcessor` and Redis/cold storage; the ops collector is unrelated.