Skip to content

Sentinel Engine

6. Sentinel Engine (Three-Layer Evaluation)

The Sentinel Engine page provides real-time observability into the Three-Layer Evaluation architecture that processes every guarded AI interaction. The engine is built on the open-source ct-toolkit Core (Constitutional AI Toolkit) and powered by the TG-Sentinel Model Family (Fast, Max, and Ultra) fine-tuned for high-stakes adjudication.

6.1 Engine Architecture Overview

Theseus Guard's evaluation pipeline consists of three sequential layers:

Layer Name Function / Model Description
L1 Semantic Engine Divergence Detection Computes embedding-based divergence scores comparing prompts/responses against kernel commitments in milliseconds.
L2 Classification Engine Sentinel Fast (4B) A fine-tuned governance judge for nuance and context with sub-100ms latency.
L3 LLM Evaluator Sentinel Max (9B) / Sentinel Ultra (27B) Provides deep contextual evaluation and detailed reasoning for edge cases escalated from L2.

6.2 Engine Health Metrics

The Sentinel Engine dashboard surfaces real-time health telemetry for each layer:

L1 Engine Stats

  • Current throughput (requests/second)
  • Average L1 score across all active guards
  • P95 latency
  • Error rate

L2 Engine Stats

  • Verdict distribution (compliant / drift / rejected / pending)
  • Average confidence score
  • Classification model version in use
  • Probe accuracy per domain

L3 Engine Stats

  • Escalation rate (% of interactions reaching L3)
  • Average L3 evaluation time (ms)
  • L3 reason distribution (categories of escalation reasons)
  • Model provider powering L3 evaluations

6.3 Model Probe Health

The Sentinel Engine also shows the health status of all active statistical probes:

  • Probe Name — The probe's identifier
  • Domain — Which guard domain this probe monitors
  • Accuracy — Current probe classification accuracy (%)
  • Status — healthy, degraded, or failed
  • Last Evaluated — Timestamp of last probe accuracy check

If a probe's accuracy drops below its configured threshold, a Probe Accuracy Degradation alert is automatically created in the Alert Center and linked to the Sentinel Engine page.

6.4 Probe Degradation Response

When the Sentinel Engine detects a degraded probe, operators can:

  1. Review the alert in Prove → Alert Center
  2. Navigate directly to the Sentinel Engine from the alert detail view
  3. Initiate a probe retraining or model rollback from the engine controls