TG-Sentinel:G1 Benchmarks:GPT-6.1 Sol vs Claude Opus 5.5 vs OSS Guardrail Models

When we introduced TG-Sentinel:G1, we shared our internal pre-production validation across our 4,050-evaluation holdout suite (1,150 Fast + 1,450 Max + 1,450 Ultra tier evaluations over 2,600 unique source rows — the same 1,450 adversarial multi-turn cases are scored twice, once under the Max contract and once under the Ultra contract). Following publication, enterprise AI security architects and independent industry reviewers asked a direct, foundational question:

On the exact same 4,050-case holdout dataset, how do commercial frontier models, standard open-source guardrails, off-the-shelf dense models, and standalone fine-tuned dense checkpoints perform compared to TG-Sentinel:G1?

We did not position TG-Sentinel:G1 as our flagship governance foundation model on the back of simple toy tests or surface-level toxicity prompts. Our holdout_v5 test suite — a Theseus Guard-authored controlled battery built from synthetic scenario templates, with zero reported exact and 6-gram overlap against our training and development sets — consists of genuinely unforgiving, production-grade edge cases—combining contradictory multi-document corporate policies (TG1_conflict, TG4_conflict), benign security incident reports (RPT) that look like attacks on the surface, encoded telemetry (safe_enc), multi-turn privilege escalations (ADM), and irreversible tool-execution commits (COMMIT). To prove just how demanding this benchmark is, we tested today's leading commercial frontier models alongside open-source guardrails and dense baselines on the exact same 4,050 tier evaluations (contract_hash: a2608b20ba891047f94d28d465428442079f20a8a97a79b94f43d6fdc52d00fe). Every prompt, raw model completion, wire-level API header, latency percentile, and SHA-256 evidence digest has been archived in our benchmark verification repository—and you can inspect the complete results below.

Evaluation Scope & Contender Matrix

To eliminate any architectural blind spots or custom-schema bias, we evaluated four distinct classes of baseline models against TG-Sentinel:G1 under deterministic decoding (temperature = 0.0):

1. Unified 7+1 BTX MoE Foundation Model (`TG-Sentinel:G1`): Our 46.24B total / 19.06B active parameter Mixture-of-Experts model (TGMoeForCausalLM), operating from a single consolidated checkpoint across Fast Mode (3-field contract), Max Mode (9-field evidence-grounded contract), and Ultra Mode (11-field forensic arbitration contract).
2. Commercial Frontier LLMs (`GPT-6.1 Sol` & `Claude Opus 5.5`): Evaluated via AWS Bedrock Global (global.openai.gpt-6.1-sol and global.anthropic.claude-opus-5-5) using the ConverseStream API with the full TG governance system prompt, constitutional policy hierarchy, and native jsonSchema structured output enforcement.
3. Internal Standalone Dense Ablation (`TG-Sentinel:Fast 4B`, `Max 9B`, `Ultra 27B`): Our earlier standalone fine-tuned dense checkpoints (4B, 9B, and 27B) trained on individual tier contracts. (Important Note: `TG-Sentinel:G1` was developed completely independently—none of the `TG-Sentinel:Fast/Max/Ultra` checkpoints, their training datasets, or the `Qwen3.5` model family were used in building `TG-Sentinel:G1`).
4. Equal-Budget Prompted Base Dense Model (`unsloth/Qwen3.5-27B`): The off-the-shelf 27B dense base model evaluated with our full governance prompt and xgrammar JSON schema enforcement, isolating the impact of domain-specific training versus raw parameter count.
5. Open-Source Guardrail Baselines (`IBM Granite Guardian 4.1 8B` & `Meta Llama Guard 4 12B`): Evaluated in both our structured JSON contract mode (unconstrained and xgrammar-constrained) and their own native guardrail formats (<score>yes/no</score> BYOC and safe/unsafe S1–S14) so that neither model is unfairly penalized by JSON formatting requirements.

Methodology Fairness: What If We Ignore the TG JSON Schema Completely?

A critical question in any benchmark is whether external models are penalized simply for failing to adhere to a proprietary JSON schema or citation syntax. To ensure complete fairness, we scored every baseline across three progressively relaxed evaluation layers:

1. Layer A — Strict Contract Decision Accuracy: Requires valid JSON schema adherence, valid policy_version, exact evidence_citations (trusted_ids), and exact (status, category) match.
2. Layer B — Schema- & Citation-Relaxed Semantic Decision (`Semantic Decision %`): Ignores citation formatting (source_id / clause_id string mismatches) and policy_version errors, scoring purely whether the model identified the right (status, TG1–TG5 category) pair. For example, GPT-6.1 Sol jumps from 14.69% (Strict) to 75.38% (Semantic) on Max Mode once citation formatting is forgiven, and Qwen3.5-27B Base jumps from 9.10% (Strict) to 64.21% (Semantic).
3. Layer C — Pure Status / Native Guardrail Format (`Status Accuracy %` — Zero Schema or Taxonomy Required): Even if we strip away both the JSON schema and the TG1–TG5 risk categories—evaluating Llama Guard 4 12B in its native safe / unsafe S1–S14 text format, Granite Guardian 4.1 8B in its native <score>yes/no</score> BYOC format, and Claude Opus 5.5 by counting upstream content_filtered refusals as blocks—the structural gap remains decisive.

Note on Frontier Instruction Following: While it is completely fair to excuse compact open-source guardrails (Llama Guard 4 12B, Granite Guardian 4.1 8B) for struggling with custom enterprise JSON contracts—since they were fine-tuned on rigid text templates—we should expect commercial frontier models like GPT-6.1 Sol and Claude Opus 5.5 to follow explicit schema, citation, and classification instructions. In our evaluation, both frontier models were given the full constitutional system prompt, explicit source_id / clause_id identifier lists, and native API-level jsonSchema structured-output enforcement. Despite this, GPT-6.1 Sol failed to follow the citation identifier rules on 85.3% of Max cases (14.69% strict vs. 75.38% semantic) and broke schema validity on 12.83% of Ultra cases, while Claude Opus 5.5 abandoned the JSON schema entirely on 31.74%–34.48% of evaluation cases due to upstream safety refusals:

Safety vs. Usability: The False-Alarm & Leak Pareto Frontier

A governance model cannot succeed by simply blocking everything, nor can it succeed by rubber-stamping ambiguous requests as safe. In production enterprise environments, two failure modes determine operational viability:

• Unsafe / Abstain Leak Rate (False Negatives): The percentage of policy-violating or insufficient-evidence (abstain) requests that the model erroneously marks as safe, allowing harmful or non-compliant actions to execute.
• Safe Over-Block Rate (False Positives): The percentage of benign enterprise prompts (including legitimate security reporting RPT, quoted policy text safe_quote, encoded telemetry safe_enc, and multi-turn administrative workflows safe_mt) that the model erroneously blocks or flags for abstention.

Real-Time Latency & Throughput Telemetry

For a governance engine sitting synchronously in the critical path of user-facing and agentic traffic, Time-To-First-Token (TTFT) and sustained token generation throughput are just as critical as decision accuracy. Calling a commercial frontier API (GPT-6.1 Sol or Claude Opus 5.5) adds 3.0 to 5.8 seconds of median TTFT per turn—rendering real-time multi-step tool governance impossible.

Complete Tier-by-Tier Benchmark Tables (Ranked Highest to Lowest Quality)

In all tables below, models are ranked from highest governance quality to lowest by Strict Contract Decision Accuracy (with Pure Semantic Decision Accuracy as the secondary sort key where citation formatting differs).

Methodology note on latency/throughput comparisons: GPU throughput figures in the tables below are aggregate batch output throughput (total generated tokens ÷ batch wall time) on local vLLM deployments, while Bedrock API figures are per-request token rates derived from server-side latency; TTFT measurement paths also differ (local decode proxy vs. network-inclusive API stream).

1. Fast Tier Evaluation (n = 1,150 Cases: 400 Safe, 600 Unsafe, 150 Abstain — 3-Field Contract)

Note: * Some requests to Claude Opus 5.5 were filtered by its internal safeguards or AWS Bedrock guardrails, producing refusals instead of benchmark decisions. The model's underlying judgment on those requests is unknown; it may have recommended the correct action, or it may not have. The holdout includes complex, high-risk attack scenarios that can appear benign, but the refusals are still observable production behavior, not missing cases to omit. We retain them because a production deployment using Opus 5.5 through the evaluated API setup would return the same refusals for these requests. We do not remap refusals to human_review or infer hypothetical decisions, since doing so could distort results and inflate performance on genuine human-review cases. This treatment is not intended to unfairly disadvantage Claude Opus 5.5 or any other model. We strive to report observed behavior and benchmark results fairly and accurately.

2. Max Tier Evaluation (n = 1,450 Cases: 500 Safe, 800 Unsafe, 150 Abstain — 9-Field Evidence-Grounded Contract)

In Max Mode, models must not only classify status and category (TG1–TG5), but also ground their decision in verifiable evidence_citations (trusted_ids) from the multi-document policy context. Notice how prompted generalist models (GPT-6.1 Sol and Qwen3.5-27B Base) suffer severe strict-contract drops (14.69% and 9.10%) because they hallucinate or misformat source_id / clause_id strings instead of emitting exact policy document identifiers:

Note: * Some requests to Claude Opus 5.5 were filtered by its internal safeguards or AWS Bedrock guardrails, producing refusals instead of benchmark decisions. The model's underlying judgment on those requests is unknown; it may have recommended the correct action, or it may not have. The holdout includes complex, high-risk attack scenarios that can appear benign, but the refusals are still observable production behavior, not missing cases to omit. We retain them because a production deployment using Opus 5.5 through the evaluated API setup would return the same refusals for these requests. We do not remap refusals to human_review or infer hypothetical decisions, since doing so could distort results and inflate performance on genuine human-review cases. This treatment is not intended to unfairly disadvantage Claude Opus 5.5 or any other model. We strive to report observed behavior and benchmark results fairly and accurately.

3. Ultra Tier Evaluation (n = 1,450 Cases: 500 Safe, 800 Unsafe, 150 Abstain — 11-Field Deep Forensic Contract)

In Ultra Mode, models adjudicate 11-field forensic contracts that include structured (source_id, clause_id) evidence pairs, human_review_required flags, and counterfactual_remediation payloads across contradictory policies (TG1_conflict, TG4_conflict) and multi-turn trajectories:

Note: * Some requests to Claude Opus 5.5 were filtered by its internal safeguards or AWS Bedrock guardrails, producing refusals instead of benchmark decisions. The model's underlying judgment on those requests is unknown; it may have recommended the correct action, or it may not have. The holdout includes complex, high-risk attack scenarios that can appear benign, but the refusals are still observable production behavior, not missing cases to omit. We retain them because a production deployment using Opus 5.5 through the evaluated API setup would return the same refusals for these requests. We do not remap refusals to human_review or infer hypothetical decisions, since doing so could distort results and inflate performance on genuine human-review cases. This treatment is not intended to unfairly disadvantage Claude Opus 5.5 or any other model. We strive to report observed behavior and benchmark results fairly and accurately.

Deep-Dive: Per-Class Failure Mode Breakdown (Ultra 1,450-Case Holdout)

Where exactly do frontier models, standalone dense models, and open-source guardrails fail? Breaking down the 1,450 Ultra holdout cases across all 15 adversarial and benign scenario classes reveals the structural differences between a specialized 7+1 MoE governance foundation model and every baseline:

Note: * Some requests to Claude Opus 5.5 were filtered by its internal safeguards or AWS Bedrock guardrails, producing refusals instead of benchmark decisions. The model's underlying judgment on those requests is unknown; it may have recommended the correct action, or it may not have. The holdout includes complex, high-risk attack scenarios that can appear benign, but the refusals are still observable production behavior, not missing cases to omit. We retain them because a production deployment using Opus 5.5 through the evaluated API setup would return the same refusals for these requests. We do not remap refusals to human_review or infer hypothetical decisions, since doing so could distort results and inflate performance on genuine human-review cases. This treatment is not intended to unfairly disadvantage Claude Opus 5.5 or any other model. We strive to report observed behavior and benchmark results fairly and accurately.

Five Core Architectural & Operational Conclusions

1. 7+1 MoE vs. Tier-Specific Standalone Dense Checkpoints (`4B / 9B / 27B` vs. `46B-A19B`): Across their respective Fast, Max, and Ultra evaluations (4,050 tier evaluations total), the three standalone fine-tuned dense checkpoints recorded 100.00% JSON schema validity in the reported setup (Fast: 99.65% without constrained decoding) and 0.00% Unsafe/Abstain Leak across the 2,650 unsafe and abstain cases. This zero-leak result came with Safe Over-Block rates of 38.75% (Fast), 30.00% (Max), and 23.80% (Ultra). The Ultra per-class results illustrate the trade-off: TG-Sentinel:Ultra scored 0.0% on benign RPT and 1.0% on unsafe COMMIT, compared with TG-Sentinel:G1's 99.0% and 100.0%, respectively; on TG4_conflict, both scored 80.0%. Overall in Ultra, G1 recorded 95.38% strict decision accuracy versus 67.59% for the dense checkpoint and reduced safe over-blocking from 23.80% to 1.00%. G1's median TTFT was <60 ms, versus 160.45 ms for the 27B dense model—at least 2.7× lower latency.
2. TG-Sentinel:G1 vs. Off-the-Shelf Prompted Dense (`Qwen3.5-27B Base`): Even with xgrammar constrained decoding and the complete TG governance prompt, the raw Qwen3.5-27B base model reaches only 70.17% on Fast, 9.10% strict / 64.21% semantic on Max, and 61.31% strict / 63.93% semantic on Ultra, while over-blocking 23.40%–28.25% of benign requests and incurring several seconds of end-to-end latency per request.
3. Why Commercial Frontier APIs Fail as Real-Time Gateways (`GPT-6.1 Sol` & `Claude Opus 5.5`): GPT-6.1 Sol is the strongest external baseline in pure semantic accuracy (79.57% Fast, 75.38% Max, 73.03% Ultra), yet it over-blocks 24.25%–27.40% of safe enterprise traffic (scoring 1.0% strict / 15.0% semantic on benign RPT reporting queries), leaks 3.26%–6.93% of unsafe/abstain cases, and incurs 3,078–4,825 ms median TTFT (50×–80× slower than G1). Meanwhile, Claude Opus 5.5 demonstrates why enterprises cannot build a security plane on top of a closed commercial API: AWS Bedrock's upstream content filter intercepts 31.74%–34.48% of adversarial evaluation cases (finish_reason = content_filtered), returning 0.0% on TG4, TG4_mixed, and TG4_conflict due to non-JSON upstream refusals.
4. Why First-Generation Open-Source Guardrails Collapse on Enterprise Governance (`Granite Guardian 4.1 8B` & `Llama Guard 4 12B`): Without external grammar enforcement, Llama Guard 4 12B (0.00%) and Granite Guardian 4.1 8B (2.09%) cannot emit valid structured governance JSON. Even when evaluated generously in their own native guardrail formats on simple binary/3-way status alone, Llama Guard 4 12B leaks 24.63%–26.93% of unsafe/abstain violations as safe (including a 35.0%–44.0% leak on ADM administrative privilege escalation — in binary native format, 65 of 100 multi-turn ADM cases and 56 of 100 Fast-tier ADM cases are blocked as unsafe while the remainder leak — and a 96.0%–100.0% leak on the ambiguous abstain class, which binary guardrails cannot represent at all), while Granite Guardian 4.1 8B over-blocks 14.00%–15.25% of benign traffic in native mode and collapses to 2.28% strict / 10.14% semantic accuracy on Ultra.
5. Standalone Model Inference vs. Full Theseus Guard Platform Orchestration (`Fast`, `Max` & `Ultra`): Every metric in this benchmark measures TG-Sentinel:G1 as a standalone raw foundation model—without any surrounding platform guardrails. Even in raw model inference, a large portion of the reported Unsafe/Abstain Leak figures (1.20% Fast, 1.68% Max, 2.00% Ultra) comes from ambiguous abstain cases (9/150 in Fast, 6/150 in Max, 9/150 in Ultra) rather than actual unsafe attacks—meaning raw True Unsafe Leak is `0.00%` (`0/600`) in Fast Mode and just `1.25%` (`10/800`) in Max and Ultra Modes. When deployed inside the Theseus Guard Enterprise AI Governance Platform, TG-Sentinel:G1 is further reinforced by L1 deterministic policy pre-flight checks, confidence-threshold & logprob-margin gating, automatic cross-tier escalation (Fast → Max → Ultra), and human_review_required routing whenever model confidence on a borderline safe decision falls below the tenant policy threshold—driving effective production leak rates even closer to zero across all three operational tiers.

Special Thanks & Acknowledgments

We extend our sincere thanks to Lambda AI for sponsoring the Lambda cluster compute resources used to run TG-Sentinel:G1 throughout these benchmark evaluations. Their support made this work possible.

We also thank Amazon Web Services (AWS) for its startup sponsorship, which enabled us to evaluate frontier models through AWS.