Each test starts by listing the ways it could fail and turns each into a case; for new features the implementation is deliberately broken to confirm the test catches it. The database is a real PostgreSQL.
| Test | Failure modes | Result | Time (s) |
|---|---|---|---|
AI settings: switches, providers, tiers, capsconsoleapi.TestAISettingsHTTP | 8 | Pass | 0.6 |
Gateway: never calls a model when switched offaigw.TestGatesNeverCallModel | 4 | Pass | 0.5 |
Gateway: only routes to usable providersaigw.TestRoutesOnlyUsableProviders | 4 | Pass | 0.5 |
Gateway: hosted model access and monthly capaigw.TestHostedGate | 4 | Pass | 0.4 |
Gateway: failover and circuit breakeraigw.TestFailoverAndBreaker | 4 | Pass | 0.6 |
Gateway: accounting and daily/monthly capsaigw.TestAccountingAndCaps | 3 | Pass | 0.6 |
Gateway: redaction of sensitive dataaigw.TestRedaction | 3 | Pass | 0.5 |
Gateway: key encryption and deletionaigw.TestKeysAndDelete | 3 | Pass | 0.5 |
Gateway: both protocols and tool callsaigw.TestProtocolsToolCalls | 3 | Pass | 0.4 |
Ask about devicesconsoleapi.TestAIQueryHTTP | 6 | Pass | 0.6 |
Explain anythingconsoleapi.TestAIExplainHTTP | 8 | Pass | 0.6 |
Explain: known-issue matchingaiexplain.TestContainsKey | 3 | Pass | 0.0 |
One-click diagnosisconsoleapi.TestAIDiagnoseHTTP | 4 | Pass | 0.6 |
Policy from one sentenceconsoleapi.TestPolicyAIDraftHTTP | 7 | Pass | 0.5 |
Policy draft: only real restriction itemsaidraft.TestSwitchesAreRealRestrictions | 9 | Pass | 0.0 |
Policy draft: conversion and removalsaidraft.TestConvert | 9 | Pass | 0.0 |
Risk review: rule checksconsoleapi.TestPolicyRiskCheckHTTP | 7 | Pass | 0.5 |
Risk review: AI second opinionconsoleapi.TestPolicyRiskReviewHTTP | 6 | Pass | 0.5 |
Weekly report and AI summaryconsoleapi.TestWeeklyReportHTTP | 8 | Pass | 0.5 |
Class assistantconsoleapi.TestClassAIDraftHTTP | 5 | Pass | 0.5 |
Class assistant: plan checksaiclass.TestCheck | 7 | Pass | 0.0 |
Class assistant: call and data sentaiclass.TestAsk | 7 | Pass | 0.0 |
App checkconsoleapi.TestAppCheckE2E | 5 | Pass | 0.6 |
App check: risk rulesappcheck.TestEvaluate | 9 | Pass | 0.0 |
App check: reading the installerappcheck.TestScanZip | 9 | Pass | 0.0 |
App check: real installersconsoleapi.TestAppCheckRealAPKs | — | Skipped (needs real installers on the test machine) | 0.0 |
MCP serverconsoleapi.TestMCPHTTP | 8 | Pass | 0.6 |
Follow the latest modelconsoleapi.TestAIFollowHTTP | 10 | Pass | 0.9 |
Follow latest: product lines and candidatesaifollow.TestCandidate | 6 | Pass | 0.0 |
Follow latest: switch and revert rulesaifollow.TestVerdictAndDegraded | 6 | Pass | 0.0 |
Evaluation set: complete, unique, scoredaieval.TestBuiltinSet | 4 | Pass | 0.0 |
Evaluation set: unrun questions count as failedaieval.TestSummarizeCountsUnrun | 4 | Pass | 0.0 |
Questions come from real incidents and design limits. Scoring checks only verifiable properties: key points covered, no forbidden wording, all three headings present, drafted plans and policy items as expected. A new model must pass before it goes live.
| Model | Passed | Average time (s) | Failed questions |
|---|---|---|---|
| gpt-6.1-sol | 32 / 32 | 4.4 | — |
| gpt-6-astra | 32 / 32 | 4.9 | — |
| gpt-6-luna | 29 / 32 | 2.2 | alert-battery, appcheck-ads-zh, draft/wifi-trap-zh |
A model that does not pass everything is not assigned to the features that need it; which model each tier actually uses is set in the school's AI settings (the fast-tier gpt-6-luna is currently used only for brand translation).
| Script | What was verified |
|---|---|
tools/walk/ai_settings.mjs | Add a provider, fetch models, test, set tiers, turn on; the key shows as *** in the audit log |
tools/walk/policy_ai_draft.mjs | Draft a policy from one sentence; an app not installed and “turn off Wi-Fi” are removed with a note; saved after the risk check |
tools/walk/app_check.mjs | App check on a real installer; findings show component names; AI advice in three parts |
tools/walk/class_ai.mjs | Draft a lesson from one sentence: survey app pinned, 20 minutes, camera; what cannot be done is stated; no class started |
tools/walk/mcp.mjs | The token page gives the connection command; a factory reset from an external AI only creates a pending request the requester cannot confirm |
tools/walk/ai_follow.mjs | After real exams the standard and reasoning tiers switched to newer versions; Revert restored them at once |