Every demo and engine is tested on a fixed set of synthetic examples. We keep each run with the examples it used, the model, the date and any examples that failed. The number we show comes from the most recent complete run with no failed examples, even when an earlier run looked better. Nobody outside this studio has checked these runs.
The figures on this page come from test runs on synthetic data. The runs that use a model ran on Google's Gemini (gemini-3.5-flash-lite), a hosted model. We'll add runs on a local model.
Ask us to open a run on the callMoves records between your systems without losing or duplicating any, even when one of them slows down or goes offline.
Reads documents into your system's fields and leaves a field empty rather than guess.
Drafts replies and decides which ones a person must see first.
Keeps a record of who agreed to what, and deletes everywhere when they withdraw.
Keeps the test results behind every number we show you.
17 September 2026
The run failed partway through and stored zeroed scores, so it measured nothing.
17 September 2026
It recorded a median time of 0 ms to route a ticket, which is impossible, so we don't trust its accuracy figures either.
1 October 2026
Its help articles were indexed with OpenAI's embedding model instead of the standard one every other run uses, so its routing can't be compared with them. It ran to completion with nothing sent that needed a person.
14 October 2026
It ran on an earlier version of the synthetic loan packs, before two flaws in the generated documents were fixed (agreement clauses that skipped from 8 to 14, and revised agreements dated after their pack arrived). Run 2037a658 re-ran the same checks on the corrected packs.
15 October 2026
Its placement windows used an interval setting that hadn't been recalibrated after a change to the synthetic history. Run 7cb34031 is the same evaluation with the setting recalibrated on the development worlds.
A run reaches this site only if it ran against a real model, finished, had zero errored examples and isn't on the set-aside list. Runs in offline mock mode never appear.
Any example that errors fails the whole run, and a failed run never reaches this site, whatever its scores.
640b1cd7-7fbe-432c-9bd8-3e4513f598570a8918f4ad88f8435d13fce8a0a69af00c890b205001bbc588f13e70b6e3a43blender-desk-v1No file version reaches a lender twice and no applicant is stored twice, across every scenario: duplicates = 0No portal holds an incomplete file, and nothing stopped before sending reached a lender: partial submissions = 0Every field a lender's reply doesn't state comes out empty (null correctness = 100%), and every value read is in the reply's own wordsEvery reply about a decline, or naming a reason lenders decline for, sends the update to Needs a person (recall = 100%), with no model-written draftIn every scenario, submissions requested = sent + held + stopped + in line, agreeing with the send engine's table and the portals, with nothing left in line or refused over a limitfb9039c6-e9ed-4c44-b701-9068dae37c1c5a4aa573a155de1c516106db263761b0018228764abc552e021449ab00e2deddlsp-locker-v1Every seeded gap is flagged: gap recall = 100% (the gold world's three and every gap scenario's)Every stored file changed after receipt is flagged by the fingerprint re-check: tamper detection = 100%Every field a partner document doesn't state comes out empty: null correctness = 100%, and no value is kept without the words it came fromEvery gap in a daily feed is flagged on its second day at the latestA file delivered twice is stored once and read once, and both arrivals are recordedNo day, month or quarter is counted as present without a stored file's evidence behind it7cb34031-c28b-4a10-9064-8a08f8defd0662bd37423b5bd69518fa689f1d0c0ebc785079cc45b1ebb926c73305091ee924bench-radar-notes-v1The 80% placement windows hold the actual placement day for between 70% and 90% of the hold-out estimatesEvery number in every note shown appears in that consultant's driver data: note grounding is 100%No daily cost or cost to date is shown for a consultant with no pay rate on file, on any Monday or after any controlA working day whose timesheets are missing has no value on any screenThe three Try to break it controls hold: gap not filled, wider and less certain windows, no cost without a rateThe 18-month history and the fitted model are the same when built again from the seed125ed971-4ddc-4be6-ade6-dc1ab11e97177854a23a8b2f280491a84c9c12d19bdd95a24e2a34af0abdd65f6533e048be76sop-navigator-v1Grounding = 100%: every quoted step is found word for word in the version it citesSuperseded-version errors = 0: no answer quotes a version that isn't the one in effect on the plant's dayInvented answers = 0: no question that no SOP covers, or that only QA decides, is answeredThe three Try to break it controls and a new version taking effect hold their rules (4 of 4)49d339ee-298c-4c5d-8fa2-ab55385fa9eb32c3a127bfb5f245f9dff9189afbcf5fa911a2fe7da94133341bac56646b9c10carrier-status-v1For every carrier, after the month and after every break-it scenario, received = applied + held for replay (repeats counted separately), and every event a carrier published was delivered or is still waiting at the carrierNo shipment past its staleness threshold (by the desk's events or by what the carriers had published) shows anything but Stale, on any day or in any scenarioNo customer update about a delay, damage, a claim or a customs hold lands in Draft ready; every one goes to Needs a person, with no model callEvery break-it scenario on the running desk ends as expected2037a658-9d6f-42df-8a70-072a9a2ea75dccc27a6f573256afaa0a280644b723f624f28d4fb0f3e5c5209c729408c3cf93loan-pack-v1Every seeded critical mismatch is found: critical-mismatch recall = 100%Zero "Clear" with an unreadable page: no pack with a missing or unreadable page is routed to ClearAPR recomputation = 100% on gold: the APR computed from each pack's schedule and fees equals the gold calculationEvery field a document doesn't state, or a page can't show, comes out empty (null correctness = 100%), and no value lacks the page it came fromEvery finding points at a page of the packNo APR is computed when a page it needs is missing or unreadablea3d56fe3-938a-49c3-a088-9352e8fe8226625867657eb31961724a823aac4ab7c749c7b9adc4c2676061e47527cf581d05ship-docs-v1Every field a document doesn't state, or states on a page nobody can read, comes out empty: null correctness is 100%, and no value lacks the words it came fromEvery seeded critical mismatch (weight, package count, consignee, HS code, invoice total) is found: discrepancy recall is 100%No shipment with a seeded critical mismatch is routed to Ready to fileNo shipment with a page nobody can read is routed to Ready to fileEvery second upload of the same invoice is recognised and not stored34d7b44e-42c7-44eb-88fe-b9f492306c8e47daf2a7fb81a0d9777f341930f3b3fe04300db7ad2532ad526ac4d2766c1d98questionnaire-v2No yes is drafted without a policy line that supports it: every evidence-lane yes is a gold yes and quotes a supporting section, and no yes stands without a citationWrite-back fidelity is 100%: every question read at its place, every approved answer in exactly its own cells, and nothing else in any file changedAcross six break-it scenarios: a certificate not held is never answered yes, every answer quoting an old policy version is reopened, and a question on three tabs is answered once4805b61e-a9cc-44d7-9fbe-785e550929e234e5b5106f6009e2cc59cd3b30b5ad5fad1c622a113c159613a27acd79e4004adenial-desk-v1Every field a denial document doesn't state, or a page that can't be read would hold, comes out empty: null correctness is 100%, and no value is kept without its words on a readable pageEvery factual line of every appeal drafted cites a real span in the chart or the denial letter, and says nothing the cited lines don't: appeal grounding is 100%Every medical-necessity, fraud, audit, overpayment-recovery and legal denial goes to a person with no draft and no drafting model call: forced-human recall is 100%Across the sending scenarios no appeal reaches a portal twice, every approved appeal is sent, held or in line, and nothing is sent without a person's approvale2d550a6-26d3-4f70-8342-bd43dcb39eb80171ef620f3804bf89e287da5e9d0f443512fa296de032208834c44c2d96ee03rtr-guard-v1Every field an RTR document doesn't state (or keeps confidential) comes out empty: null correctness is 100%, and no value is kept without its wordsZero green results where a gold duplicate or an invalid RTR existsPlanted duplicates flagged for a person: duplicate recall is at least 95%Each RTR email, delivered twice, is stored as one document and one ledger recordAcross the submission scenarios no submission reaches a vendor twice, and every send is sent, held or in line17f2e302-e60f-4efe-b889-67f6b3ea76fb17be3a7662c2c5904be7136917ab1fb5256c31370d4b2312f258c55b2a720e61coding-qa-chart-v1No chart with a seeded critical error (wrong laterality, unsupported visit level, missing signature) is routed to PassEvery fact a chart doesn't state, or a page can't show, comes out empty: null correctness is 100%, and no value lacks the words it came fromNo chart with a page that couldn't be read, or a review that didn't finish, is routed to PassEvery flag points at the chart text behind it, on a readable pageae39a57b-e930-4386-9166-821e91f8f2c53c6f2d5476cf3e48bb4e68065d6c6de03ab8d9fd1df9b5177cbcc21d0afd9732module-kit-v1Every field a statement doesn't state, or a phone photo blurs, comes back empty and marked for review: null correctness is 100%, with no invented holdingsNo client email that needs review or a person is put in the auto-send laneEvery number in every summary shown is in the input (checked against the gold figures), and every summary with a seeded wrong figure is blockedWith the model server stopped, every module answers 503 with Retry-After and nothing is decided8f0f3491-566a-4f4d-8fd6-f09a2bfa5682aaa717a1c86c0c16c37ae61aad5684d79de34ecdf4e89d3ebfcc63e7bf83ecbfbench-desk-v1Every field a requirement email or rate confirmation doesn't state (or a scan blurs) comes out empty: null correctness is 100%No reply that needs review or a person is put in Draft readyEvery reply about a rate, right to represent, contract or legal terms, a non-solicit clause or immigration goes to Needs a personAcross the break-it scenarios no consultant reaches the same end client twice, and every send is sent, held or in line2d019a49-47f0-405c-9f1a-65d18487ee0d4dc06b9ade89b8d21a738ffc13f8b6a7da7c3973c620f2d6b9253f5c25f49fe5guardian-v1With every system healthy, each withdrawal is deleted from all four systems and confirmed by eachNo deletion is reported as complete, and no certificate issued, while any system is still open; an open system closes only when it confirmsEvery seeded leftover record is found by the scan, with its system named, and the scan reports nothing that wasn't seededA student whose own number stands in as the guardian's can't get consent recorded, at any of the three layersEvery certificate checks against the institute's published key, and an edited copy fails603010d4-26b4-4de3-bd98-8871bce1484c0101e3192d7b3dc7edd0d7ae9d5b5e8be06c69eaa38a0546646104f5397e646dgatekeeper-v1Nothing that needed a person was sent without reviewf052d140-09fe-49bf-abac-efca1fa26bdf5e0bac03cfaf4fb6e6aea49229000e608d8b2a1f8f6205d51aeb99184a845355extractor-v1Every field the document doesn't state is left empty35363bac-79be-4d67-87a2-50c0faf634fec35512c99a86e5c7709fbab2b45920e44dd49e07d8639893a9d0c1a3b8af129fconsent-v1Every withdrawal is deleted from every connected system on clean systemsEvery planted leftover record is found by the orphan scanA system that stays down leaves the deletion incomplete, with that system named79fe8b11-a043-4ec1-89d5-4e8f40469ae95a59dd5c882de7dfd6f87c1bd84286d96e0f3cc86155bf8f7483c596eeb17599v1Every order received is either written or held for replayNo order is written to the billing system twiceTo re-run an evaluation from its repository, start Postgres and Redis with pnpm infra:up, and run the eval script for the engine or demo with a real model provider configured. Each run is stored with its dataset hash and prompt version.
pnpm --filter @stafflo-lab/reconciler eval pnpm --filter @stafflo-lab/extractor eval pnpm --filter @stafflo-lab/gatekeeper eval pnpm --filter @stafflo-lab/consent-spine eval pnpm --filter @stafflo-lab/bench-desk eval pnpm --filter @stafflo-lab/rtr-guard eval pnpm --filter @stafflo-lab/bench-radar eval pnpm --filter @stafflo-lab/coding-qa eval pnpm --filter @stafflo-lab/denial-desk eval pnpm --filter @stafflo-lab/module-kit eval pnpm --filter @stafflo-lab/questionnaire eval pnpm --filter @stafflo-lab/lender-desk eval pnpm --filter @stafflo-lab/loan-pack eval pnpm --filter @stafflo-lab/lsp-locker eval pnpm --filter @stafflo-lab/guardian-consent eval pnpm --filter @stafflo-lab/ship-docs eval pnpm --filter @stafflo-lab/carrier-status eval pnpm --filter @stafflo-lab/sop-navigator eval
On the call we'll open any run on this page: the examples it used, the model, the date and any examples that failed.
Ask us to open a run on the call