Independent evaluation for benchmarks built on SEC filings.
AITrailblazer audits and builds evaluations for SI systems that read financial documents. Every grade is checked against the full filing, exhibits included, never against the excerpt a dataset happens to ship with.
Evaluation services are the eval practice of DeltaVerdict, AITrailblazer's grading practice.
Fixed-fee engagements, scoped in writing. Findings go to the benchmark owner privately first. (c) 2026 AITrailblazer
What we do
Two services, one standard: a claim about a filing stands only if the filing itself supports it.
Benchmark audits
We re-grade a sample of your benchmark under blind conditions and report what holds up. The delivery is a per-item flaw list under a fixed taxonomy (wrong gold answer, ambiguous question, evidence mismatch, contaminated item, stale source, difficulty mislabeled), a severity roll-up of clean, fixable, and invalid items, and the agreement statistics behind both. After you fix the items, we re-verify the corrections. The same audit runs over any benchmark or automated judge in document-heavy domains; SEC filings are our home ground, not the limit of it.
Eval construction
We author evaluation tasks from SEC filings: a question, a gold answer, the filing and page evidence behind it, a weighted scoring rubric, and a difficulty tier. Sets ship with public and private splits. Private holdouts stay private, with scheduled refresh so a set does not go stale or leak into training data.
Method
The protocol is the product. It runs the same way on our own record as it does on yours.
Primary sources only
Grades rest on the complete filing package, exhibits included. Dataset excerpts and snippets are locators. They are never treated as evidence.
Two blind passes
Two graders work independently under one written rubric. Neither sees the other pass, any prior labels, or the benchmark owner's gold calls.
Adjudication
Where the passes disagree, or either one abstains, a fresh adjudicator decides, seeing the competing calls anonymized. Items that stay unsettled ship as abstains, never as forced labels.
Agreement, reported
Every delivery reports raw agreement, Cohen's kappa, the adjudication rate, and abstain counts, with the sample's composition stated next to the numbers. A percentage without its set is not a result.
The record
We grade our own work in public, on the same standard we sell.
AITrailblazer runs DeltaSignal, a public SEC filing intelligence practice. Its calls are graded on a public scoreboard, wins and misses both, and its filing briefs ship through DeltaSignal Radar on this site. The evaluation practice grew out of grading our own output first and publishing the results either way.
The proof asset behind these offers is the DeltaSignal Decider, our blind two-pass SEC filing grading program, with its record stated in public.
- Re-run it yourself: our grading harness is open, a frozen-prompt runner that never reads labels and a scorer that accepts any labels file, at github.com/aitrailblazer/atlas7-filing-sentinel.
- The grading card: on October 26, 2026, two engines graded the same 179 frozen SEC filing packets against our locked key. Muse Spark 1.3 agreed on 87.7% of filings; Muse Spark Open on 85.5%. Precision, recall, false-confident rates, and the 21 filings both engines missed are printed on the card.
- The Empty Unison, our dated public grading of the market's summary feeds, at atlas7.app/empty-unison.
Independence
We take no money from anyone our public records grade, and our income never depends on a verdict coming out one way or the other. A benchmark owner cannot credibly audit its own benchmark. Independence is not our posture; it is the product.
How engagements run
Owner-private first, fixed fee, documented method.
Pilot first
An engagement can start as a pilot audit of 30 to 50 items, fixed fee, two to three weeks, with success criteria agreed in writing before work begins. The pilot fee is credited in full toward a standard engagement.
Private first
Findings go to the benchmark owner privately, with a response window, before anything is published. What we publish afterward is method and aggregate rates, with the sample size attached.
Fixed fee
A standard audit covers a 150 item sample at a fixed fee. Construction pilots are fixed fee as well. Remediation and re-verification are scoped separately, in writing, before work begins.
Confidential
Client benchmarks and data stay inside the engagement. Our method is documented and shared with the client. Client material is never published, reused, or trained on.
The practice is led by Constantine Vassilev, founder of AITrailblazer, with Rada Vassileva. Engagements start with a scoping conversation, not a purchase.