Working with your squad
Findings & scores
Plain pass/fail tells you something broke. Structured measurement tells you what, how bad, whether it's on your core loop — and whether the fix actually held. All of it is additive: a round is valid with none of it.
Statuses: the base layer
Every persona×scenario attempt gets one of four statuses:
| Status | Meaning |
|---|---|
pass | The persona's goal was actually accomplished — as judged from the transcript, not from the persona's mood. |
fail | The goal wasn't accomplished. Every fail carries a failure class (below). |
blocked | The attempt couldn't proceed at all — an environment outage, a login wall with no session prepared. |
| Inconsistency | The result may be transient or unreliable. Re-run it before treating it as a regression. |
Failure class: the most valuable signal in the ledger
Every failure is classified by what actually failed:
| Class | Means | Example |
|---|---|---|
app | Your product is genuinely broken or confusing for this persona. | A button does nothing; an error on submit; the needed feature isn't findable within the persona's patience. |
test | The scenario or persona definition is the problem, not the product. | The scenario references a feature you intentionally removed; the expectations are unmeetable as written. |
agent | The tester or harness erred — nothing to do with your product. | The actor misread a working login flow as broken; the harness didn't supply a needed session. |
Getting this right is why a round is worth anything. A wave of app failures is a real regression. A wave of agent failures means the harness — or the model playing your testers — needs fixing, not your product. Anything that smells like an authentication blocker gets agent-level suspicion first: did the harness actually supply what the scenario needed?
Findings: what specifically went wrong
A finding is one observed problem: which feature, a short title, and a severity — plus optionally what was observed, how it affected this persona, and a suspected cause. Severities:
| Severity | Reads as |
|---|---|
| S0 | Showstopper — the persona's task is dead in the water. |
| S1 | Serious — major impact on the task or trust. |
| S2 | Moderate — real friction, task still achievable. |
| S3 | Minor — polish, small confusions. |
Severity is set from observed impact — what the transcript shows actually happened to the task — never from how dramatically the persona complained. A theatrical persona doesn't inflate your severity counts.
Findings are capped at 25 per result to force prioritization. And the same underlying problem — same feature, same title — reported by different personas or across different rounds is grouped automatically, so five reports of the save bug read as one issue seen five times, not five separate mysteries.
Scores: five dimensions, never blended
Each attempt can carry five scores, each rated 1, 3, or 5:
- Discoverability — could they find the thing?
- Comprehension — did they understand what they were looking at?
- Task success — did they actually get it done?
- Effort — how hard did they have to work for it?
- Confidence / trust — would they trust the product with the real thing?
The five are never blended into one number. A product that scores high on task success and low on trust has a very different problem from one with the reverse — a single blended score is precisely what would hide that.
Coverage: what was actually exercised
Per feature touched in an attempt, coverage records a state:
| State | Meaning |
|---|---|
NE | Not exercised — the attempt never reached this feature. |
EN | Encountered — seen but not really used. |
AT | Attempted — actively used in the attempt. |
UN | Unexpected behavior observed. |
SU | Success — it worked for the persona. |
UN and SU are independent and both always recorded — a feature can work and surprise. Coverage is what separates “this feature is fine” from “this feature was never tested,” which look identical in a plain pass/fail world.
Priorities: worst first, computed for you
When a round completes, its findings roll up into a priority list: each finding scored by severity × how many distinct personas hit it × whether it landed on one of your project's declared core features (which doubles its weight), sorted worst-first, top ten. That list is the answer to “what do I look at first?” — reviews lead with it rather than re-deriving a ranking.
Declare your core features on the project (“the core loop is capture and review”). It costs one sentence and makes the priority math match what actually matters to your business.
Issues: the cross-round memory
A finding is per-attempt; an issue is the workspace's persistent record of that problem across rounds. Every finding automatically opens or updates its matching issue — no extra step, ever. The lifecycle:
- open
- fixed (your claim)
- verified-fixed
- …or REGRESSED
- Open — a finding arrived and no matching issue existed.
- Dispositions are your calls on an issue: fixed, planned, investigating, wontfix, thanks, deferred. Only fixed and wontfix move the status; the rest are commentary on the record. Your AI never sets a disposition on its own — these are your decisions.
- Fixed is a claim, not a result. The real check is a verify round: re-run the same scenario with the same persona that found the issue. If it doesn't recur, the issue resolves to verified-fixed. If it comes back, it flips to regressed.
- Regressed is the loudest alarm in SQUAT — it means a claimed fix didn't hold. The issues queue sorts regressed first, always.
Verify rounds
To verify claimed fixes, say “run a verify round for the save bug.” SQUAT targets the named issues in the next round, and on completion resolves each to verified-fixed or regressed. The standing rule is same-persona verification — partial fixes are common, and they only surface under a genuine re-attempt by the user who hit the problem. If the original finder isn't in the round's cast, the verify still completes but is flagged as weaker evidence rather than silently counted as equal.
Invalidation: correcting the record without erasing it
When a session was compromised — a deploy landed mid-round, the harness misfired, wrong credentials were used — the result (or the whole round) is invalidated:
- A real, plain-language reason is always required.
- The original data isn't touched — invalidation adds a marker, a reason, and a timestamp.
- Invalidated data stays visible everywhere (the ledger marks it with
⊘and shows the reason) but is excluded from every count, trend, priority list, and comparison. - There is deliberately no un-invalidate. If that feels strict, that's the point: the trend you make decisions on only means something if nothing can be quietly rewritten.
Pretending a bad session never happened would erase real information about what went wrong with the harness. Invalidate, don't delete. (Deleting exists too — see the trash — but it's for data that shouldn't be there at all: a wrong import, test noise. A compromised session is real history; it stays.)