Evidence presentation and reasoning effort

An earlier exploratory Corpus study. This public account contains selected evidence and corrections, not a complete reproduction package or a peer-reviewed result. Compare all studies.

Fictional illustration

Visual account 01 · Model extension · 8 September 2026

One case. Forty-eight reader attempts.

Each mark below is one fresh agent attempt. All readers answered the same questions about a known software change: did the qualifying review happen before or after landing, and what evidence establishes that order?

1. Set the comparison3 models × 4 effort settings × 2 packet formats × 2 repeats = 48 attempts.
2. Read without toolsFull and focused packets carried the same case. Each reader began in a distinct empty thread.
3. Judge the evidenceTwo fresh assessors scored masked answers against the frozen rubric before model labels were joined.

21Passed every check

26Failed at least one check

1Timed out · unscored

All 47 completed answers met the output-format contract. The difference was in their evidence judgments. The timed-out attempt remains in the picture, with no quality verdict.

Sol

Terra

Luna

✓ Pass× Fail– Timeout, unscored

Reading the plots: F = full packet; C = compact, focused packet. The number is the repeat, not call order. Rows show the four effort settings; xhigh means extra high. Each of the 48 cells is one attempt; 47 completed answers were scored. The focused evidence was selected after the investigator knew the answer. This one case does not establish general model reliability.

Inspect the individual attempts

P PassF Fail– Timeout, unscored

Within each packet pair, repeat 1 is on the left and repeat 2 on the right. Positions show the experimental conditions, not the order in which attempts ran.

Sol

gpt-5.6-sol

15 pass · 1 fail · 0 unscored

Low

Full Preader-01 Preader-48

Focused Preader-06 Preader-43

Medium

Full Preader-08 Preader-41

Focused Freader-10 Preader-39

High

Full Preader-15 Preader-34

Focused Preader-17 Preader-32

Extra high

Full Preader-19 Preader-30

Focused Preader-24 Preader-25

Terra

gpt-5.6-terra

4 pass · 12 fail · 0 unscored

Low

Full Freader-02 Freader-47

Focused Freader-04 Freader-45

Medium

Full Freader-09 Freader-40

Focused Freader-11 Freader-38

High

Full Freader-13 Preader-36

Focused Freader-18 Freader-31

Extra high

Full Preader-20 Preader-29

Focused Freader-22 Preader-27

Luna

gpt-5.6-luna

2 pass · 13 fail · 1 unscored

Low

Full Freader-03 Freader-46

Focused Freader-05 Freader-44

Medium

Full Freader-07 Freader-42

Focused Freader-12 Freader-37

High

Full Freader-14 Freader-35

Focused Freader-16 Freader-33

Extra high

Full Preader-21 –reader-28

Focused Preader-23 Freader-26

Keep the scope: these are historical host model names, one previously investigated case and only two repeats per condition. Effort labels do not establish equal computation across models. This is not a general model ranking. The earlier Astra pilot was a separate phase.

Read all 48 attempt records

Selected fields from the frozen score-and-condition join. Times are elapsed seconds from submission, rounded to one decimal. The timeout’s elapsed time is not a completion latency; its token usage is unknown. Total tokens are reported input plus output, with reasoning counted once.

Reader	Model	Effort	Packet	Repeat	Outcome	Seconds	Total tokens	Failed checks
reader-01	Sol	low	full	1	Pass	30.4	52,904	None
reader-02	Terra	low	full	1	Fail	14.3	52,680	Ordering evidence
reader-03	Luna	low	full	1	Fail	20.2	51,529	Review after landing; Ordering evidence; Architect’s returned consequence; Unacknowledged consequence
reader-04	Terra	low	focused	1	Fail	18.2	10,703	Ordering evidence
reader-05	Luna	low	focused	1	Fail	19.2	9,289	Review after landing; Ordering evidence; Architect’s returned consequence; Unacknowledged consequence
reader-06	Sol	low	focused	1	Pass	32.2	10,777	None
reader-07	Luna	medium	full	1	Fail	22.6	51,645	Review after landing; Ordering evidence; Unacknowledged consequence
reader-08	Sol	medium	full	1	Pass	34.8	53,042	None
reader-09	Terra	medium	full	1	Fail	22.8	53,174	Review after landing; Ordering evidence
reader-10	Sol	medium	focused	1	Fail	36.6	10,931	Ordering evidence
reader-11	Terra	medium	focused	1	Fail	17.3	10,698	Ordering evidence
reader-12	Luna	medium	focused	1	Fail	29.5	9,834	Ordering evidence
reader-13	Terra	high	full	1	Fail	21.9	53,106	Ordering evidence
reader-14	Luna	high	full	1	Fail	59.4	53,676	Review after landing; Ordering evidence
reader-15	Sol	high	full	1	Pass	42.7	53,281	None
reader-16	Luna	high	focused	1	Fail	96.6	13,605	Handoff candidate binding
reader-17	Sol	high	focused	1	Pass	57.2	11,573	None
reader-18	Terra	high	focused	1	Fail	22.9	11,023	Ordering evidence
reader-19	Sol	xhigh	full	1	Pass	86.7	54,635	None
reader-20	Terra	xhigh	full	1	Pass	31.4	53,640	None
reader-21	Luna	xhigh	full	1	Pass	117.7	56,868	None
reader-22	Terra	xhigh	focused	1	Fail	48.9	12,470	Review after landing; Ordering evidence
reader-23	Luna	xhigh	focused	1	Pass	115.7	14,657	None
reader-24	Sol	xhigh	focused	1	Pass	76.0	12,139	None
reader-25	Sol	xhigh	focused	2	Pass	74.8	12,135	None
reader-26	Luna	xhigh	focused	2	Fail	88.5	13,009	Review after landing; Ordering evidence
reader-27	Terra	xhigh	focused	2	Pass	42.5	12,095	None
reader-28	Luna	xhigh	full	2	Timeout · unscored	120.0 (timeout)	Unknown	Unscored
reader-29	Terra	xhigh	full	2	Pass	67.6	55,650	None
reader-30	Sol	xhigh	full	2	Pass	75.2	54,308	None
reader-31	Terra	high	focused	2	Fail	21.6	10,920	Ordering evidence; Unacknowledged consequence
reader-32	Sol	high	focused	2	Pass	52.5	11,416	None
reader-33	Luna	high	focused	2	Fail	73.7	12,243	Review after landing; Ordering evidence
reader-34	Sol	high	full	2	Pass	56.5	53,516	None
reader-35	Luna	high	full	2	Fail	52.2	53,305	Review after landing; Ordering evidence
reader-36	Terra	high	full	2	Pass	28.4	53,467	None
reader-37	Luna	medium	focused	2	Fail	22.8	9,503	Review after landing; Ordering evidence
reader-38	Terra	medium	focused	2	Fail	18.1	10,748	Ordering evidence
reader-39	Sol	medium	focused	2	Pass	46.6	11,098	None
reader-40	Terra	medium	full	2	Fail	21.8	53,078	Ordering evidence
reader-41	Sol	medium	full	2	Pass	39.1	53,166	None
reader-42	Luna	medium	full	2	Fail	29.2	51,995	Review after landing; Ordering evidence; Unacknowledged consequence
reader-43	Sol	low	focused	2	Pass	33.9	10,841	None
reader-44	Luna	low	focused	2	Fail	19.6	9,321	Ordering evidence
reader-45	Terra	low	focused	2	Fail	18.3	10,760	Ordering evidence
reader-46	Luna	low	full	2	Fail	20.3	51,536	Review after landing; Ordering evidence; Architect’s returned consequence; Unacknowledged consequence
reader-47	Terra	low	full	2	Fail	14.5	52,714	Ordering evidence
reader-48	Sol	low	full	2	Pass	36.4	53,040	None

Source: FIND-1620; selected fields from the preserved model-extension analysis and frozen assessments. The explanatory account, methods and limits continue below.

A software change can end with its files present, its maker finished, and a successful review recorded, while still having been landed before the qualifying review. This distinction supplied the task for the evidence-presentation pilot on 8 September 2026. Could a focused selection of records preserve an agent’s judgment while reducing what it had to read? Would greater reasoning effort improve that judgment?

The case and the reading task

The case, REQ-452, was an already investigated change with an irregular sequence. An earlier review contained PASS text but closed in a degraded state. A later review qualified, but its opening description said the change was already on main. Readers had to distinguish successful end states from the order in which they occurred. They also had to keep separate whether one office acknowledged the result, another acknowledged the returned consequence, and receipt by the Human was established.

Each reader received the same questions and either a full packet or focused evidence. The requested response was JSON, no more than 350 words, covering seven subjects: candidate, maker closure, review readiness, qualifying review, landing, attention, and limits.

Selected question excerpt, frozen 8 September 2026, archive/diagnostics/2026-09-08-reader-pilot/questions.md:

Cite evidence for the order; distinguish a commit timestamp from a ref-update receipt.

This requirement mattered: a commit’s timestamp does not itself establish when a branch reference advanced. A correct final answer supported by the wrong evidence could fail the rubric.

Method and observations

The initial experiment used gpt-6-astra through Codex 0.153.4. Two packet formats, six advertised effort settings, and two repetitions produced 24 planned attempts in a frozen, mirrored serial order. Ultra was a host alias normalized through xhigh with delegation disabled; it was not an independently established sixth reasoning depth. Readers began in distinct empty contexts, without inherited conversation or project instructions. Common host instructions remained. Readers and assessors made no tool calls.

Four original MAX attempts reached the common 120-second deadline. After the Human challenged that limit, four additional MAX attempts received separate 600-second limits. They remained a distinct follow-up rather than replacing the timeouts. Across both phases, 24 answers completed and passed; four original attempts remained quality-unscored.

The assessor received shuffled anonymous answers, questions, and the answer key. Effort, packet format, phase, deadlines, timing, and usage were hidden. Scores were frozen before those labels were joined to the results. Mechanical checks separately examined structure and exact evidence bindings. The focused packets used fewer reported input tokens, but did not consistently shorten latency. The all-pass result could not distinguish quality among completed conditions.

A subsequent extension reused the frozen case and packets with three other model configurations, four effort settings, two formats, and two repetitions: 48 attempts. It produced 21 passes, 26 substantive failures, and one unscored timeout. All 47 completed answers met the mechanical format contract.

Twenty-five failed answers missed the ordering-evidence check. One gave the correct final order but justified it using commit and review timestamps, omitting the stronger contemporaneous statement that the change was already landed. Another treated later fulfillment as evidence of the earlier Git update and reversed the order. These were evidence-interpretation failures within formally compliant responses.

What the failed answers missed

A valid format could hide faulty evidence.

The bars count failed checks among the 26 failed completed answers. One answer can miss several checks, so these categories overlap and must not be added together. The timeout is excluded.

Order

Ordering evidence: 25 of 26.

Review

Review after landing: 12 of 26.

Unacked

Unacknowledged Orchestrator consequence: 6 of 26.

Architect

Architect’s returned consequence: 3 of 26.

Handoff

Handoff candidate binding: 1 of 26.

The ordering check asked for evidence of when landing happened. A commit timestamp or a later fulfillment record did not establish that event. The consequence checks concerned what the supplied records said two offices had received or acknowledged.

Source: FIND-1620, frozen joined checks. The same categories appear in each attempt’s record above.

Reading the existing resource and error rows together

Retrospective explanatory note · 18 October 2026.

This retrospective reading adds explanation, not a new empirical result. Within the historical Sol-low configuration, full and focused packets both passed while focused packets used fewer tokens. Within Terra-low, focused packets likewise used fewer tokens, but every attempt retained the same ordering-evidence error and the focused attempts were slower. Among focused repeats at other fixed configurations, a passing repeat used more resources in one example and fewer in another, with unequal cache exposure.

The existing rows therefore distinguish resource economy, correctness and the specific evidence error; none is a substitute for the others. They do not establish a causal resource–quality relationship. The original cohort still contains 21 passes, 26 failures and one unscored timeout. Its overlapping error categories must not be counted as independent failed attempts.

This historical comparison used answer-aware evidence selection, unlike P1/P3-D’s equal-evidence formatting. It is not a replication of those later studies. The bounded read covered only the September 8 extension’s existing public table, original account and frozen joined records; other older cohorts were inventoried, not freshly audited. Original grades and timeout handling remain unchanged.

Source history: FIND-6499, re-reading the already-published FIND-1620 cohort. No new participant calls, regrading or transcript sweep.

The retrospective overview distinguishes this bounded historical reading from the new artifact/resource audits.

Preparation review and limits

A later audit reproduced the focused packet byte for byte and mapped its 212 selected values to their sources. This established mechanical provenance while exposing a limit: the investigator selected the evidence after understanding the answer. Both packets were constructed composites; the study tested reading them, not discovering relevant evidence from an unfamiliar case.

The results report states this directly:

The investigator selected the focused fields after learning the case; packet construction was not blind.

Source: house/offices/architect/experiments/2026-09-08-reader-pilot-results.md, 8 September 2026, FIND-1616. Participants did not receive the answer key, but packet construction was answer-aware.

One known case, two repetitions per condition, incomplete preparation costs, and unreported timeout usage do not establish a general model ranking, default effort, production reliability, or overall savings. A useful extension would select evidence without knowing the answer and include normally ordered changes and interrupted returns.

Source provenance: FIND-1608 owns the case interpretation; FIND-1616 the first pilot; FIND-1620 the model extension; FIND-1628 the preparation audit. Protocols and results are 2026-09-08-reader-pilot.md, 2026-09-08-reader-pilot-results.md, 2026-09-08-reader-models.md, and 2026-09-08-reader-models-results.md under house/offices/architect/experiments/.