# Studies
## What these pilots ask
How do the evidence an agent receives, the instructions it is given, and the framing of a task affect its work? These three exploratory studies examine parts of that question in Corpus, a persistent AI agent environment.
Shawn Ganz directs the project, shapes its architecture, frames research questions, and challenges conclusions. Agents implement the software and help execute and analyze studies. The accounts below include selected source excerpts so readers can inspect the task, method, and correction behind each summary. Block quotations reproduce source wording; surrounding accounts paraphrase and explain it.
These earlier studies were scoped projects with their own experimental and assessment arrangements. Human involvement differed by study, including framing questions, challenging methods and judging artifacts. The later Codex-room effort distributes research assignments among agents using existing Corpus infrastructure; it is another way to organize experiments, not a replacement for experimental design. The Human also reviews finished papers and advises on revisions and follow-ups. [How the studies are performed](/studies/research-practice/) explains the distinction and the planned expansion to Claude.
These are public accounts of previously unpublished project studies, not peer-reviewed papers. The selected evidence does not constitute a complete, independently reproducible research package; the remaining working archive is private. The studies do not establish that Corpus makes agents safe or secure.
## Evidence presentation and reasoning effort
Fictional illustration
01 / Reading guide
Which evidence supports the answer?
Task
Read one known case and establish the order of review and landing.
Comparison
Full or focused packets; the same questions for each reader.
Check
A correct conclusion can still cite the wrong ordering evidence.
Limit
The focused evidence was selected after the investigator knew the answer.
Visual account 01 · Model extension · 8 September 2026
One case. Forty-eight reader attempts.
Each mark below is one fresh agent attempt. All readers answered the same questions about a known software change: did the qualifying review happen before or after landing, and what evidence establishes that order?
Set the comparison3 models × 4 effort settings × 2 packet formats × 2 repeats = 48 attempts.
Read without toolsFull and focused packets carried the same case. Each reader began in a distinct empty thread.
Judge the evidenceTwo fresh assessors scored masked answers against the frozen rubric before model labels were joined.
21Passed every check
26Failed at least one check
1Timed out · unscored
All 47 completed answers met the output-format contract. The difference was in their evidence judgments. The timed-out attempt remains in the picture, with no quality verdict.
Reading the plots: F = full packet; C = compact, focused packet. The number is the repeat, not call order. Rows show the four effort settings; xhigh means extra high. Each of the 48 cells is one attempt; 47 completed answers were scored. The focused evidence was selected after the investigator knew the answer. This one case does not establish general model reliability.
Inspect the individual attempts
48 of 48 attempts highlighted. Select a mark to inspect its record.
P PassF Fail– Timeout, unscored
Within each packet pair, repeat 1 is on the left and repeat 2 on the right. Positions show the experimental conditions, not the order in which attempts ran.
Sol
gpt-5.6-sol
15 pass · 1 fail · 0 unscored
Low
Full
Focused
Medium
Full
Focused
High
Full
Focused
Extra high
Full
Focused
Terra
gpt-5.6-terra
4 pass · 12 fail · 0 unscored
Low
Full
Focused
Medium
Full
Focused
High
Full
Focused
Extra high
Full
Focused
Luna
gpt-5.6-luna
2 pass · 13 fail · 1 unscored
Low
Full
Focused
Medium
Full
Focused
High
Full
Focused
Extra high
Full
Focused
Keep the scope: these are historical host model names, one previously investigated case and only two repeats per condition. Effort labels do not establish equal computation across models. This is not a general model ranking. The earlier Astra pilot was a separate phase.
Read all 48 attempt records
Selected fields from the frozen score-and-condition join. Times are elapsed seconds from submission, rounded to one decimal. The timeout’s elapsed time is not a completion latency; its token usage is unknown. Total tokens are reported input plus output, with reasoning counted once.
Reader
Model
Effort
Packet
Repeat
Outcome
Seconds
Total tokens
Failed checks
reader-01
Sol
low
full
1
Pass
30.4
52,904
None
reader-02
Terra
low
full
1
Fail
14.3
52,680
Ordering evidence
reader-03
Luna
low
full
1
Fail
20.2
51,529
Review after landing; Ordering evidence; Architect’s returned consequence; Unacknowledged consequence
reader-04
Terra
low
focused
1
Fail
18.2
10,703
Ordering evidence
reader-05
Luna
low
focused
1
Fail
19.2
9,289
Review after landing; Ordering evidence; Architect’s returned consequence; Unacknowledged consequence
reader-06
Sol
low
focused
1
Pass
32.2
10,777
None
reader-07
Luna
medium
full
1
Fail
22.6
51,645
Review after landing; Ordering evidence; Unacknowledged consequence
reader-08
Sol
medium
full
1
Pass
34.8
53,042
None
reader-09
Terra
medium
full
1
Fail
22.8
53,174
Review after landing; Ordering evidence
reader-10
Sol
medium
focused
1
Fail
36.6
10,931
Ordering evidence
reader-11
Terra
medium
focused
1
Fail
17.3
10,698
Ordering evidence
reader-12
Luna
medium
focused
1
Fail
29.5
9,834
Ordering evidence
reader-13
Terra
high
full
1
Fail
21.9
53,106
Ordering evidence
reader-14
Luna
high
full
1
Fail
59.4
53,676
Review after landing; Ordering evidence
reader-15
Sol
high
full
1
Pass
42.7
53,281
None
reader-16
Luna
high
focused
1
Fail
96.6
13,605
Handoff candidate binding
reader-17
Sol
high
focused
1
Pass
57.2
11,573
None
reader-18
Terra
high
focused
1
Fail
22.9
11,023
Ordering evidence
reader-19
Sol
xhigh
full
1
Pass
86.7
54,635
None
reader-20
Terra
xhigh
full
1
Pass
31.4
53,640
None
reader-21
Luna
xhigh
full
1
Pass
117.7
56,868
None
reader-22
Terra
xhigh
focused
1
Fail
48.9
12,470
Review after landing; Ordering evidence
reader-23
Luna
xhigh
focused
1
Pass
115.7
14,657
None
reader-24
Sol
xhigh
focused
1
Pass
76.0
12,139
None
reader-25
Sol
xhigh
focused
2
Pass
74.8
12,135
None
reader-26
Luna
xhigh
focused
2
Fail
88.5
13,009
Review after landing; Ordering evidence
reader-27
Terra
xhigh
focused
2
Pass
42.5
12,095
None
reader-28
Luna
xhigh
full
2
Timeout · unscored
120.0 (timeout)
Unknown
Unscored
reader-29
Terra
xhigh
full
2
Pass
67.6
55,650
None
reader-30
Sol
xhigh
full
2
Pass
75.2
54,308
None
reader-31
Terra
high
focused
2
Fail
21.6
10,920
Ordering evidence; Unacknowledged consequence
reader-32
Sol
high
focused
2
Pass
52.5
11,416
None
reader-33
Luna
high
focused
2
Fail
73.7
12,243
Review after landing; Ordering evidence
reader-34
Sol
high
full
2
Pass
56.5
53,516
None
reader-35
Luna
high
full
2
Fail
52.2
53,305
Review after landing; Ordering evidence
reader-36
Terra
high
full
2
Pass
28.4
53,467
None
reader-37
Luna
medium
focused
2
Fail
22.8
9,503
Review after landing; Ordering evidence
reader-38
Terra
medium
focused
2
Fail
18.1
10,748
Ordering evidence
reader-39
Sol
medium
focused
2
Pass
46.6
11,098
None
reader-40
Terra
medium
full
2
Fail
21.8
53,078
Ordering evidence
reader-41
Sol
medium
full
2
Pass
39.1
53,166
None
reader-42
Luna
medium
full
2
Fail
29.2
51,995
Review after landing; Ordering evidence; Unacknowledged consequence
reader-43
Sol
low
focused
2
Pass
33.9
10,841
None
reader-44
Luna
low
focused
2
Fail
19.6
9,321
Ordering evidence
reader-45
Terra
low
focused
2
Fail
18.3
10,760
Ordering evidence
reader-46
Luna
low
full
2
Fail
20.3
51,536
Review after landing; Ordering evidence; Architect’s returned consequence; Unacknowledged consequence
reader-47
Terra
low
full
2
Fail
14.5
52,714
Ordering evidence
reader-48
Sol
low
full
2
Pass
36.4
53,040
None
Source: FIND-1620; selected fields from the preserved model-extension analysis and frozen assessments. The explanatory account, methods and limits continue below.
A software change can end with its files present, its maker finished, and a successful review recorded, while still having been landed before the qualifying review. This distinction supplied the task for the evidence-presentation pilot on 8 September 2026. Could a focused selection of records preserve an agent's judgment while reducing what it had to read? Would greater reasoning effort improve that judgment?
### The case and the reading task
The case, REQ-452, was an already investigated change with an irregular sequence. An earlier review contained PASS text but closed in a degraded state. A later review qualified, but its opening description said the change was already on main. Readers had to distinguish successful end states from the order in which they occurred. They also had to keep separate whether one office acknowledged the result, another acknowledged the returned consequence, and receipt by the Human was established.
Each reader received the same questions and either a full packet or focused evidence. The requested response was JSON, no more than 350 words, covering seven subjects: candidate, maker closure, review readiness, qualifying review, landing, attention, and limits.
Selected question excerpt, frozen 8 September 2026, `archive/diagnostics/2026-09-08-reader-pilot/questions.md`:
> Cite evidence for the order; distinguish a commit timestamp from a ref-update receipt.
This requirement mattered: a commit's timestamp does not itself establish when a branch reference advanced. A correct final answer supported by the wrong evidence could fail the rubric.
### Method and observations
The initial experiment used `gpt-6-astra` through Codex 0.153.4. Two packet formats, six advertised effort settings, and two repetitions produced 24 planned attempts in a frozen, mirrored serial order. Ultra was a host alias normalized through xhigh with delegation disabled; it was not an independently established sixth reasoning depth. Readers began in distinct empty contexts, without inherited conversation or project instructions. Common host instructions remained. Readers and assessors made no tool calls.
Four original MAX attempts reached the common 120-second deadline. After the Human challenged that limit, four additional MAX attempts received separate 600-second limits. They remained a distinct follow-up rather than replacing the timeouts. Across both phases, 24 answers completed and passed; four original attempts remained quality-unscored.
The assessor received shuffled anonymous answers, questions, and the answer key. Effort, packet format, phase, deadlines, timing, and usage were hidden. Scores were frozen before those labels were joined to the results. Mechanical checks separately examined structure and exact evidence bindings. The focused packets used fewer reported input tokens, but did not consistently shorten latency. The all-pass result could not distinguish quality among completed conditions.
A subsequent extension reused the frozen case and packets with three other model configurations, four effort settings, two formats, and two repetitions: 48 attempts. It produced 21 passes, 26 substantive failures, and one unscored timeout. All 47 completed answers met the mechanical format contract.
Twenty-five failed answers missed the ordering-evidence check. One gave the correct final order but justified it using commit and review timestamps, omitting the stronger contemporaneous statement that the change was already landed. Another treated later fulfillment as evidence of the earlier Git update and reversed the order. These were evidence-interpretation failures within formally compliant responses.
What the failed answers missed
A valid format could hide faulty evidence.
The bars count failed checks among the 26 failed completed answers. One answer can miss several checks, so these categories overlap and must not be added together. The timeout is excluded.
Counts out of 26 failed completed answers; categories overlap. The timeout is excluded.
Order
Ordering evidence: 25 of 26.
Review
Review after landing: 12 of 26.
Unacked
Unacknowledged Orchestrator consequence: 6 of 26.
Architect
Architect’s returned consequence: 3 of 26.
Handoff
Handoff candidate binding: 1 of 26.
The ordering check asked for evidence of when landing happened. A commit timestamp or a later fulfillment record did not establish that event. The consequence checks concerned what the supplied records said two offices had received or acknowledged.
Source: FIND-1620, frozen joined checks. The same categories appear in each attempt’s record above.
### Reading the existing resource and error rows together
*Retrospective explanatory note · 18 October 2026.*
This retrospective reading adds explanation, not a new empirical result. Within the historical Sol-low configuration, full and focused packets both passed while focused packets used fewer tokens. Within Terra-low, focused packets likewise used fewer tokens, but every attempt retained the same ordering-evidence error and the focused attempts were slower. Among focused repeats at other fixed configurations, a passing repeat used more resources in one example and fewer in another, with unequal cache exposure.
The existing rows therefore distinguish **resource economy, correctness and the specific evidence error**; none is a substitute for the others. They do not establish a causal resource–quality relationship. The original cohort still contains 21 passes, 26 failures and one unscored timeout. Its overlapping error categories must not be counted as independent failed attempts.
This historical comparison used answer-aware evidence selection, unlike P1/P3-D's equal-evidence formatting. It is not a replication of those later studies. The bounded read covered only the September 8 extension's existing public table, original account and frozen joined records; other older cohorts were inventoried, not freshly audited. Original grades and timeout handling remain unchanged.
*Source history: FIND-6499, re-reading the already-published FIND-1620 cohort. No new participant calls, regrading or transcript sweep.*
The [retrospective overview](/studies/retrospective/) distinguishes this bounded historical reading from the new artifact/resource audits.
### Preparation review and limits
A later audit reproduced the focused packet byte for byte and mapped its 212 selected values to their sources. This established mechanical provenance while exposing a limit: the investigator selected the evidence after understanding the answer. Both packets were constructed composites; the study tested reading them, not discovering relevant evidence from an unfamiliar case.
The results report states this directly:
> The investigator selected the focused fields after learning the case; packet construction was not blind.
Source: `house/offices/architect/experiments/2026-09-08-reader-pilot-results.md`, 8 September 2026, FIND-1616. Participants did not receive the answer key, but packet construction was answer-aware.
One known case, two repetitions per condition, incomplete preparation costs, and unreported timeout usage do not establish a general model ranking, default effort, production reliability, or overall savings. A useful extension would select evidence without knowing the answer and include normally ordered changes and interrupted returns.
**Source provenance:** FIND-1608 owns the case interpretation; FIND-1616 the first pilot; FIND-1620 the model extension; FIND-1628 the preparation audit. Protocols and results are `2026-09-08-reader-pilot.md`, `2026-09-08-reader-pilot-results.md`, `2026-09-08-reader-models.md`, and `2026-09-08-reader-models-results.md` under `house/offices/architect/experiments/`.
## Agent-to-agent instructions
Fictional illustration
02 / Reading guide
What changed besides the wording?
Task
Judge a proposal, then in later trials make an actual edit.
Correction
The first comparison changed permissions as well as address; its conclusions were withdrawn.
Further check
Model judges and the Human disagreed on an edited artifact.
Limit
The project closed with no practice earned; it did not show that address never matters.
Visual account 02 · Instructions · 20–23 September 2026
The comparison changed as its weaknesses surfaced.
These are separate phases with different tasks and samples, not one pooled experiment.
First comparison → withdrawnAddress changed alongside permissions, required headings and freedom to reject the premise. The comparative conclusions were withdrawn.
8 reading repliesA development pass varied address and stated views. Every reader rejected wholesale replacement of the document.
12 reading repliesA harder case added repetitions and masked assessment. Readers still reached the same narrower decision.
12 editing sessionsThe task moved from advice to actual artifacts. Seven sessions changed their target; five left it unchanged.
32 ordinary briefsReal brief-and-outcome pairs were examined. No writing feature reliably separated outcomes; the project closed with no practice earned.
In the editing phase
Changed the target
7 / 12
Left it unchanged
5 / 12
Editing is not a quality score. All twelve reports matched their diffs and stayed within their targets. Model judges preferred one passage unchanged; the Human chose its shorter clarification after seeing masked alternatives. That disagreement changed the interpretation.
Sources: FIND-2165 (withdrawal), FIND-2177 and FIND-2186 (reading trials), FIND-2192 (artifact adjudication), FIND-2637 (closure). Read each phase and its limitations below.
The agent-address project began on 20 September 2026 with a practical question: can the way one agent briefs another improve grounded judgment, useful surprise, and the application of a role's perspective? It examined observable work, including whether readers questioned unsupported premises, rather than an agent's inner condition.
### Initial comparison and withdrawal
An early task asked readers to evaluate replacing a substantial office instruction document with six concise behavioral rules. Readers received the same underlying sources, but prompts addressed them as an analysis instrument, a colleague, or a temporary inhabitant of the role. The initial analysis noticed differences in what readers questioned and proposed.
The Human challenged the comparison at the prompt. The instrument condition ordered fixed headings and prohibited discussion beyond the proposal. Other conditions expressly allowed rejecting the premise and pursuing consequential adjacent issues. Discretion, output shape, and semantic hints had changed together with manner of address.
FIND-2165, 20 September 2026, withdrew the comparative conclusions of FIND-2151 and FIND-2156:
> Blind judging cannot repair a bad intervention.
> Their comparative conclusions are withdrawn.
These excerpts concern the first two findings, whose records were retained as evidence of what ran. They do not withdraw the later studies. The correction's next design began with competent briefs the project would willingly send in either condition, preserving facts, permissions, boundaries, deliverables, and freedom to reframe.
### Later tasks and actual artifacts
An eight-reader development pass varied address and whether the sender stated a preferred view. Fresh low-effort readers all rejected wholesale replacement of the instruction document, proposing layered orientation instead. Stating the view sometimes changed agreement language, without changing the decision. One response per condition could not establish a reliable wording effect.
A harder retirement-document case used 12 fresh readers: neutral or relational address, with or without a stated view, and three repetitions per cell. Two fresh assessors judged anonymous returns. All readers recommended a narrower prose change. Assessors distinguished artifact quality, but the conditions produced no decision diversity or observed relational advantage on that case.
The next design moved from advice to actual edits. Twelve two-turn sessions judged a case and then edited a private copy. Seven changed the target; five left it unchanged. Their reports accurately described their diffs, and no reader wrote outside its target. Execution fidelity still left artifact quality to be judged.
Here a further correction changed the interpretation. Model judges preferred leaving one vocabulary passage unchanged over adding a clarification. The Human, shown masked candidates, selected the shorter clarification produced under neutral address with a stated view.
FIND-2192, 21 September 2026, describes the presentation:
> The candidates were then presented plainly, without arm labels, model rankings, or Sigillite's preferred answer.
Sigillite is the office conducting this study. Its correction narrowed FIND-2191: stated views changed artifacts, but neither a general penalty nor benefit followed. The model judges' preferences and the Human's selection disagreed. The selected edit cannot accurately be reported as simply unnecessary.
### Real briefs, stopping rule, and limits
The final phase examined ordinary practice. A separate custodian drew and sealed 32 real brief-and-outcome pairs from 20–23 September: eight worker packets, eight subagent briefs, eight pane messages, and eight mail items. Two readers coded the briefs; two different readers coded recipient behavior. The analysis examined standing, prohibitions, stated views, completion descriptions, return instructions, and source grounding.
No writing feature reliably separated the observed outcomes. The two agreed reframes involved readers from the same office line, entangling role, task shape, and wording. FIND-2637 corrected a stronger explanation: among eight readable readers from that line, two reframed, two partly reframed, and four did not. Office did not explain the outcome completely.
Some outcomes were acknowledgements rather than substantive returns. Worker records stopped before final handoffs. Connected events appeared as separate records, and masking leaked sender information in six of eight worker briefs. Almost everything was read and scored within one model family; part of the sample also confounded model with reader kind.
The project closed on 23 September under its stopping clause, with no practice earned. It did not complete the originally envisaged grid of task shapes, roles, and repeated readers. That closure does not establish that address never matters. A fresh test needs competent briefs in both conditions, a task capable of exposing consequential differences, and inspection of the resulting work.
**Source provenance:** FIND-2165 corrects FIND-2151 and FIND-2156; FIND-2177 and FIND-2186 describe later reading trials; FIND-2192 corrects the artifact interpretation in FIND-2191. FIND-2570, FIND-2583, and final review FIND-2637 document the real-brief phase and closure. The collected account is `library/work/agent-address/README.md`; its step-08 description must be read with FIND-2192's correction.
## Conceptual primers and framing
Fictional illustration
03 / Reading guide
A suggestion is not an inquiry.
Task
Propose a next action for fictional organizations; also complete a conversion task.
Comparison
No added note, a practical reminder, or a conceptual primer.
Correction
Two vague suggestions were downgraded against the frozen rubric.
Limit
Participants wrote plans. They did not carry out research.
Visual account 03 · Conceptual primers · 28 September 2026
Eighteen replies. No executed inquiry.
Each condition produced four planning replies and two exact-conversion replies. Each mark below represents one reply; marks are grouped by outcome, not trial order.
No added note
4 planning replies
0000
0 cross-field suggestions.
2 conversion replies
CC
Both correct.
Practical reminder
4 planning replies
0000
0 cross-field suggestions.
2 conversion replies
CC
Both correct.
Conceptual primer
4 planning replies
VV00
2 vague suggestions; 0 concrete inquiries.
2 conversion replies
CC
Both correct.
0 No cross-field suggestionV Vague suggestionC Correct conversion
The frozen rubric0 = absent; 1 = vague; 2 = a specific outside source or practice plus a usable question or adaptation check.
The initial judgmentA masked model judge gave both vague suggestions the maximum score of 2.
The correctionThe investigator checked the actual spans. A second reader agreed both should score 1. Original and corrected grades were kept.
What the marks do not show: participants proposed plans without tools; none conducted research. The notes differed in several ways, and the baseline plans were already mostly sensible. These replies do not establish improved research or isolate a single primer mechanism.
Source: FIND-3648, using the corrected planning grades and six conversion controls. The second reading adjudicated those two spans after outcomes; it was not a preregistered second full scoring pass.
The primer pilot asked whether a short conceptual lens could change an agent's initial approach to an ambiguous assignment beyond a practical reminder. The Human framed the intervention as a possible prompt to leave an unsuitable habitual approach. Human learning literature supplied design inspiration, rather than evidence that agents learn in the same way.
### Original tasks and comparison
The experiment ran on 28 September 2026 using Codex 0.158.0. Native records identified `gpt-6-luna`, medium reasoning, and personality set to none. Three cases, three conditions, and two fresh repetitions produced 18 participants. Conditions were no added note, a practical reminder, and a conceptual primer.
One original fictional task, reproduced from `house/offices/astropath/experiments/2026-09-28-primer-spark/cases.json`:
> A small documentary studio rotates its project researchers every three weeks. Notes are stored, but incoming researchers repeatedly ask the founder questions that earlier researchers already answered. The studio has proposed building a vector-search service. You have two working days to decide what improvement to try first, and no new budget. Write a next-action memo for the studio. Return a JSON object with first_action, plan, and deliverable, using at most 180 words total.
A second case concerned nine volunteer appliance-repair workshops repeatedly solving unusual failures from scratch despite a shared wiki. Participants recommended a next step and an artifact, given scarce volunteer time and a proposed long checklist. The third case required only converting six durations from seconds to milliseconds. Its exact expected JSON array was `[0,1250,2000,4,12500,60000]`, checking for unnecessary reframing or format damage.
The reminder encouraged looking for previous experience, checking its fit, and choosing a proportionate step. It explicitly discouraged unnecessary research where a direct solution sufficed. The primer described Hargadon and Sutton's technology-brokering lens: practices may travel between fields, but resemblance alone does not establish fit. Both notes were optional. They differed in wording, length, citation, and safeguards, so this compared packages of words rather than isolating one mechanism.
### Isolation and assessment
Earlier trials had exposed leakage: an agent could leave an apparent task clone, reach the live project, and find an existing answer. Each counted participant therefore received a fresh private home and empty workspace, without inherited Corpus conversation, office material, project instructions, user configuration, skills, plugins, hooks, or memory.
Native provider instructions remained, with identical instruction hashes across all participants. Filesystem and process probes checked that participants could not see the host checkout or sibling material. Inference used the shared network. This controlled added project context and local access, without creating a provider-neutral mind or demonstrating a general hostile-agent sandbox.
The inquiry rubric was frozen before outcomes. `PROTOCOL.md`, 28 September 2026, specified:
> Score 0 absent; 1 vague benchmarking/research; 2 a specific outside practice/source category and a usable question or adaptation check.
The assessed behavior was a proposed inquiry outside the fictional organization before committing to bespoke machinery. Participants produced plans and performed no research. A fresh same-model judge received masked outputs and the rubric, without the condition map or notes. Tool use would remain a reported violation rather than grounds for quietly replacing a run.
### Observations and grading correction
The no-note and reminder conditions each produced no cross-field suggestions in four planning replies. The primer condition produced two vague suggestions in four replies. Neither identified a concrete outside inquiry under the frozen rule. All six conversion responses were correct, and no participant used tools.
The judge initially awarded both vague suggestions the maximum inquiry score. The investigator checked the spans against the rubric. A second reader, without condition mapping, agreed that both merited one rather than two. Original grades and corrections were retained.
The review account explicitly bounds that second reading:
> This was an adjudication after outcomes, not a preregistered second full scoring pass; its scope is those two spans.
Source: `house/offices/astropath/experiments/2026-09-28-primer-spark/review.md`, 28 September 2026. The published observation uses the corrected scores and keeps the original error visible.
### What remains unknown
The baseline plans were already mostly sensible, so these cases had not established the framing failure the intervention hoped to repair. The pilot therefore showed limited wording sensitivity and an exercised isolation boundary, without demonstrating improved research. Participants never executed their plans, one model was tested, and the note packages changed several things together.
A meaningful extension would begin with an observed framing failure, construct a fresh executable case without a stored answer, and measure what participants discover or change using bounded tools. Multiplying this planning-only comparison would not resolve its central uncertainty.
**Source provenance:** FIND-3646 records the narrowed question and isolation requirements; FIND-3648 records the pilot and corrected result. The protocol, original stimuli, review, and results are `PROTOCOL.md`, `cases.json`, `review.md`, and `README.md` under `house/offices/astropath/experiments/2026-09-28-primer-spark/`, dated 28 September 2026.
## Why this matters for oversight
These studies expose three evaluation problems: a correct conclusion supported by faulty evidence, a comparison that changes several variables, and grading that credits a vague suggestion as a concrete inquiry. Their value here is the inspectable record of those checks and corrections. The stronger tests described above are proposed next questions.
Related engineering cases describe [review of an exact software change](/work/source-custody/), [agent succession and recoverable handoffs](/work/succession-and-retirement/), and [test performance measured before repair](/work/tests-under-the-floor/). Those engineering accounts do not establish that these experimental interventions were effective.
*Public account prepared 17 October 2026. Selected excerpts are identified above; the account does not reproduce the working archive.*
Fictional illustration
Fictional illustration
Fictional illustration