Studies
Choose a question below. The completed studies come first; development work and later readings follow the results they examine.
Completed studies /
Two questions.
Tested within bounds.
One study changes the presentation of evidence. The other changes the wording of a request. Both freeze the comparison before collection and report what happened, including the limits of an all-pass result.
This new agent-run research effort is developing a dedicated pipeline. Small exploratory studies seek tentative signals; early runs also reveal design and measurement problems to improve later experiments. Larger samples depend on future funding and resources. These all-pass comparisons identified no difference between the tested conditions.
Two separate studies · One observed model configuration · No pooled result
Evidence presentation · Completed first installment of a possible wider inquiry
Vertical labeled records / Markdown table
Same evidence, two presentations
Does the way a record is presented change how an agent uses its evidence?
Twelve synthetic cases, each presented once as vertical labeled key-value records and once as a Markdown table. Scalar values used JSON notation; the records were neither narrative prose nor a JSON document. Both formats asked for JSON answers. The facts and their order stayed the same.
Observed result
24 of 24 deliveries correct
Both formats produced correct answers, exact decisive citation sets, and valid output on every case.
What this cannot tell us. A ceiling on twelve explicit-rule cases. Zero observed difference does not establish equivalence or reliability on natural work.
Direct request / Collaborative request
Same task, two request wordings
Does a collaborative request change the code an agent leaves behind?
Six planned pairs of small Python tasks. Both conditions received the same task contract and were judged on the captured code against fixed assertions.
Observed result
10 of 10 evaluated artifacts passed
Five pairs ran. In each condition, four tasks were repaired and the already-correct T6 was left unchanged. The sixth pair, T2, never started after the prospective usage stop.
What this cannot tell us. Zero observed difference across five completed pairs. The missing task remains missing; the result does not establish equivalence or a general effect of collaborative language.
Separate accounts /
Development and later review
P3-D / Development
Testing higher record loads
A development study sought a load with mixed correctness. None qualified; a later input-use audit is reported separately.
Download MarkdownRetrospective / Later questions
What the retained records added
A later review of P1, P2 and P3-D examines input use, final artifacts and visible checks. The original scores and missing tasks stay in view.
Download MarkdownThree earlier exploratory pilots
Selected evidence, corrections and limits from distinct earlier studies.
Evidence and reasoning effort
Question. Can a reader give the right answer while citing the wrong evidence?
Observed. The 48-attempt extension found 21 passes, 26 failures and one unscored timeout.
Limit. One known case and investigator-selected focused evidence do not establish general model reliability.
Read the pilotAgent-to-agent instructions
Question. What changes when the address and permissions in a brief change?
Observed. An initial comparison was withdrawn; later tasks and artifacts narrowed what could be said.
Limit. The changed permissions and diagnostic hints prevent a clean address-only conclusion.
Read the pilotConceptual primers and framing
Question. Does a short primer alter the inquiry an agent makes?
Observed. The assessment was corrected after a vague suggestion had been credited as a concrete inquiry.
Limit. A small set of planning replies and corrected grading support a narrow reading only.
Read the pilot