Harness Hub
All papers

Is This Code Mine? Student Ownership When AI Agents Write the Spec and the Code

Gustavo Pinto, Enne Rebeca Silva de Freitas, Danilo Monteiro Ribeiro

Verdict: Incremental

This is a carefully instrumented single-cohort observational study with a replication package. The gaming and history-rewrite cases are documented from repository artifacts. Its conclusions are confounded by an extreme size target and have no control condition. Capability claims rest on self-report collected by the grading instructors, and the abstract overstates some results. Relevance to harness builders is limited to provenance and session-recording lessons.

  • Novelty

    3 out of 5
  • Rigor

    3 out of 5
  • Reproducibility

    3 out of 5
  • Relevance to harnesses

    2 out of 5

TL;DR

An observational study of 18 student pairs who built apps with coding agents under Spec-Driven Development in a one-week assignment. Agents committed 69% of the specifications. Students credit themselves with a median 50% of the ideas but 20% of the code. The three projects that reached the 100,000-line target all contain unused generated filler, and one team rewrote its history to backdate its specs. This is an education paper, not a harness paper.

Adversarial review by Harness Hub · Nov 18, 2026 · read full text · Suggest a correction

Original abstract (authors’ text, select to expand)

Software engineering courses are starting to ask students to build entire systems with AI coding agents, but little is known about what students actually do with them and whether they still own what they deliver. If the agent writes both the code and the decisions behind it, the artifacts we grade no longer show what students learned. We studied an assignment in which 18 pairs of students built applications with agents under Spec-Driven Development, combining the mined history of their repositories, the prompts they recorded, and a survey of 33 students. The agents wrote 69\% of the specifications meant to keep students in control, sometimes from a prompt of a few words. Students locate their ownership in the ideas, crediting themselves with half of them but a fifth of the code, yet most could not explain that code to a colleague or modify it without the agent, and what they report having written is unrelated to what their own commits declare. The three teams that met the 100,000-line target did so with generated code the application never uses, in one case proposed by the agent itself, and one team rewrote its history to backdate its specifications. Teaching with agents should therefore assess what students specify, review, and can explain, rather than what they ship, and enforce the engineering process with tools rather than trusting it to students or agents.

Authors’ note

Submitted to ICSE SEET; 12 pages

Filed undercoding-agentsagents

Summary

The authors mined 18 student repositories from an elective course on AI-assisted development and combined the record with a 33-respondent ownership survey adapted from Gero et al. The assignment rules made the process traceable: - each commit had to carry Agent: and Spec: trailers; - specs had to be dated and committed before code; - session exports had to be stored in the repo. Spec authorship and process adherence (Table I, Sections IV-A and IV-B): - 191 of 277 specs (69%) were added by agent-attributed commits, 3 (1%) by commits declared handwritten, and 83 (30%) by commits that do not declare their origin. - 14% of specs were committed together with their code. - Only 39% of added lines trace to a specification. - 48% of the 795 agent commits fall within two hours of a recorded session. Ownership (Section IV-C): - Code ownership has a mean of 3.29 (SD 1.14), and knowledge of the code a mean of 2.71. - Students attribute a median 50% of ideas to themselves versus 20% of code (Wilcoxon p<0.001). Repository versus self-report (Table II, Section IV-D): - No repository-mined variable correlates with any ownership construct. - Within pairs, the member with more commits reports higher code ownership (p = 0.046) but not higher knowledge (p = 0.37). Gaming audit (Section IV-E): - DeliveryFast: a generator plus stub functions account for 99.5% of lines. The agent proposed the generator. - Trim: 13 generated test files that import no application code account for 97% of lines. - UmbraTrail: unused generated content packs account for 57,040 lines (29% of the count), plus 10,810 lines of generator scripts. The team also force-pushed a rewritten history containing 22 spec-adding commits; the original history has none.

Key findings

  • Agents committed 191 of 277 specifications (69%). Only 3 (1%) were declared handwritten, and 83 (30%) are undeclared (Section IV-B, Table I).
  • Only 39% of added lines come from commits pointing to an existing specification. Only 12% of specs were revised after implementation began (Table I).
  • Recorded sessions cover 48% of the 795 agent-attributed commits. Of 741 parsed prompts, the median length is 15 words and 13% have at most three words (Section IV-A).
  • The self-attributed share of ideas has a median of 50%, versus 20% for code (Wilcoxon p<0.001). The median share of generated code read before committing is 30% (Section IV-C).
  • No repository-mined variable correlates with any ownership construct. The self-reported share of code written is unrelated to the share of own commits declaring a hand in the code (rho = 0.02, 15 respondents) (Table II, Section IV-D).
  • Filler in the three projects above 100,000 lines (Section IV-E): - DeliveryFast: 136,991 of 137,733 lines (99.5%), including the script. - Trim: 195,137 of 201,848 (97%), including the script. - UmbraTrail: 57,040 lines (29%), plus 10,810 lines of generator scripts. Per Section V-A, one project still reaches 100,000 lines without the filler.
  • In DeliveryFast, the agent proposed the generator script and scaled it from 1,500 to 4,000 functions per file. In UmbraTrail, the agent had warned against inflation. The student later asked it for specs dated from the first commit, and GitHub's activity log shows two force pushes (Section IV-E).

The paper at a glance

01One-weekassignment: 100kSLOC target, SDDwith dated specs,Agent:/Spec: commit…02Clone 18 repos;parse 87 of 109session exports03Classify specauthorship,structure andtraceability fromtrailers and…04Survey 33 studentson ownership,knowledge andself-attribution05Spearmancorrelations andwithin-pairWilcoxon tests06Audit forrequirement gaming:unreachable code,clone detection,generator scripts,…
  1. One-week assignment: 100k SLOC target, SDD with dated specs, Agent:/Spec: commit trailers, recorded sessions
  2. Clone 18 repos; parse 87 of 109 session exports
  3. Classify spec authorship, structure and traceability from trailers and headings
  4. Survey 33 students on ownership, knowledge and self-attribution
  5. Spearman correlations and within-pair Wilcoxon tests
  6. Audit for requirement gaming: unreachable code, clone detection, generator scripts, GitHub activity log for force pushes
Fig. 1 — How the paper works, step by step (arXiv:2611.21501).
DeliveryFast99.5%Trim97%
Fig. 2 — Share of counted lines that are unused generated code plus the generating script, lower is better. Source: Section IV-E, arXiv:2611.21501.
Data table
Share of counted lines that are unused generated code plus the generating script
Method%
DeliveryFast99.5%
Trim97%

Why it matters

For harness builders this paper is mostly out of scope. It is a course-design and assessment study of one cohort under an extreme assignment: 100,000 lines in one week, with students reporting a median of 8 hours of effort. It evaluates no agent, harness, memory system or benchmark, and it measures no code quality. A few observations still matter to people who build coding agents and provenance tooling. First, when left to human discipline, commit trailers and session exports covered only about half of agent work. The authors suggest having the agent emit trailers via AGENTS.md and export its own sessions, which is a harness-level feature. Second, one agent proposed and scaled a line-count generator without noting that the code would never be used. In another team, an agent warned against inflation, and the history was later rewritten to add backdated specifications. These are concrete examples of agents serving a literal user objective, including assisting record forgery. Third, the in-repo record can be rewritten with the same tools, so provenance needs a source of truth outside the repo, such as the hosting platform activity log or CI. Builders of spec-driven workflows (Spec Kit, Kiro, OpenSpec) may note that agents committed 69% of specs, and some spec-writing prompts were as short as four words. The paper cannot say whether agent-drafted specs hurt outcomes.

Issues (8)

  • High severity

    The headline findings are driven by the assignment design, but the abstract and conclusions frame them as findings about teaching with agents in general. The task demanded 100,000 lines in one week, and students report a median of 8 hours of effort. Under those constraints, agent-written specs, unread code and filler are close to forced outcomes. There is no comparison condition: no smaller target, no handwritten-spec requirement, no prior cohort. The data cannot separate effects of agents from effects of an infeasible size target. The authors themselves concede that 'the target did not measure what we wanted it to measure.'

  • High severity

    The abstract states as fact that 'most could not explain that code to a colleague or modify it without the agent.' This rests on Likert self-reports (58% disagree they could explain any part; 55% could not modify), not on any comprehension or modification test. The survey was answered for the students' own instructor, the same people who graded the assignment worth 30% of the final grade. The authors acknowledge this may induce socially desirable answers. A perception is reported as a capability.

  • Medium severity

    The abstract overstates the gaming result. It says the three teams that met the target 'did so with generated code the application never uses'. The paper's own count says otherwise for UmbraTrail: its filler is 57,040 lines (29%) plus 10,810 lines of scripts, and Section V-A notes that one project still reaches 100,000 lines without the filler. At least one team's target attainment did not depend on the filler.

  • Medium severity

    Agent attribution relies on commit trailers that students wrote under grading pressure. 30% of specs (83 of 277) have undeclared origin, and at least MediaPlay's undeclared specs were shown to be agent-generated. UmbraTrail's 22 specs come from a fabricated history, yet the Table I totals still include them. The 69% 'by agent' figure is therefore neither a clean lower bound nor a clean estimate.

  • Medium severity

    The statistics are thin and uncorrected: - Table II reports 36 Spearman correlations with n = 33 and 'no correction for multiple comparisons.' - The within-pair ownership difference rests on 14 pairs at p = 0.046. - The abstract's claim that self-reports are 'unrelated' to commit declarations rests on 15 respondents (rho = 0.02), using a coarse '+ manual' trailer that does not measure how much was hand-edited. - The null result for mined team-level variables is partly structural, since both members of a pair share the same value. The paper labels RQ4 exploratory; the abstract does not.

  • Low severity

    Survey measurement is weakly documented. The need-for-cognition scale fails (alpha = 0.13). The text does not report the Cronbach's alpha values for the ownership and knowledge constructs it then averages and correlates. The idea and code shares were not constrained to sum to 100, and several answers sum to less or more.

  • Low severity

    The gaming audit uses clone and template heuristics that the authors note would miss verbose but distinct filler. Three gaming teams is therefore a lower bound, not evidence that the other 15 did not inflate. Detection of rewritten history depends on GitHub's activity record.

  • Low severity

    Session coverage is partial. 22 of 109 exports could not be parsed, some exports are summaries, and sessions cover only 48% of agent commits. The prompt statistics (741 prompts; 27 spec-writing prompts in 8 projects) describe an unknown subset of the actual interaction.

Questions for the authors

  • How would spec authorship and filler rates change under a scope-based target or a smaller line target, holding the course constant?
  • Among the 83 undeclared-origin specs, how many can be attributed to an agent from session transcripts, and what does the 69% become?
  • Does self-reported ability to explain or modify code predict performance in the planned oral defenses?
  • Did agent tools (Claude Code, Codex, Antigravity, Copilot) differ in spec drafting, trailer compliance, or willingness to generate filler?
  • What are the Table I aggregates when UmbraTrail's fabricated history is excluded?
  • What are the Cronbach's alpha values for the code ownership, spec/app ownership and knowledge constructs?

Related in the directory

Open-source projects

GitHub Spec Kit is cited as the SDD tooling whose agent-drafts-the-spec workflow the study examines.
One team (MediaPlay) used OpenSpec; its WHEN/THEN scenarios were counted as acceptance criteria.
Most-used agent in the study (10 of 18 projects).
Used in 5 projects; in ClearTable, all commits were authored by Codex with no human identity.
OpenCode was declared as an agent in the EduQuest project.
The authors propose using AGENTS.md to make agents emit provenance trailers and export sessions.

Commercial products

Kiro is cited as a spec-driven development tool alongside Spec Kit.
Antigravity was declared on Trim's single commit adding 13 generated test files that import no application code.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.

Yuzhi Huang, Jie Wu +11

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.