A Evaluation Appendix
A.1 Spec-kit Release Audit
The audit installs each release of the span as a project and diffs each against the next, since Spec-kit renders its templates as it installs them. The compared files are the commands of Section 4.2 together with file templates, scripts, and a context file they instruct the agent to use, each compared byte for byte.
What a change amounts to is a matter of interpretation, since one line can carry a great deal of instruction to an agent. The audit therefore ships a unified diff per release step, alongside a manifest hashing every installed file at every release that ties each diff to the release it comes from.
Six of the eight steps leave the files identical. Of the other two, v0.8.9 cleans up whitespace in three templates, leaving v0.8.6 as the one release to alter what the agent is told, adding a line that directs the implementing agent to reread the project constitution where one exists. Appendix A.2 locates the script and its output, where each change can be read in full.
A.2 Software and Data Availability
The study’s software is openly available, and its dataset is published encrypted, in the following repositories:
-
The task and participant-materials repository, holding the task brief, the participant guide, and the portable testing environment, at
https://github.com/sdi2200262/bsc-apm-case-study-task; -
The evaluation infrastructure, holding the transcript parser, the grader with its scoring rubric, and the grading environment, at
https://github.com/sdi2200262/bsc-apm-case-study-infra; -
The dataset of participant submissions, from which our measurements are computed, as one encrypted archive whose key is available to reviewers on request, at
https://github.com/sdi2200262/bsc-apm-case-study-data; -
The analysis scripts that compute the reported statistics and produce the figures and table data from the parser and grader outputs, in the source repository under
analysis/, at
https://github.com/sdi2200262/bsc-apm-thesis; -
The Spec-kit release audit of Appendix A.1, holding its script together with the diffs and per-file hashes it produced, in the same repository under
spec-kit-audit/.
A.3 Questionnaire Instrument
The second-session question, “Compared to the framework you used in your first session, completing the task with this framework felt…”, offers five levels:
-
much harder
-
harder
-
about the same
-
easier
-
much easier
The completion-state question, “Completion state”, offers four options:
-
Completed the task comfortably within the three-hour cap
-
Completed the task near the three-hour cap
-
Hit the three-hour cap without completing
-
Stopped early without completing
The two free-text prompts are:
-
“What stood out about working with this framework on this task?”
-
“Anything else you want to add about this session?”
The eight rating items and the wording at each end of their scale are given in Table A.1.
| Question | 1 | 5 |
|---|---|---|
| How familiar were you with this framework before this study? | Never used it | Use it regularly |
| How confident are you in your submitted solution? | Very low | Very high |
| How effective was the framework in helping you complete the task? | Not effective | Very effective |
| How mentally demanding was the session? | Very low | Very high |
| How frustrated did you feel during the session? | Not at all | Extremely |
| How much effort did you have to invest? | Very low | Very high |
| How much effort went into checking what the AI in this framework produced? | Very low | Very high |
| How much effort went into instructing and steering the AI in this framework? | Very low | Very high |
A.4 Rate Card
The dollar-equivalent cost prices each request’s tokens at the rates in Table A.2. The rates cover five token categories: fresh input and output, reads from the prompt cache, and writes to the cache at its five-minute and one-hour retention periods.
| Model | Input | Output | Cache read | Cache write (5 min) | Cache write (1 h) |
|---|---|---|---|---|---|
| Opus 4.7 | 5.00 | 25.00 | 0.50 | 6.25 | 10.00 |
| Sonnet 4.6 | 3.00 | 15.00 | 0.30 | 3.75 | 6.00 |
| Haiku 4.5 | 1.00 | 5.00 | 0.10 | 1.25 | 2.00 |
Subagents launched within study sessions and some background operations were at times routed to Haiku. Some sessions therefore include a small share of Haiku tokens alongside their assigned model. The evaluation infrastructure listed in Appendix A.2 holds the rates in rates.yaml.
A.5 Scoring Rubric
The rubric’s ninety checks divide into twenty-seven over the Python server, worth thirty points, and sixty-three over the C replica, worth seventy. Ten checks carry two points and the remaining eighty carry one.
A string-matched check compares the observed response against the responses the rubric accepts. An exact match earns the check’s full points. Otherwise the grader measures how near the response came: it takes the edit distance to the nearest accepted response, divides it by the length of the longer of the two, and subtracts the result from one. At a ratio of 0.45 or above the check earns its points scaled by that ratio; below it the check earns zero. Every partial check is flagged for review before the score is accepted.
Structural checks and memory checks carry no similarity. They pass or fail outright. Twenty-six of the C memory checks are each paired with a functional check and are awarded only where that check passes. A clean memory run of a non-functional program earns no points.