4 Evaluation Methodology

4.1 Evaluation Scope and Research Questions

We examine the research gap established in Section 2.6 through a reproducible case-study methodology, implemented here as a within-participant case study of APM and Spec-kit. To the best of our knowledge, this is among the first methodologies for evaluating frameworks of this class in use by developers.

Both frameworks belong to the class introduced in Section 2.5, and each has its own realisation of SDD described in Section 2.3.

The comparison treats each framework as a complete workflow, including how it carries specification, planning, and implementation and how it coordinates the agents involved. Four Research Questions (RQs) organise the primary measured comparison:

• RQ1 (Economic): How do APM and Spec-kit differ in the dollar cost of completing a demanding task, and is that difference consistent across models of differing capability?

• RQ2 (Process): How do APM and Spec-kit differ in the time the task takes to complete and in the form and frequency of the user’s recorded interactions with the frameworks’ agents?

• RQ3 (Functional): How do APM and Spec-kit differ in functional outcome, including how much of the task each completes and how they fail when they fall short?

• RQ4 (Perception): What are the participants’ self-reported experiences with each framework and how do they relate to the behaviour measured in their sessions?

4.2 Spec-kit, the Comparison Framework

Spec-kit is GitHub’s open-source spec-driven development toolkit [6]. Evaluating APM alongside Spec-kit compares two realisations of spec-driven development, contrasting single-agent and multi-agent coordination. In Spec-kit, one agent applies the discipline from end to end, producing a specification and a plan and then implementing them, all within one continuous conversation. That agent may still launch subagents through the underlying coding agent, just as APM’s agents can.

Spec-kit’s workflow is driven by a set of commands that the user triggers or the agent invokes on its own. Each is a pre-written prompt whose instructions walk the agent through one step of the workflow. The first is /speckit-constitution, which fixes the project’s governing principles in a file; these come either from the user or from the agent’s reading of the existing project, and the agent is to uphold them throughout. The core workflow then produces three successive artifacts:

1. /speckit-specify: directs the agent to produce a structured specification of requirements, user stories, and acceptance criteria from a feature description.

2. /speckit-plan: has it turn that specification into a technical implementation plan with architecture decisions, data models, and API contracts.

3. /speckit-tasks: instructs it to derive an executable task list from the plan, annotated with dependencies and sequencing constraints.

Finally, /speckit-implement has the agent work through the task list in sequence, while further commands support iterative refinement, cross-artifact consistency checking, and custom quality checklists along the way.

Across these steps the agent works largely on its own. It fills gaps in a brief with documented assumptions and puts a question to the user only where a choice is consequential or genuinely open, offering each as a short table of options with a recommendation.

4.3 Case-Study Conditions

The study was conducted from 5 May to 5 June 2026. Both frameworks operated through the Claude Code command-line interface, and participants used the Claude Opus 4.7 or Claude Sonnet 4.6 models. Pricing, model, and framework descriptions refer to this study window.

The study used version 1.0.1 of both the APM templates and the agentic-pm command-line tool [45], [42]. It used Spec-kit versions from v0.8.5, tagged on 4 May 2026, through v0.8.13, tagged on 21 May 2026 [46]. Because a fair comparison needs a fixed target, we audited that span release by release over the instructions that reach the agent and compose the framework’s workflow. The change across the eight release steps is minimal, and we therefore read the comparison as operating against one fixed version. Appendix A.1 reports the audit, and Section 6.7 treats the risk it leaves.

4.4 Case-Study Methodology

The case study employs a within-participant comparison across twelve university students and working professionals. Each participant attempts the same task twice, once per framework, holding individual skill, prior experience, and working style constant across both attempts. Each attempt has a three-hour limit.

Participants receive materials bundled as an open-source task package: a task brief, a portable testing environment, and a self-contained guide that walks them through setup and their task attempts. An optional agent helper skill accompanies the guide to automate or assist with setup; agents evaluated in the study sessions do not have this skill. The two subsections that follow detail the brief and the environment. Appendix A.2 lists all package materials.

4.4.1 Task Specification

The task is delivered to each participant as a product requirements document (PRD) that defines its objectives, requirements, and acceptance criteria. It builds on eclass-mcp-server [47], an existing open-source Python Model Context Protocol (MCP) [48] server that connects to Open eClass [49], the learning-management platform used across Greek academic institutions. The existing server exposes four MCP tools for authentication and course retrieval. Participants are required to add two tools that retrieve course and platform announcements. This requires investigating the Open eClass PHP codebase to determine how the platform exposes those announcements, then implementing the new retrieval behaviour in the Python server.

Participants are also required to build a standalone native C replica that exposes the four existing and the two new tools and matches the Python server’s input schemas, responses, and error behaviour. The Python server receives its MCP and JSON-RPC protocol machinery from MCP’s Python SDK; the C replica must implement that layer itself. Additional constraints from the PRD prohibit the use of external JSON-parsing libraries and require the C program to leak no memory across the supported tool-session surface under Valgrind. Together, the two deliverables give both frameworks a demanding end-to-end problem, spanning codebase investigation, application behaviour, and manual protocol implementation across the Python server and its native C replica.

4.4.2 Task Environment

Each participant works in a freshly created, dedicated Linux virtual machine. The participant’s workspace contains the PRD and an opening prompt alongside two pinned codebases: eclass-mcp-server, which they are required to modify, and Open eClass, which they can investigate as a reference. Installed in its own directory inside the same virtual machine, the portable testing environment runs a containerised instance of the same Open eClass release together with a mock single-sign-on service. The installed instance holds a deterministic mock dataset and is reached over localhost, letting participants develop in their workspace and validate their work against a live platform on the same virtual machine.

4.4.3 Sessions and Data

We call one attempt to complete the task with the assigned framework a study session. It begins with the first prompt sent to the agent and ends when the participant stops task work or reaches a three-hour cap. Across the cohort, this gives twenty-four study sessions. Between each pair the environment is reset to its initial state, so that no code, workspace modification, or assistant context from the first session carries into the second.

Participants were assigned one of the two models across both study sessions, forming two six-participant model groups. Framework order was assigned separately, placing APM in the first or the second session. Counterbalancing the framework order lets us investigate whether improvement from growing task familiarity can be distinguished from the framework difference. Together, the two assignments formed four model-and-order groups of three participants each.¹

A post-session procedure has the participants package their solution patch written against the pinned Python codebase and their Claude Code transcripts for every agent conversation they opened. They then submit that package and answer a short questionnaire.

4.5 Metrics

This section defines the study’s metrics, which combine automated computation with participant self-reporting, and the terminology used with them. Automated tooling built for the study extracts the objective outcomes deterministically: a transcript parser processes the participants’ Claude Code conversation logs to compute the economic and process metrics, while a test-based grader scores the participant’s code patch against the functional assertions. A post-session questionnaire supplies the perception metrics. The parser and grader belong to the open-source infrastructure of Appendix A.2, and Appendix A.3 details the questionnaire items.

Every transcript-derived measurement covers the three-hour attempt window. Records timestamped more than three hours after a session’s first user message are excluded before any measurement is computed. Any excluded tail is flagged for review before the session’s measurements are accepted.

Each transcript-derived measurement is computed at one or more scopes. A measurement scope is the level of the recorded workflow over which a value is aggregated. The session scope covers the whole study session. The phase scope gives separate values for planning and implementation. The role scope gives APM-only values for each agent role, since Spec-kit has no roles to separate. A measurement breakdown further separates a value at one scope into named components.

The phase and role scopes follow the frameworks’ own workflow as recorded in the transcript. Spec-kit’s single conversation turns to implementation when its workflow reaches /speckit-implement, whether the participant types it or the agent invokes it on its own. APM divides by role, with its Planner conversation forming planning and its Manager and Worker conversations forming implementation.

Within a phase, each measurement is attributed as the work happens: cost, tokens, and model-request counts follow the individual record, and a turn that crosses from planning into implementation divides its time at the crossing. The counts of user messages instead follow the turn, attributed to the phase in which the participant opened it.

4.5.1 Economic

The economic measurement is the dollar-equivalent cost, computed by pricing the recorded token usage of each billed model request against Anthropic’s published per-token rate card in effect during the study window, recorded in Appendix A.4. Cost is measured at session and phase scope, and for APM at role scope. Its session-scope value is also broken down by token category to locate where the spending accrues.

4.5.2 Process

The process measures describe the session’s temporal shape and the user’s interaction with the agent.

The temporal shape of the session is captured by the session duration and the two measures that decompose it, the turn time and the inter-turn time, all reported at session and phase scope only:

• Session duration is the wall-clock span from the prompt that opens the first turn to the latest retained timestamped transcript record within the three-hour attempt window.

• Turn time is the wall-clock union of the turn intervals, each spanning from its opening prompt to its last retained assistant record. Its breakdown separates the time covered by exactly one interval (solo) from the time covered by two or more intervals (concurrent). For APM this shows whether its agents’ turns overlap or proceed serially.

• Inter-turn time is the part of the session duration covered by no turn interval. It is not read as a measure of participant activity, since a gap may be spent reading, thinking, or away from the task.

The interaction shape is captured through two forms of user input:

• A turn prompt is a message submitted while the agent is idle, opening a new turn that lasts until the agent’s idle return.

• A steer is an interjected message submitted while the agent is still working. In Claude Code these are enqueued until the return of the agent’s next tool call, which delivers the message to the agent mid-turn unless the participant has removed it from the queue first.

Turn prompts are counted, and each falls into one of three content buckets:

• Protocol for a prompt that invokes an APM or Spec-kit framework command.

• Short for a non-command prompt below 50 characters.

• Standard for a non-command prompt of 50 characters or more.

The turn-prompt breakdown is described by the bucket-share distribution, which gives each bucket’s share of the prompts, and the magnitude per bucket, their mean character count within each bucket. Together with the turn-prompt count, these measures describe how often the user opened turns, the form those prompts took, and the amount of text recorded in each form.

Unlike the temporal measures, the turn-prompt measures are also reported at role scope for APM, attributing each prompt to the agent the participant was addressing.

Steers are reported at session scope only, as two counts: those delivered to the agent and those not delivered.

4.5.3 Functional

The functional dimension is measured by ninety atomic functional tests applied to each submission, each testing one expected behaviour and carrying one or two points towards a hundred-point total. The grader tests the resulting behaviour against the MCP surface required by the task. It observes only the running Python server’s and the compiled C binary’s responses, never the participant’s source, and is therefore indifferent to how the solution is built. It runs each submission against an environment equivalent to the one participants used in their sessions, with its own seed data for finer-grained checks. Appendix A.5 specifies the rubric’s weighting and its partial-credit rule.

Each check returns one of three verdicts. A passed check matches an accepted form and earns its full points. A partial check matches no accepted form exactly but comes close to one, measured by Levenshtein edit distance [50] and computed with the Wagner-Fischer algorithm [51]; it earns points scaled by that similarity and is flagged for review. A failed check earns zero.

The total score sums the points earned across the rubric. Its breakdown separates a strict part, the full points earned by passed checks, from a partial part, the fractional credit earned by partial checks. A task-section breakdown divides the 100 points between the Python extension, worth 30, and the heavier C replica, worth 70. The resulting per-section scores locate where each framework’s work lands.

4.5.4 Perception

The perception measures come from the post-session questionnaire, the self-reported counterpart to the measured behaviour. Its core is eight rating items on a one-to-five ordinal scale, each answered once per session and paired by framework. The items cover the participant’s prior familiarity with the framework, confidence in the submitted solution, the framework’s perceived effectiveness, the session’s mental demand and frustration, the overall effort invested, and the effort spent checking and steering the agent. The items differ in polarity, a higher rating being the better outcome for some, such as confidence and effectiveness, and the worse for others, such as mental demand, frustration, and the three effort items.

Beyond the rating items, the questionnaire asks a comparison question after the second session: whether the task felt easier or harder than with the first framework. It also records the participant’s own completion state against the three-hour cap. It closes with two free-text prompts, read as the participants’ own accounts, on what stood out and any closing remarks.

4.6 Analysis Plan

Our findings lead with the measured values for APM and Spec-kit. Where a measure supports a within-participant comparison, it is followed by each participant’s APM-minus-Spec-kit difference and summaries of its sign, typical size, and uncertainty. Model-group breakdowns present the respective group’s paired differences separately and compare them descriptively.

Each participant i contributes the paired difference $d_i = x_i^{\text{APM}} - x_i^{\text{SK}}$, where SK denotes Spec-kit. The cohort analysis uses these twelve differences, while each model-group breakdown uses the group’s six. Since the Opus and Sonnet differences come from different participants, a contrast between the groups carries their differences in skill and working style alongside the assigned model.

The number of positive against negative differences gives the sign count, following the convention of the paired sign test [52]. It reports the directional consistency of the difference across participants without assuming symmetry or a specific distribution shape, with zero-difference pairs contributing to neither count.

The typical difference is reported through the median, the Hodges-Lehmann (HL) estimate [53], and the mean. The Hodges-Lehmann estimate is the median of the averages of every pair of differences,

$$\hat{\theta} = \operatorname{median}\left\{ \frac{d_i + d_j}{2} \;:\; 1 \le i \le j \le n \right\}.$$

It reflects more of the observed differences than the median while being less affected by a few large differences than the mean. Read together, the three show the centre of the paired differences and whether asymmetry changes that reading.

A confidence interval (CI) based on the exact Wilcoxon signed-rank procedure [54], [55], [52] captures the uncertainty surrounding the typical difference. Rather than approximating the distribution, the procedure computes it directly, fitting a small sample where normality cannot be assumed. The test statistic ranks the absolute paired differences and sums the ranks of the positive differences,

$$W^{+} = \sum_{i \,:\, d_i > 0} \operatorname{rank}\!\left( |d_i| \right).$$

Under the null hypothesis, each rank is equally likely to be positive or negative. Comparing $W^+$ against this distribution determines which values of the typical difference are compatible with the data; those values, expressed in the original units of the paired differences, form the confidence interval. Its exact coverage is discrete: 95.8% across the twelve cohort pairs and 96.9% across each model group’s six pairs. At six pairs, its endpoints are the smallest and largest observed differences, so it excludes zero only when all six differences share the same sign.

The dollar-equivalent cost, temporal process measures, turn-prompt count, and total score are each presented first through the two frameworks’ measured values and then through their within-participant paired difference.

Measurements at phase or role scope, and their component breakdowns, are compared within participant when they give one directly comparable value for each framework. This applies to phase-scope cost, temporal measures, and turn-prompt count, as well as cost by token category and score by task section. Role-scope values and turn-prompt bucket distributions are presented descriptively through their median, Hodges-Lehmann estimate, and mean, without a paired difference or confidence interval. The solo and concurrent breakdown of turn time is shown per participant because concurrent time applies only to APM.

When an irregularity in one participant’s data materially shifts a summary, the affected table row identifies it. Alongside these summaries, participant-level values are shown when the individual pattern is needed to read the result or when the events are too sparse for an aggregate summary.

The perception ratings are ordinal, so each item is summarised by order-based statistics. These are its sign count, read with the item’s polarity, and each framework’s full distribution across the five points. Responses to the comparison question and the completion state are presented descriptively as categorical distributions.

The interpretation in Chapter 6 draws further on supplementary descriptive counts where they support a specific reading. These counts are supporting evidence and not additional measures of the framework comparison, and their presentation follows the question they help interpret.

These statistics are computed, and the figures and table data generated, deterministically by the analysis scripts listed in Appendix A.2.

---

Footnotes

Footnotes

1. One participant departed from their assigned order in practice: a tooling failure ended their first Spec-kit attempt before any task work was completed, they ran APM the following day, and then repeated Spec-kit. Analyses by order therefore use realised framework order. The realised APM-first and APM-second groups contain seven and five participants. Within them, the two realised Opus model-and-order groups contain three participants each, while the two realised Sonnet model-and-order groups contain four and two participants. All tables grouped by realised framework order in Appendix B.1, Appendix B.2, Appendix B.3, and Appendix B.4 use this classification. ↩