01 · The mismatch
Capstone Project 2 · Final Report · Sunway University

Your prompt file is an ML artefact.
Nobody is tracing it to a requirement.

In 2026 a consumer app's "ML" is half trained models and half natural-language instruction files. Requirements traceability research only knows how to follow the first half. This project built — and then deliberately broke — a framework that follows both.

Justin Yong Wenn Weii · 18119677 BSc Software Engineering (Hons) Supervisor: Prof. Lau Sian Lun 3 August 2026
Scroll

The mismatch

Requirements engineering assumes software that behaves the same way twice.

A user story says what the system shall do. That grammar was built for deterministic code, where a requirement is either met or it isn't. Machine learning does not offer that courtesy.

McKinsey's State of AI reports put organisational AI adoption at 60% in 2023 and 90% in 2025. Consumer teams are shipping ML features inside ordinary agile sprints — with no more time for requirements analysis than before, and specification techniques that never anticipated a probabilistic answer.

Traditional specification
"The system shall display
the user's owed balance."

Binary pass/fail. Exceptions are enumerable. Verified by functional testing.

BECOMES
ML specification
"The system shall categorise expenses with more than 80% precision for each category on a held-out test set of 30 receipts."

Confidence levels, not booleans. Degradation at a threshold. Verified statistically, or against a fixed eval set.


The 2026 split

And then the ML artefact stopped being a model.

Since ChatGPT in 2022, consumer teams increasingly compose features on hosted foundation models rather than training their own. The result is a stack that spans two paradigms with different artefacts, different versioning surfaces, and different ways of failing.

Existing RE4AI traceability work — PROV-ML, ML-Schema, MSR4ML — assumes the traced artefact is the output of a training procedure. None of them has a representation for a natural-language specification file. That leaves a growing share of shipped ML behaviour outside any traceability chain at all.

Paradigm A

Trained model

Behaviour is fixed once the weights exist.

  • Artefact · model checkpoint (.pkl)
  • Versioned by · MLflow run + DVC dataset
  • Validated by · held-out test metrics
  • Changes when · you retrain it
Paradigm B

LLM-agent

Behaviour is governed by a text file — and by someone else's model.

  • Artefact · versioned SKILL.md
  • Versioned by · a Git commit
  • Validated by · an eval harness on a fixed set
  • Changes when · the provider updates the model

That last row is the quiet one. A trained model can't change under you. An LLM-agent component can shift behaviour with no commit, no run, and no diff — which is why the framework snapshots the provider model identity as part of the evidence.


Design Science Research

Six gaps in the literature, turned into nine requirements the framework itself has to pass.

The project follows the DSRM process model of Peffers et al.: identify the problem, define objectives, build the artefact, demonstrate it, evaluate it. The point of the derivation below is that the framework's own success criteria were written before it was built, from documented gaps rather than assumed ones.

G1No requirements-first documentation — Model Cards are post-hoc
DR1 · DR2Prospective templates with acceptance criteria written up front
G2Consumer ML context underexplored; RE4AI targets safety-critical
DR3 · DR4Agile-compatible; tolerant of error rates and iteration
G3MLOps tooling disconnected from RE semantics
DR5 · DR6Tagging conventions; extend existing artefacts, invent none
G4No standard way to specify probabilistic outputs
DR7Confidence thresholds and uncertainty handling
G5Comprehensive Model Cards are rare because of overhead
DR8Minimal effort beyond MLOps and Git practice already in use
G6No framework treats prompt files as first-class versioned objects
DR9One requirement ID, traceable to artefacts of either paradigm

Each DR carries a testable criterion and a verification method — template inspection, convention inspection, process-model review, or overhead assessment. A DR violation counts as framework inadequacy, not as a finding to be explained away.


The artefact

Four components. Two of them are documents you already half-write.

The framework deliberately extends existing practice rather than introducing a new tool — new tooling imposes adoption barriers, which is exactly the overhead that kills documentation in the first place.

1 · Requirements specification approach

A paradigm-agnostic way to state ML requirements with probabilistic thresholds. The grammar of the requirement doesn't change based on whether a classifier or an LLM ends up implementing it. DR2 DR4 DR7

2 · Traceability conventions

Requirement IDs (REIDs) tagged into MLflow runs, DVC tags, Git commit messages and eval-harness log entries. Bidirectional navigation falls out of the naming. DR5 DR6 DR9

3 · RAMC — Requirements-Aware Model Card

The standard Model Card, with four new sections that link it back to the requirements it exists to satisfy. For trained-model components. DR1 DR3 DR8

4 · RASC — Requirements-Aware Skill Card ◆ novel

The same four sections, applied to a versioned SKILL.md instead of a model checkpoint. This is the project's original contribution. DR1 DR3 DR8 DR9

The four sections that turn documentation into a test
Sections 1–9 of a Model Card are unchanged. Sections 10–13 are new, and identical in name and purpose across both templates.
RAMC · trained model
1 Model Details
2 Intended Use
3 – 8 · standard Model Card sections
9 Caveats & Recommendations
10 Linked Requirements REIDs
11 Acceptance Criteria MAPE, precision
12 Validation Evidence MLflow run, DVC ver.
13 Uncertainty Communication
RASC · LLM-agent
1 Hosted model identity + snapshot
2 Intended Use of the skill
3 – 8 · adapted standard sections
9 Caveats & Recommendations
10 Linked Requirements REIDs
11 Acceptance Criteria agreement, refusal rate
12 Validation Evidence harness run, commit SHA
13 Uncertainty Communication

Structural parallelism is a deliberate trade. A practitioner who knows Model Cards can adopt RASC with almost no new learning — but "RAMC" and "RASC" differ by one character, and the evaluation later paid for that in navigation time.


Traceability conventions

The whole mechanism is a naming discipline.

No new tool, no plugin, no database. A requirement identifier is written into the artefacts each tool already produces, so the chain is queryable in both directions with what a team is running anyway.

# Trained model — MLflow run tags and registry description
mlflow.set_tag("req_ids", "REQ-FCT-01")
description="Satisfies REQ-CAT-01. RAMC: docs/ramc-classifier.md"

# Dataset version — DVC via annotated Git tags
git tag -a "data-v1.1" -m "REQ-OCR-01: Added receipts"

# LLM-agent — a commit message on the instruction file
git commit -m "[REQ-CAT-01] Add Beverage / Food disambiguation rule"

# Eval harness — one JSONL entry per criterion run
{ req_id: "REQ-CAT-01", skill_md_sha: "c0fc2aa",
  provider_model: "moonshot-v1-8k", agreement: 0.926,
  test_set_sha: "9574b207…" }

In the ShareLah repository this produced a main branch where 29 of 71 commits carry a REID prefix, and the skill files advanced through 13 REID-tagged revisions. One commit message — 500a7a3 "[REQ-OCR-01] Updated OCR skill v1.7 – strengthen non-MYR pre-condition gate after eval test" — records the originating requirement, the artefact version produced, and the evaluation finding that motivated it, in a single line.


Walkthrough

One requirement, four hops, no help.

On 29 July 2026 an external evaluator (E01) — a second-year CS undergraduate with no prior exposure to the repository — was asked to traverse the chain in both directions. The only briefing given was what MLflow is for.

REQ-CAT-01 requirement · from user survey rasc-categorisation.md RASC · §10 Linked Requirements §11 Acceptance Criteria agreement ≥ 0.80 per category results.jsonl cat-2026-07-24-r1 SKILL.md @ c0fc2aa produces the behaviour TASK A · FORWARD · 4 / 4 · 03:25 MLflow c2a3bc33… req_ids = REQ-FCT-01 ramc-forecast.md RAMC · LightGBM checkpoint REQ-FCT-01 originating requirement TASK B · BACKWARD · 3 / 3 · 02:10 Same REID vocabulary. Same four sections. Different paradigm at the far end.
Step A1 · 01:22 · pass

Start from a requirement.

E01 found docs/ quickly, then opened the OCR card before reaching the categorisation one. RAMC and RASC differ by a single character — the first empirical cost of the parallelism decision.

Step A2 · 00:43 · pass

Find the criteria it must satisfy.

He landed on §7 (Agreement Metrics Definition) before §11. Both show numbers, and inheriting Mitchell et al.'s numbering puts the requirements-bearing content at the end of a thirteen-section document.

Step A3 · 00:30 · pass

Find the evidence the criteria were run against.

One JSONL entry, carrying the requirement ID, the test-set SHA, the provider model snapshot, and the recorded metric.

Step A4 · 00:50 · pass

Reach the artefact that actually produces the behaviour.

A commit on a markdown file. This is the hop no existing traceability framework can make.

Task B · 02:10 · 3/3

Now run it backwards, through the other paradigm.

From an MLflow run's req_ids tag, to the RAMC, to the originating requirement — same vocabulary, same evaluator, same session.

Result

7 / 7 in 5 minutes 35 seconds of active navigation.

Task totals aren't comparable — four steps versus three. The comparable quantity is cost per traversal step: 51 s forward against 49 s backward once the one confounded step is excluded. Within 4%.

Time per traversal step
External evaluator E01, single session, 29 July 2026
Task A · forward (REID → artefact) Task B · backward (artefact → REID)
Read this cautiously. One evaluator, seven steps, no inter-evaluator reliability — so the 100% rate carries a wide confidence interval. Step B2 (00:33) measured re-location, not first location, because the RAMC had already been opened during Task A. Step B1's 40 s is a lower bound: E01 had been briefed on MLflow.

Instantiation

ShareLah: three ML components chosen to straddle the split.

A social expense management prototype — the kind of app where ML is a convenience, not a safety function. Two components are LLM-agents governed by instruction files; one is a trained regression model. The framework had to cover all three with the same conventions.

Receipt OCR

A two-call LLM-agent pipeline: extraction, then itemisation.

ocr-skill.md v1.7
itemisation-skill.md v1.2
→ rasc-ocr.md

Expense categorisation

Single-call LLM-agent with disambiguation rules derived from a consumer survey.

categorisation-skill.md v1.2
→ rasc-categorisation.md

Spending forecast

Trained LightGBM regression, versioned through MLflow and DVC.

MLflow c2a3bc33…
DVC dataset v1.4
→ ramc-forecast.md


Evaluation harness

The criteria were written first. Then they were run.

Every criterion below existed as a quantified threshold in §11 of a RASC before the harness executed. That is the property Model Cards lack: a post-hoc document cannot fail its own criteria, because its criteria are written from the results.

Receipt OCR · run eh-ocr-2026-07-21-r2
13-receipt fixed eval set · test-set SHA 665e766… · provider snapshot moonshot-v1-8k-vision-preview
CriterionMetricTargetRecordedOutcome
AC-OCR-01-1confidence (mean, call 1)≥ 0.700.760✓ PASS
AC-OCR-01-2requires_review_trigger_rateconf < 0.71.000✓ PASS
AC-OCR-01-3refusal_rate_non_myr≥ 0.951.000✓ PASS
AC-OCR-01-4itemisation_confidence (mean)≥ 0.850.960✓ PASS
AC-OCR-01-5empty_line_items_rate1.000.333✕ FAIL
Expense categorisation · run eh-cat-2026-07-24-r1
30-transaction fixed eval set · test-set SHA 9574b207… · provider snapshot moonshot-v1-8k · overall agreement 0.926 across 27 in-scope transactions
CriterionCategoryTargetRecordedOutcome
AC-CAT-01-1 … 8Food, Drinks, Transport, Flights & Transit, Accommodation, Entertainment, Shopping, Bills & Utilities≥ 0.801.000✓ 8× PASS
AC-CAT-01-9agreement_others≥ 0.800.333✕ FAIL
AC-CAT-01-10refusal_rate_oos≥ 0.950.000✕ FAIL
The three most analytically significant passes are hidden inside that first row. The AirAsia flight, the Redang hotel stay and the Langkawi road-trip petrol purchase were survey boundary cases where human respondents themselves failed to reach 80% agreement (69%, 67%, 64%). All three were resolved correctly by disambiguation rules in categorisation-skill.md v1.2 — an unbroken chain from survey finding, to instruction-file rule, to eval-set transaction, to recorded result.

The three failures

The failures are the finding.

Each of these could have been made to pass. None of them was.

AC-OCR-01-5 · single-charge receipts got itemised anyway

Of three single-charge receipts, only the Bolt ride-hailing one correctly returned an empty line-item array. The Maxis telco bill and the Shell petrol receipt both extracted line items — despite itemisation-skill.md v1.2 explicitly naming telco and petrol as single-charge categories. The instruction file was deliberately not over-specified to name the two offending merchants. Doing so would have secured a passing metric while concealing the real property: instruction compliance on visually complex receipts cannot be guaranteed by natural-language specification alone.

AC-CAT-01-9 · the catch-all category, held to the same bar

An Angpow gift went to Shopping, a bank transfer fee to Bills & Utilities. Both sit at boundaries a human annotator would contest. The instructive part is the asymmetry it exposed: the RAMC had excluded "Others" from its requirement because a meaningful threshold is indefensible for a residual category — yet the RASC kept it and scored it against the same 0.80. A uniform requirements-first discipline applied across paradigms surfaced an inconsistency in how the same category was treated. Ad-hoc documentation would have left it unrecorded.

AC-CAT-01-10 · refusal, the recurring defect class

None of the three out-of-scope inputs were refused: conversational text became Entertainment, a JPY-denominated dinner became Food, gibberish became Others. The SKILL.md refusal instruction was insufficient to override the hosted model's default extraction behaviour. The same defect arose independently in two separately authored skill files and was surfaced in both by the same criterion structure — evidence that §11 works as a detection instrument, not a descriptive one. Prompt specifications sit close to requirements; proximity does not entail compliance by the model reading them.

And when a failure was fixed, the fix was itself traceable.

Under ocr-skill.md v1.6, the non-MYR refusal criterion returned 0.333. The pre-condition currency gate was strengthened in v1.7 through REID-prefixed commits 500a73 and d1c5e08, and the next run recorded 1.000 against an unchanged test-set SHA. A detected criterion failure produced a targeted, requirement-tagged specification revision whose effect was measurable in the next run — the LLM-agent analogue of retraining, executed through a Git commit.

One more, filed under caveats: high confidence on a wrong answer.

Receipt 004 (Watsons) returned an amount of 99.0 in the harness but 69.0 in live testing — with a model-reported confidence of 0.95 attached to the incorrect extraction. That falsifies self-reported confidence as a proxy for correctness, and justifies treating uncertainty communication and user correction as separate requirements rather than one.


Overhead

Three quarters of the evidence section was already lying around.

DR8 claims the framework adds little work beyond MLOps and Git practice a team already runs. Rather than rest that on self-reported timings, every field instance in §12 of all three artefacts was audited and classified as machine-derivable (copyable from MLflow, DVC, the Git log or results.jsonl without interpretation) or author-supplied.

§12 Validation Evidence — field instances by origin
176 field instances audited across one RAMC and two RASCs
Machine-derivable Author-supplied
The residue is where the thinking happened. Author-supplied content concentrates in evidence identifiers, the linkage back to §11 (a copy, not novel content), and the interpretive notes on the four failing or boundary criteria. Explaining why agreement_others = 0.333 is a defensible boundary case rather than a model defect is an act of judgement with no automated substitute.

This is where the report argues against itself. The accompanying estimate — one to two hours per component, four to six hours total — was self-reported by a single developer who is also the framework's author, and reconstructed retrospectively rather than logged. The planned comparison against an ad-hoc documentation baseline was not achieved, because one developer cannot credibly serve as both treatment and control arm. The derivability audit is offered as the more defensible instrument precisely because it is reproducible by inspecting the committed artefacts. DR8 therefore stands validated at the level of mechanism, with the comparative claim untested.


User study · n = 30

Every acceptance metric passed. That was not the interesting part.

Thirty participants, four-point forced-choice Likert (no neutral midpoint), acceptance threshold 3.0. Behaviourally: 28 of 30 (93%) accepted the suggested category unmodified. 7 (23%) reported an incorrect or missing extracted field — and overall satisfaction still sat at 3.67 with every participant at or above threshold. Consumers accepting a component with a visible error rate is precisely the tolerance that separates consumer from safety-critical contexts.

User acceptance metrics against the 3.0 threshold
Mean score on a 1–4 forced-choice scale · n = 30 · dashed marker is the acceptance threshold
ItemConstructMean% ≥ 3.0Outcome

The confidence badge finding

77% said they understood the badge. Four of them thought it meant the receipt was real.

The confidence badge is how the OCR component discharges its uncertainty-communication requirement. On the Likert item, 23 of 30 scored at or above threshold. The free-text answers told a different story: four participants — two of whom had claimed understanding — described the badge as signalling document authenticity.

"The authenticity of the document is high" · "its legit" · "Thought it was ligitamacy of the company" · "Receipt validity"

Self-reported understanding against demonstrated interpretation
Comprehension is not the complement of the Likert score · n = 30
Interpreted correctly
Authenticity misreading
Total
Self-reported ≥ 3.0
21
2claimed, but wrong
23
Self-reported < 3.0
5
2
7
Total
26
4
30
Combining the seven who did not claim understanding with the two who claimed it but demonstrably misread the signal: 9 of 30 participants (30%) did not receive the intended uncertainty signal. The same pattern recurred in the skill-version transparency tag — 14 noticed it, 10 did not, 6 noticed it without knowing what it meant, so 16 of 30 (53%) derived no meaning from it. Two independent §13 mechanisms failing alike points at the template, not the implementation.

This is not a usability defect in ShareLah, and reading it that way misidentifies the unit of analysis. §13 of the OCR RASC specifies the uncertainty behaviour completely — three confidence bands, defined thresholds, badge labels, per-band interface behaviour aligned to Amershi et al.'s guidelines. The implementation instantiated that specification, and the badge appeared on screen for every participant.

The study established that specification completeness and user comprehension are separable properties — and that the RASC captures the first while providing no instrument for the second. The framework revealed a gap that its own criteria, being behavioural rather than perceptual, were incapable of detecting.

The fix is structural: §13 should carry a perceptual acceptance criterion of the same falsifiable form as §11's behavioural ones. For example — at least 80% of participants correctly state the badge's meaning unprompted. That makes DR7 a refinement target rather than a violation: as written it constrains the specification of uncertainty without constraining its reception.


Scorecard

Nine design requirements, judged against the evidence.

6
validated by direct empirical evidence
2
partially validated — mechanism shown, comparison untested
1
met as written, but shown under-specified
0
falsified
DRRequirementEvidenceVerdict
DR1Prospective, development-guiding templatesCriteria authored before the harness ran; 29/71 REID-prefixed commits✓ Validated
DR2Quantified acceptance criteria before developmentThree criteria could only fail because they were pre-specified✓ Validated
DR3Fits agile workflowsIncremental, commit-level adoption observed — not dedicated measurement◐ Supported
DR4Tolerates error rates and iterationComponents carried into the user study with open failures; 23% saw an error, satisfaction 3.67✓ Validated
DR5REID conventions across MLOps and GitUnfamiliar evaluator traversed both directions unassisted✓ Validated
DR6Extend existing artefacts, not replace themAll nine Model Card sections preserved; RASC structurally parallel✓ Validated
DR7Probabilistic criteria and uncertainty handlingBanded thresholds present in every §13 — but 30% did not receive the signal◐ Under-specified
DR8Minimal overhead beyond existing practice74.4% machine-derivable; baseline comparison not achieved◐ Mechanism only
DR9One REID, either paradigmPer-step cost invariant to direction and paradigm (51 s vs 49 s)✓ Validated

Seven things this does not establish

1. Validation is single-domain — every claim rests on ShareLah.   2. The forecast model was trained on synthetic data (16 users, 2024–2025), so its MAPE results are of limited external validity.   3. One evaluator means no inter-evaluator reliability; 7/7 carries a wide interval.   4. Step B2 is confounded — uncontaminated backward evidence is two steps, not three.   5. The overhead estimate is retrospective self-report by the framework's own author.   6. DR7 constrains specification, not reception, and the template has no test for the latter.   7. The sample was demographically concentrated — 23 of 30 participants (76.6%) aged 18–24.

User acceptance of the forecast component was deferred on methodological grounds: participants had ten minutes and no transaction history, and REQ-FCT-01 requires at least a week of it. Showing them a forecast built from a synthetic stranger's spending would have measured nothing. REQ-FCT-01 therefore stands validated technically — all six tiered MAPE thresholds met on held-out test users — but not at the acceptance level.


The claim

A single traceability chain, one vocabulary, both paradigms.

One evaluator, in one session, traced a requirement forward through a RASC to a commit on a markdown file, and traced an MLflow run backward through a RAMC to its originating requirement. The cost per step proved effectively invariant to direction and to the paradigm of the artefact at the far end. That invariance — not any aggregate success rate — is the operational content of the claim.

It is a capability PROV-ML, ML-Schema and MSR4ML cannot supply, each taking the traced artefact to be the product of a training procedure. It is equally the respect in which RAMC and RASC extend rather than duplicate Model Cards and Datasheets for Datasets, both of which remain descriptive artefacts produced after the decisions they document.

A framework reporting only the passes on components built by its own author would demonstrate nothing at all. Chapter 5 · Conclusion