In 2026 a consumer app's "ML" is half trained models and half natural-language instruction files. Requirements traceability research only knows how to follow the first half. This project built — and then deliberately broke — a framework that follows both.
A user story says what the system shall do. That grammar was built for deterministic code, where a requirement is either met or it isn't. Machine learning does not offer that courtesy.
McKinsey's State of AI reports put organisational AI adoption at 60% in 2023 and 90% in 2025. Consumer teams are shipping ML features inside ordinary agile sprints — with no more time for requirements analysis than before, and specification techniques that never anticipated a probabilistic answer.
Binary pass/fail. Exceptions are enumerable. Verified by functional testing.
Confidence levels, not booleans. Degradation at a threshold. Verified statistically, or against a fixed eval set.
Since ChatGPT in 2022, consumer teams increasingly compose features on hosted foundation models rather than training their own. The result is a stack that spans two paradigms with different artefacts, different versioning surfaces, and different ways of failing.
Existing RE4AI traceability work — PROV-ML, ML-Schema, MSR4ML — assumes the traced artefact is the output of a training procedure. None of them has a representation for a natural-language specification file. That leaves a growing share of shipped ML behaviour outside any traceability chain at all.
Behaviour is fixed once the weights exist.
Behaviour is governed by a text file — and by someone else's model.
SKILL.mdThat last row is the quiet one. A trained model can't change under you. An LLM-agent component can shift behaviour with no commit, no run, and no diff — which is why the framework snapshots the provider model identity as part of the evidence.
The project follows the DSRM process model of Peffers et al.: identify the problem, define objectives, build the artefact, demonstrate it, evaluate it. The point of the derivation below is that the framework's own success criteria were written before it was built, from documented gaps rather than assumed ones.
Each DR carries a testable criterion and a verification method — template inspection, convention inspection, process-model review, or overhead assessment. A DR violation counts as framework inadequacy, not as a finding to be explained away.
The framework deliberately extends existing practice rather than introducing a new tool — new tooling imposes adoption barriers, which is exactly the overhead that kills documentation in the first place.
A paradigm-agnostic way to state ML requirements with probabilistic thresholds. The grammar of the requirement doesn't change based on whether a classifier or an LLM ends up implementing it. DR2 DR4 DR7
Requirement IDs (REIDs) tagged into MLflow runs, DVC tags, Git commit messages and eval-harness log entries. Bidirectional navigation falls out of the naming. DR5 DR6 DR9
The standard Model Card, with four new sections that link it back to the requirements it exists to satisfy. For trained-model components. DR1 DR3 DR8
The same four sections, applied to a versioned SKILL.md instead of a model checkpoint. This is the project's original contribution. DR1 DR3 DR8 DR9
Structural parallelism is a deliberate trade. A practitioner who knows Model Cards can adopt RASC with almost no new learning — but "RAMC" and "RASC" differ by one character, and the evaluation later paid for that in navigation time.
No new tool, no plugin, no database. A requirement identifier is written into the artefacts each tool already produces, so the chain is queryable in both directions with what a team is running anyway.
# Trained model — MLflow run tags and registry description mlflow.set_tag("req_ids", "REQ-FCT-01") description="Satisfies REQ-CAT-01. RAMC: docs/ramc-classifier.md" # Dataset version — DVC via annotated Git tags git tag -a "data-v1.1" -m "REQ-OCR-01: Added receipts" # LLM-agent — a commit message on the instruction file git commit -m "[REQ-CAT-01] Add Beverage / Food disambiguation rule" # Eval harness — one JSONL entry per criterion run { req_id: "REQ-CAT-01", skill_md_sha: "c0fc2aa", provider_model: "moonshot-v1-8k", agreement: 0.926, test_set_sha: "9574b207…" }
In the ShareLah repository this produced a main branch where 29 of 71 commits carry a REID prefix, and the skill files advanced through 13 REID-tagged revisions. One commit message — 500a7a3 "[REQ-OCR-01] Updated OCR skill v1.7 – strengthen non-MYR pre-condition gate after eval test" — records the originating requirement, the artefact version produced, and the evaluation finding that motivated it, in a single line.
On 29 July 2026 an external evaluator (E01) — a second-year CS undergraduate with no prior exposure to the repository — was asked to traverse the chain in both directions. The only briefing given was what MLflow is for.
E01 found docs/ quickly, then opened the OCR card before reaching the categorisation one. RAMC and RASC differ by a single character — the first empirical cost of the parallelism decision.
He landed on §7 (Agreement Metrics Definition) before §11. Both show numbers, and inheriting Mitchell et al.'s numbering puts the requirements-bearing content at the end of a thirteen-section document.
One JSONL entry, carrying the requirement ID, the test-set SHA, the provider model snapshot, and the recorded metric.
A commit on a markdown file. This is the hop no existing traceability framework can make.
From an MLflow run's req_ids tag, to the RAMC, to the originating requirement — same vocabulary, same evaluator, same session.
Task totals aren't comparable — four steps versus three. The comparable quantity is cost per traversal step: 51 s forward against 49 s backward once the one confounded step is excluded. Within 4%.
A social expense management prototype — the kind of app where ML is a convenience, not a safety function. Two components are LLM-agents governed by instruction files; one is a trained regression model. The framework had to cover all three with the same conventions.
A two-call LLM-agent pipeline: extraction, then itemisation.
ocr-skill.md v1.7
itemisation-skill.md v1.2
→ rasc-ocr.md
Single-call LLM-agent with disambiguation rules derived from a consumer survey.
categorisation-skill.md v1.2
→ rasc-categorisation.md
Trained LightGBM regression, versioned through MLflow and DVC.
MLflow c2a3bc33…
DVC dataset v1.4
→ ramc-forecast.md
Every criterion below existed as a quantified threshold in §11 of a RASC before the harness executed. That is the property Model Cards lack: a post-hoc document cannot fail its own criteria, because its criteria are written from the results.
| Criterion | Metric | Target | Recorded | Outcome |
|---|---|---|---|---|
| AC-OCR-01-1 | confidence (mean, call 1) | ≥ 0.70 | 0.760 | ✓ PASS |
| AC-OCR-01-2 | requires_review_trigger_rate | conf < 0.7 | 1.000 | ✓ PASS |
| AC-OCR-01-3 | refusal_rate_non_myr | ≥ 0.95 | 1.000 | ✓ PASS |
| AC-OCR-01-4 | itemisation_confidence (mean) | ≥ 0.85 | 0.960 | ✓ PASS |
| AC-OCR-01-5 | empty_line_items_rate | 1.00 | 0.333 | ✕ FAIL |
| Criterion | Category | Target | Recorded | Outcome |
|---|---|---|---|---|
| AC-CAT-01-1 … 8 | Food, Drinks, Transport, Flights & Transit, Accommodation, Entertainment, Shopping, Bills & Utilities | ≥ 0.80 | 1.000 | ✓ 8× PASS |
| AC-CAT-01-9 | agreement_others | ≥ 0.80 | 0.333 | ✕ FAIL |
| AC-CAT-01-10 | refusal_rate_oos | ≥ 0.95 | 0.000 | ✕ FAIL |
Each of these could have been made to pass. None of them was.
Of three single-charge receipts, only the Bolt ride-hailing one correctly returned an empty line-item array. The Maxis telco bill and the Shell petrol receipt both extracted line items — despite itemisation-skill.md v1.2 explicitly naming telco and petrol as single-charge categories. The instruction file was deliberately not over-specified to name the two offending merchants. Doing so would have secured a passing metric while concealing the real property: instruction compliance on visually complex receipts cannot be guaranteed by natural-language specification alone.
An Angpow gift went to Shopping, a bank transfer fee to Bills & Utilities. Both sit at boundaries a human annotator would contest. The instructive part is the asymmetry it exposed: the RAMC had excluded "Others" from its requirement because a meaningful threshold is indefensible for a residual category — yet the RASC kept it and scored it against the same 0.80. A uniform requirements-first discipline applied across paradigms surfaced an inconsistency in how the same category was treated. Ad-hoc documentation would have left it unrecorded.
None of the three out-of-scope inputs were refused: conversational text became Entertainment, a JPY-denominated dinner became Food, gibberish became Others. The SKILL.md refusal instruction was insufficient to override the hosted model's default extraction behaviour. The same defect arose independently in two separately authored skill files and was surfaced in both by the same criterion structure — evidence that §11 works as a detection instrument, not a descriptive one. Prompt specifications sit close to requirements; proximity does not entail compliance by the model reading them.
Under ocr-skill.md v1.6, the non-MYR refusal criterion returned 0.333. The pre-condition currency gate was strengthened in v1.7 through REID-prefixed commits 500a73 and d1c5e08, and the next run recorded 1.000 against an unchanged test-set SHA. A detected criterion failure produced a targeted, requirement-tagged specification revision whose effect was measurable in the next run — the LLM-agent analogue of retraining, executed through a Git commit.
Receipt 004 (Watsons) returned an amount of 99.0 in the harness but 69.0 in live testing — with a model-reported confidence of 0.95 attached to the incorrect extraction. That falsifies self-reported confidence as a proxy for correctness, and justifies treating uncertainty communication and user correction as separate requirements rather than one.
DR8 claims the framework adds little work beyond MLOps and Git practice a team already runs. Rather than rest that on self-reported timings, every field instance in §12 of all three artefacts was audited and classified as machine-derivable (copyable from MLflow, DVC, the Git log or results.jsonl without interpretation) or author-supplied.
This is where the report argues against itself. The accompanying estimate — one to two hours per component, four to six hours total — was self-reported by a single developer who is also the framework's author, and reconstructed retrospectively rather than logged. The planned comparison against an ad-hoc documentation baseline was not achieved, because one developer cannot credibly serve as both treatment and control arm. The derivability audit is offered as the more defensible instrument precisely because it is reproducible by inspecting the committed artefacts. DR8 therefore stands validated at the level of mechanism, with the comparative claim untested.
Thirty participants, four-point forced-choice Likert (no neutral midpoint), acceptance threshold 3.0. Behaviourally: 28 of 30 (93%) accepted the suggested category unmodified. 7 (23%) reported an incorrect or missing extracted field — and overall satisfaction still sat at 3.67 with every participant at or above threshold. Consumers accepting a component with a visible error rate is precisely the tolerance that separates consumer from safety-critical contexts.
| Item | Construct | Mean | % ≥ 3.0 | Outcome |
|---|
The confidence badge is how the OCR component discharges its uncertainty-communication requirement. On the Likert item, 23 of 30 scored at or above threshold. The free-text answers told a different story: four participants — two of whom had claimed understanding — described the badge as signalling document authenticity.
"The authenticity of the document is high" · "its legit" · "Thought it was ligitamacy of the company" · "Receipt validity"
This is not a usability defect in ShareLah, and reading it that way misidentifies the unit of analysis. §13 of the OCR RASC specifies the uncertainty behaviour completely — three confidence bands, defined thresholds, badge labels, per-band interface behaviour aligned to Amershi et al.'s guidelines. The implementation instantiated that specification, and the badge appeared on screen for every participant.
The study established that specification completeness and user comprehension are separable properties — and that the RASC captures the first while providing no instrument for the second. The framework revealed a gap that its own criteria, being behavioural rather than perceptual, were incapable of detecting.
The fix is structural: §13 should carry a perceptual acceptance criterion of the same falsifiable form as §11's behavioural ones. For example — at least 80% of participants correctly state the badge's meaning unprompted. That makes DR7 a refinement target rather than a violation: as written it constrains the specification of uncertainty without constraining its reception.
| DR | Requirement | Evidence | Verdict |
|---|---|---|---|
| DR1 | Prospective, development-guiding templates | Criteria authored before the harness ran; 29/71 REID-prefixed commits | ✓ Validated |
| DR2 | Quantified acceptance criteria before development | Three criteria could only fail because they were pre-specified | ✓ Validated |
| DR3 | Fits agile workflows | Incremental, commit-level adoption observed — not dedicated measurement | ◐ Supported |
| DR4 | Tolerates error rates and iteration | Components carried into the user study with open failures; 23% saw an error, satisfaction 3.67 | ✓ Validated |
| DR5 | REID conventions across MLOps and Git | Unfamiliar evaluator traversed both directions unassisted | ✓ Validated |
| DR6 | Extend existing artefacts, not replace them | All nine Model Card sections preserved; RASC structurally parallel | ✓ Validated |
| DR7 | Probabilistic criteria and uncertainty handling | Banded thresholds present in every §13 — but 30% did not receive the signal | ◐ Under-specified |
| DR8 | Minimal overhead beyond existing practice | 74.4% machine-derivable; baseline comparison not achieved | ◐ Mechanism only |
| DR9 | One REID, either paradigm | Per-step cost invariant to direction and paradigm (51 s vs 49 s) | ✓ Validated |
1. Validation is single-domain — every claim rests on ShareLah. 2. The forecast model was trained on synthetic data (16 users, 2024–2025), so its MAPE results are of limited external validity. 3. One evaluator means no inter-evaluator reliability; 7/7 carries a wide interval. 4. Step B2 is confounded — uncontaminated backward evidence is two steps, not three. 5. The overhead estimate is retrospective self-report by the framework's own author. 6. DR7 constrains specification, not reception, and the template has no test for the latter. 7. The sample was demographically concentrated — 23 of 30 participants (76.6%) aged 18–24.
User acceptance of the forecast component was deferred on methodological grounds: participants had ten minutes and no transaction history, and REQ-FCT-01 requires at least a week of it. Showing them a forecast built from a synthetic stranger's spending would have measured nothing. REQ-FCT-01 therefore stands validated technically — all six tiered MAPE thresholds met on held-out test users — but not at the acceptance level.
One evaluator, in one session, traced a requirement forward through a RASC to a commit on a markdown file, and traced an MLflow run backward through a RAMC to its originating requirement. The cost per step proved effectively invariant to direction and to the paradigm of the artefact at the far end. That invariance — not any aggregate success rate — is the operational content of the claim.
It is a capability PROV-ML, ML-Schema and MSR4ML cannot supply, each taking the traced artefact to be the product of a training procedure. It is equally the respect in which RAMC and RASC extend rather than duplicate Model Cards and Datasheets for Datasets, both of which remain descriptive artefacts produced after the decisions they document.
A framework reporting only the passes on components built by its own author would demonstrate nothing at all. Chapter 5 · Conclusion