Trainees Are Outsourcing Clinical Judgement to Models. The Curriculum Still Grades the Answer.

TLDR: A language model can now produce the answer, the differential and the written rationale that medical assessments reward, so programmes have to score what only the trainee can show: commitment, doubt, error-catching and escalation.

A model now writes the reasoning that examiners score

In a 2024 randomised trial of 50 United States physicians, the large language model GPT-4 (Generative Pre-trained Transformer 4) working alone scored a median 92% on a structured diagnostic-reasoning rubric¹. Physicians given the same model scored 76%, and physicians using conventional resources scored 74%. The rubric was a serious instrument: it awarded points for plausible differentials, for findings that support and oppose each diagnosis, for the final diagnosis and for next diagnostic steps. That is the written trace of reasoning that many programmes treat as evidence of clinical thinking, and a model produced it better than the clinicians did.

The same pattern appeared earlier on licensing items. In 2023, ChatGPT performed at or near the approximately 60% pass threshold² on 350 publicly released United States Medical Licensing Examination (USMLE) sample questions, including a format that forced it to justify each answer choice.

Those two results shift the problem. When a model can generate the answer and its justification, a correct answer with a fluent rationale stops being proof that the learner did the thinking. The warnings now circulating about medical trainees and artificial intelligence (AI) point at learners. The sharper target is the assessment design.

The warning is loud and the direct evidence is still thin

Three high-profile pieces set the current debate. A Nature Medicine perspective published in May 2026 names the risk “never-skilling”³: trainees who lean on AI early risk skipping the reasoning needed for independent practice, a risk distinct from deskilling in experienced clinicians and from “mis-skilling”, the acceptance of AI errors as fact. An editorial in BMJ Evidence-Based Medicine warns that automation bias, cognitive off-loading and deskilling⁴ threaten students and new doctors most, because they are still building the experience to question AI advice. A 2025 review in the New England Journal of Medicine proposes a supervision routine called DEFT-AI⁵, in which supervisors probe both the learner’s reasoning and how the learner used and verified the model.

The Nature Medicine authors are candid about the evidence base. Their abstract states that direct evidence from medical training is absent, and that the concern rests on learning theory and early signals from non-clinical settings. The article frames its own three-phase framework as a research agenda that needs testing before it informs policy.

The evidence that does exist sits in three different categories, and each supports a narrower claim than the headlines suggest.

Measured deskilling in experienced clinicians. At four Polish endoscopy centres, the adenoma detection rate (ADR) on standard colonoscopies fell from 28.4% to 22.4%⁶ in the three months after AI polyp detection arrived, compared with the three months before. The study is observational and covers experienced endoscopists rather than trainees, so other changes over the period remain possible explanations alongside AI exposure.

Measured learning loss outside medicine. In a field experiment with nearly 1,000 Turkish high-school students, unrestricted GPT-4 access raised practice grades by 48% and cut exam grades by 17%⁷ once access was removed. A tutor version that gave hints instead of answers raised practice grades by 127% and largely removed the exam penalty.

Thin, short-term data in medical students. A 2026 systematic review found only four eligible studies covering 408 medical students⁸. AI use was associated with faster task completion and better basic-knowledge scores; effects on clinical reasoning were mixed, retention was measured once over ten days, and the authors conclude that the potential risks of deskilling warrant attention while describing the evidence base as limited.

Self-reported data add a fourth layer. In a survey of 319 knowledge workers, higher confidence in generative AI was associated with less critical thinking⁹, and higher self-confidence with more. That finding concerns perceptions and covers office work, so it supports a hypothesis about trainees rather than a measured effect.

Read together, the evidence for widespread trainee outsourcing of judgement remains a warning built on adjacent data, while the evidence that a model can satisfy an assessment is direct and replicated.

Grading the output measures performance with the tool

The Bastani experiment isolates the mechanism that matters for curriculum design. Students using the unrestricted model looked stronger during practice, the phase most programmes observe, and weaker on the test that removed the tool. Any assessment that scores a submitted artefact produced with the tool available measures the combined performance of learner plus model.

Medical education carries that exposure in specific places: take-home cases, written reflections, portfolio entries, research assignments and any structured write-up completed away from an examiner. The Goh trial adds a second layer. Physicians with GPT-4 performed at the level of those without it, while the model alone outscored both groups. The physicians who had the model scored 76% against the model’s own 92%, which suggests that the skill being exercised was the human’s own judgement about what to accept. That skill is precisely what an output-only rubric leaves unscored.

Medicine already assesses reasoning, and AI shows where the gaps sit

Medical education built reasoning assessment long before language models, and those tools deserve credit before any new proposal.

Script concordance tests (SCTs) present ambiguous scenarios and ask how a new finding shifts a hypothesis; a validity review reports high internal consistency and weak correlation with tests of pure factual knowledge¹⁰, which is the property a reasoning measure needs. The mini clinical evaluation exercise (mini-CEX) places an examiner beside a real encounter; its foundational study analysed 1,228 encounters with 421 residents across 21 internal medicine programmes¹¹ and found measurement characteristics comparable to other performance assessments, with higher fidelity. The Association of American Medical Colleges (AAMC) Core Entrustable Professional Activities (EPAs) treat self-awareness of limitations that leads to appropriate help-seeking¹² as foundational to every activity. The Accreditation Council for Graduate Medical Education (ACGME) requires that each resident know the limits of their scope of authority¹³ and the circumstances that permit conditional independence.

That infrastructure is already in place, and AI exposes three gaps in how programmes use it.

1. Fluent explanation passes as understanding. A rationale written with a model available reads identically to one written without it. Only observation, timing or a pre-commitment record separates the trainee’s reasoning from the model’s.
2. Model-error detection sits outside most rubrics. The BMJ editorial recommends cases that mix correct and deliberately flawed AI outputs, with learners accepting, amending or rejecting each and justifying the decision from primary evidence. That task belongs in the blueprint as a scored item.
3. Escalation is valued as a foundation and seldom earns its own marks. Help-seeking appears in the EPA framework and the ACGME rules as a foundation, while, in the Kainjoo Institute’s reading of common case-level marking schemes, the marks go to the diagnosis and the plan; the decision to call a senior deserves a mark of its own.

The Trace Test makes the trainee’s own reasoning observable

The Kainjoo Institute proposes a five-step routine that makes a trainee’s own reasoning observable when a model is in the room. It is a design proposal for educators to pilot, awaiting validation as an assessment instrument, and it draws directly on the BMJ recommendations, the DEFT-AI supervision routine and the EPA help-seeking foundation.

1. Commit. The trainee records a leading hypothesis and a short differential before opening any model. Time-stamped, unedited.
2. Contest. The trainee names one finding that supports and one that argues against the leading hypothesis, each tied to patient data or a primary source.
3. Catch. The trainee reviews a seeded model output containing one planted error and either finds it or explains why the output is sound.
4. Calibrate. The trainee rates confidence before and after the model input, and states what changed the rating.
5. Call. The trainee decides to proceed, defer or escalate, names who would be called, and gives the trigger for that decision.

Each step produces an artefact a model can only partially fake, because three of the five are anchored in time or in a planted error the examiner controls. Commit and Calibrate expose the difference between the trainee’s view and the model’s. Catch scores mis-skilling directly. Call turns help-seeking from a background value into a marked behaviour.

The routine scales by setting. In a written examination, Commit and Catch fit into a proctored station. On the ward, the same five prompts structure a supervisor conversation and can feed the entrustment evidence that programmes already collect.

What to do on Monday: five roles can pilot reasoning evidence this week

Assessment leads can audit one existing written assessment by running its items through a current general-purpose model; any item the model answers with a convincing rationale measures learner-plus-model performance and needs a proctored or observed counterpart.

Course directors can add one AI-free baseline station per rotation, following the BMJ recommendation for supervised critical-skills assessment, so that every trainee produces at least one reasoning record without assistance.

Case writers can seed one flawed model output into each teaching case and mark the accept, amend or reject decision with its justification.

Clinical supervisors can add a single question to existing workplace assessments: what was your hypothesis before you checked the tool, and when would you have called someone? The answer becomes scoreable evidence for the Call step.

Programme directors can record Commit and Call artefacts alongside logbook entries, which gives entrustment committees evidence of independent reasoning and escalation behaviour over time.

The same gap sits in every regulated training programme

This section is analysis by transfer: the studies above concern medicine and education, and comparable trials in other regulated sectors remain an open research gap.

Pharmaceutical quality, financial compliance and legal practice all train people through competency records that log a completed task or a passed test. Those records share the weakness this article describes: they evidence an output. Once staff draft deviation reports, suitability assessments or contract reviews with a model available, the record proves that the task was completed, and leaves open who did the judging.

The regulatory direction points the same way. Article 4 of the European Union’s Artificial Intelligence Act, as replaced by the Digital Omnibus on AI in force since 27 July 2026, requires providers and deployers to take measures to support the development of AI literacy¹⁴ of their staff and of other persons operating AI systems on their behalf, and states that the duty stops short of guaranteeing any specific level of literacy in any individual. The obligation therefore attaches to the measures an organisation takes. For a regulated organisation, a training record that captures a pre-model hypothesis, a caught error and an escalation decision is stronger evidence of those measures than a completion certificate. The Trace Test steps translate directly: commit before consulting the tool, contest the output against source data, catch the planted error, calibrate confidence, and call the qualified person. Kainjoo .life reached a related split in diagnosis: generative AI matches non-specialist doctors on accuracy, while production and policy decide the rest¹⁵.

A model now supplies the graded answer, so programmes must grade the trainee’s moves

Medical education spent decades learning to look past recall towards reasoning, and it already owns the instruments to do so. Generative AI moves the boundary again. The correct answer and its written justification are now the most reproducible parts of a trainee’s work; the hypothesis committed before help, the error caught, the honest change in confidence and the decision to call a senior remain the trainee’s own. Programmes that score those behaviours will know what their graduates can do without the tool and how safely they use it.

Kainjoo .life supports life-sciences and healthcare organisations whose training, medical affairs and quality teams now work alongside AI tools. Teams that want their competency records to evidence judgement as well as completion can contact Kainjoo .life¹⁶ to scope an assessment review.

This article has been reviewed by Haider Alleg, Managing Partner at Kainjoo SA.

References

1. Goh E, Gallo R, Hom J, et al. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Network Open. 2024;7(10):e2440969. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2825395
2. Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023. https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000198
3. Ke Y, Jin L, Ong JCL, et al. AI-induced never-skilling in medical education. Nature Medicine. 2026;32(6):1997–2006. https://www.nature.com/articles/s41591-026-04438-y
4. Potential risks of GenAI on medical education (editorial). BMJ Evidence-Based Medicine. 2025. doi:10.1136/bmjebm-2025-114339. https://ebm.bmj.com/lookup/doi/10.1136/bmjebm-2025-114339
5. Abdulnour RE, Gin B, Boscardin CK. Educational Strategies for Clinical Supervision of Artificial Intelligence Use. New England Journal of Medicine. 2025;393(8):786–797. https://www.nejm.org/doi/full/10.1056/NEJMra2503232 (summary: https://www.the-hospitalist.org/hospitalist/article/40662/in-the-literature/educational-strategies-for-clinical-supervision-of-ai-use/)
6. Budzyń K, Romańczyk M, Kitala D, et al. Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. Lancet Gastroenterology & Hepatology. 2025;10(10):896–903. https://doi.org/10.1016/S2468-1253(25)00133-5
7. Bastani H, Bastani O, Sungu A, et al. Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences. 2025;122(26):e2422633122. https://doi.org/10.1073/pnas.2422633122
8. Turney J, Young TM, Chauhan DR, Beeharry R, Mahmud M. AI Use for Medical Students: Impact on Clinical Skill Acquisition and Retention. A Systematic Review. Advances in Medical Education and Practice. 2026;17. https://pmc.ncbi.nlm.nih.gov/articles/PMC13084638/
9. Lee H-P, Sarkar A, Tankelevitch L, et al. The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. CHI ’25. 2025. https://www.microsoft.com/en-us/research/?p=1135061
10. Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten CPM. Script concordance testing: a review of published validity evidence. Medical Education. 2011;45(4):329–338. https://cris.maastrichtuniversity.nl/en/publications/script-concordance-testing-a-review-of-published-validity-evidenc/
11. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The Mini-CEX: A Method for Assessing Clinical Skills. Annals of Internal Medicine. 2003;138(6):476–481. https://www.ovid.com/journals/aime/pdf/00000605-200303180-00012~the-mini-cex-a-method-for-assessing-clinical-skills
12. Association of American Medical Colleges. Core Entrustable Professional Activities for Entering Residency: Faculty and Learners’ Guide. 2014. https://store.aamc.org/downloadable/download/sample/sample_id/66
13. Accreditation Council for Graduate Medical Education. ACGME Common Program Requirements (Residency), effective 1 July 2026. https://prod.acgme.org/globalassets/pfassets/programrequirements/2026-prs/cprresidency_2026.pdf
14. Regulation (EU) 2026/1744 of the European Parliament and of the Council of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 (Digital Omnibus on AI); Article 1(5) replaces Article 4 (AI literacy) of Regulation (EU) 2024/1689. Official Journal L, 24 July 2026. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L_202601744
15. Kainjoo .life. AI Diagnoses Like a Generalist. Production and Policy Decide the Rest. 13 June 2026. https://kainjoo.life/ai-diagnosis-production-and-policy/
16. Kainjoo .life. Expertise. https://kainjoo.life/kainjoo-life-expertise/