1
Postgraduate Medical Education and Training Board. Developing and maintaining an assessment system - a PMETB guide to good practice. 2007.https://www.researchgate.net/profile/Christopher_Oliver/publication/264405740_Developing_and_Maintaining_an_Assessment_System-a_PMETB_Guide_to_Good_Practice/links/53dba0b50cf2cfac9928f468/Developing-and-Maintaining-an-Assessment-System-a-PMETB-Guide-to-Good-Practice.pdf
2
Association for the Study of Medical Education. Understanding medical education: evidence, theory and practice. Second edition. Chichester, West Sussex, UK: : Wiley Blackwell 2014. http://lib.myilibrary.com/ProductDetail.aspx?id=543123&entityid=https://idp.warwick.ac.uk/idp/shibboleth
3
Cox M, Irby DM, Epstein RM. Assessment in Medical Education. New England Journal of Medicine 2007;356:387–96. doi:10.1056/NEJMra054784
4
Jolly, Brian, Grant, Janet. The good assessment guide: a practical guide to assessment and appraisal for higher specialist training. London: : Joint Centre for Education in Medicine 1997.
5
Schuwirth, Lambert W. T., Vleuten, C. van der, Association for the Study of Medical Education. How to design a useful test: the principles of assessment. Edinburgh: : ASME 2006.
6
Black, H. D., Devine, Marion, Scottish Council for Research in Education. Assessment purposes: a study of the relationship between diagnostic assessment and summative assessment for certification. Edinburgh: : Scottish Council for Research in Education 1986.
7
Bloom, Benjamin S. Taxonomy of educational objectives: the classification of educational goals, Handbook 1: Cognitive domain. London: : Longman Group Ltd 1956.
8
Cangelosi JS. Designing tests for evaluating student achievement. New York: : Longman
9
Hart IR. Trends in clinical assessment. In: Approaches to the Assessment of Clinical Competence, Part 1 and 2. 1992.
10
Livingston SA, Zieky MJ. Passing scores. 1982.http://www.ets.org/Media/Research/pdf/passing_scores.pdf
11
Samuel Messick. The Psychology of Educational Measurement. Journal of Educational Measurement;21:215–37.http://0-www.jstor.org.pugwash.lib.warwick.ac.uk/stable/1434780
12
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine 1990;65:63–7.http://0-ovidsp.dc2.ovid.com.pugwash.lib.warwick.ac.uk/sp-4.02.1a/ovidweb.cgi?WebLinkFrameset=1&S=LJBFFPKHHDEBEBIDIPCKNGPEPMFBAA00&returnUrl=ovidweb.cgi%3fMain%2bSearch%2bPage%3d1%26S%3dLJBFFPKHHDEBEBIDIPCKNGPEPMFBAA00&fromjumpstart=0&directlink=http%3a%2f%2fovidsp.dc2.ovid.com%2fovftpdfs%2fFPEBIPPENGIDHD00%2ffs046%2fovft%2flive%2fgv023%2f00001888%2f00001888-199009000-00045.pdf&filename=The+assessment+of+clinical+skills%2fcompetence%2fperformance.&link_from=S.sh.22.23.27.31%7c45&pdf_key=FPEBIPPENGIDHD00&pdf_index=/fs046/ovft/live/gv023/00001888/00001888-199009000-00045&D=ovft&link_set=S.sh.22.23.27.31|45|sl_10|tocsiblings|S.sh.22.23.27.31.37|0
13
Ozuah PO, Reznik M. Using unannounced standardised patients to assess residents’ professionalism. Medical Education 2008;42:532–3. doi:10.1111/j.1365-2923.2008.03083.x
14
Peile E. Knowing and knowing about. BMJ 2006;332:645–645. doi:10.1136/bmj.332.7542.645
15
G. Rasch. Probabilistic models for some intelligence and attainment tests. Chicago: : University of Chicago Press 1980.
16
Rethans J-J, Norcini JJ, Baron-Maldonado M, et al. The relationship between competence and performance: implications for assessing practice performance. Medical Education 2002;36:901–9. doi:10.1046/j.1365-2923.2002.01316.x
17
Rethans J-J, Norcini JJ, Barón-Maldonado M, et al. The relationship between competence and performance: implications for assessing practice performance. Medical Education 2002;36:901–9. doi:10.1046/j.1365-2923.2002.01316.x
18
Rowntree, Derek. Assessing students: how shall we know them? Rev. ed. London: : Kogan Page 1987.
19
Vleuten CPM. The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education 1996;1:41–67. doi:10.1007/BF00596229
20
Research Methods - Validity and Reliability in AllPsych Online. https://allpsych.com/research-methods/variablesvalidityreliability/validityreliability/
21
Research Methods Knowledge Base. http://www.socialresearchmethods.net/kb/
22
Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychological Bulletin 1955;52:281–302. doi:10.1037/h0040957
23
Messick S. Validity. In: Educational Measurement (The American Council on Education/Macmillan series on higher education). Macmillan USA
24
Developing and maintaining an assessment system. https://www.researchgate.net/publication/264405740_Developing_and_Maintaining_an_Assessment_System-a_PMETB_Guide_to_Good_Practice
25
Schuwirth LWT, van der Vleuten CPM. Programmatic assessment and Kane’s validity perspective. Medical Education 2012;46:38–48. doi:10.1111/j.1365-2923.2011.04098.x
26
Breakwell, Glynis M., Smith, Jonathan A., Wright, Daniel B. Research methods in psychology. 4th ed. Los Angeles: : SAGE 2012.
27
Downing SM, Haladyna TM. Validity threats: overcoming interference with proposed interpretations of assessment data. Medical Education 2004;38:327–33. doi:10.1046/j.1365-2923.2004.01777.x
28
Schuwirth LW. Assessing medical competence: finding the right answers. The Clinical Teacher 2004;1:14–8. doi:10.1111/j.1743-498X.2004.00012.x
29
Schuwirth LW. Assessing medical competence: finding the right answers. The Clinical Teacher 2004;1:14–8. doi:10.1111/j.1743-498X.2004.00012.x
30
Downing SM. Reliability: on the reproducibility of assessment data. Medical Education 2004;38:1006–12. doi:10.1111/j.1365-2929.2004.01932.x
31
Murphy DJ, Bruce DA, Mercer SW, et al. The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Advances in Health Sciences Education 2009;14:219–32. doi:10.1007/s10459-008-9104-8
32
Tighe J, McManus I, Dewhurst NG, et al. The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations. BMC Medical Education 2010;10. doi:10.1186/1472-6920-10-40
33
Schuwirth LWT, van der Vleuten CPM. General overview of the theories used in assessment: AMEE Guide No. 57. Medical Teacher 2011;33:783–97. doi:10.3109/0142159X.2011.611022
34
Nunnally, Jum C., Bernstein, Ira H. Psychometric theory. 3rd ed. New York: : McGraw-Hill 1994.
35
Ricketts C. A plea for the proper use of criterion-referenced tests in medical assessment. Medical Education 2009;43:1141–6. doi:10.1111/j.1365-2923.2009.03541.x
36
Friedman Ben-Davis M. AMEE Guide No. 18: Standard setting in student assessment. Medical Teacher 2000;22:120–30. doi:10.1080/01421590078526
37
Norcini JJ. Setting standards on educational tests. Medical Education 2003;37:464–9. doi:10.1046/j.1365-2923.2003.01495.x
38
Norcini JJ. Setting standards on educational tests. Medical Education 2003;37:464–9. doi:10.1046/j.1365-2923.2003.01495.x
39
Downing SM, Tekian A, Yudkowsky R. RESEARCH METHODOLOGY: Procedures for Establishing Defensible Absolute Passing Scores on Performance Examinations in Health Professions Education. Teaching and Learning in Medicine 2006;18:50–7. doi:10.1207/s15328015tlm1801_11
40
Bandaranayake RC. Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37. Medical Teacher 2008;30:836–45. doi:10.1080/01421590802402247
41
Liu M, Liu K-M. Setting Pass Scores for Clinical Skills Assessment. The Kaohsiung Journal of Medical Sciences 2008;24:656–63. doi:10.1016/S1607-551X(09)70032-4
42
Wood TJ, Humphrey-Murto SM, Norman GR. Standard Setting in a Small Scale OSCE: A Comparison of the Modified Borderline-Group Method and the Borderline Regression Method. Advances in Health Sciences Education 2006;11:115–22. doi:10.1007/s10459-005-7853-1
43
Cohen-Schotanus J, van der Vleuten CPM. A standard setting method with the best performing students as a point of reference: Practical and affordable. Medical Teacher 2010;32:154–60. doi:10.3109/01421590903196979
44
Hurley KF. OSCE and clinical skills handbook. Second edition. Toronto: : Elsevier/Saunders 2011.
45
Epstein RM. Assessment in Medical Education. New England Journal of Medicine;356:387–96. doi:10.1056/NEJMra054784
46
Schuwirth, Lambert W. T., Vleuten, C. van der, Association for the Study of Medical Education. How to design a useful test: the principles of assessment. Edinburgh: : ASME 2006.
47
Schuwirth LWT. ABC of learning and teaching in medicine: Written assessment. BMJ 2003;326:643–5. doi:10.1136/bmj.326.7390.643
48
Schuwirth LWT, van der Vleuten CPM. ABC Of Learning And Teaching In Medicine: Written Assessment. BMJ: British medical journal 2003;326:643–5.http://0-www.jstor.org.pugwash.lib.warwick.ac.uk/stable/25453994
49
Dory V, Gagnon R, Charlin B. Is case-specificity content-specificity? An analysis of data from extended-matching questions. Advances in Health Sciences Education 2010;15:55–63. doi:10.1007/s10459-009-9169-z
50
Farmer EA, Page G. A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education 2005;39:1188–94. doi:10.1111/j.1365-2929.2005.02339.x
51
Farmer EA, Page G. A practical guide to assessing clinical decision-making skills using the key features approach. Medical education 2005;39:1188–94. doi:10.1111/j.1365-2929.2005.02339.x
52
Gagnon R, Charlin B, Roy L, et al. The Cognitive Validity of the Script Concordance Test: A Processing Time Study. Teaching and Learning in Medicine 2006;18:22–7. doi:10.1207/s15328015tlm1801_6
53
Tweed M, Wilkinson T. A randomized controlled trial comparing instructions regarding unsafe response options in a MCQ examination. Medical Teacher 2009;31:51–4. doi:10.1080/01421590802146018
54
National Board of Medical Examiners. Constructing Written Test Questions For the Basic and Clinical Sciences. http://www.nbme.org/publications/item-writing-manual.html
55
Brigden D. Constructing a learning portfolio. BMJ 1999;319:2a–2a. doi:10.1136/bmj.319.7201.2a
56
Challis M. AMEE Medical Education Guide No.11 (revised): Portfolio-based learning and assessment in medical education. Medical Teacher 1999;21:370–86. doi:10.1080/01421599979310
57
Challis M. Portfolios and assessment: meeting the challenge. Medical Teacher 2001;23:437–40. doi:10.1080/01421590120075643
58
Does a student log provide a means to better structure clinical education? Medical Education 1999;33:89–94. doi:10.1046/j.1365-2923.1999.00285.x
59
Driessen E, van Tartwijk J, Vermunt J, et al. Use of portfolios in early undergraduate medical training. Medical Teacher 2003;25:18–23. doi:10.1080/0142159021000061378
60
Driessen E, van der Vleuten C, Schuwirth L, et al. The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study. Medical Education 2005;39:214–20. doi:10.1111/j.1365-2929.2004.02059.x
61
Driessen EW, Overeem K, van Tartwijk J, et al. Validity of portfolio assessment: which qualities determine ratings? Medical Education 2006;40:862–6. doi:10.1111/j.1365-2929.2006.02550.x
62
Driessen EW, Muijtjens AMM, van Tartwijk J, et al. Web- or paper-based portfolios: is there a difference? Medical Education 2007;41:1067–73. doi:10.1111/j.1365-2923.2007.02859.x
63
du Boulay C. From CME to CPD: getting better at getting better? BMJ 2000;320:393–4. doi:10.1136/bmj.320.7232.393
64
Freeman, Richard T., Lewis, Roger. Planning and implementing assessment. London: : Kogan Page 1998.
65
David MFB, Davis MH, Harden RM, et al. AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical Teacher 2001;23:535–51. doi:10.1080/01421590120090952
66
Hays RB. Reflecting on learning portfolios. Medical Education 2004;38:801–3. doi:10.1111/j.1365-2929.2004.01831.x
67
Brian Jolly. Clinical logbooks: recording clinical experiences may not be enough. Medical Education 1999;33:86–8. doi:10.1046/j.1365-2923.1999.00386.x
68
Mathers NJ, Challis MC, Howe AC, et al. Portfolios in continuing medical education - effective and efficient? Medical Education 1999;33:521–30. doi:10.1046/j.1365-2923.1999.00407.x
69
O’sullivan PS, Reckase MD, McClain T, et al. Demonstration of Portfolios to Assess Competency of Residents. Advances In Health Sciences Education 2004;9:309–23. doi:10.1007/s10459-004-0885-0
70
Pearson DJ, Heywood P. Portfolio use in general practice vocational training: a survey of GP registrars. Medical Education 2004;38:87–95. doi:10.1111/j.1365-2923.2004.01737.x
71
Pitts J, Coles C, Thomas P. Enhancing reliability in portfolio assessment: ‘shaping’ the portfolio. Medical Teacher 2001;23:351–6. doi:10.1080/01421590120057021
72
Pitts J, Coles C, Thomas P. Educational portfolios in the assessment of general practice trainers: reliability of assessors. Medical Education 1999;33:515–20. doi:10.1046/j.1365-2923.1999.00445.x
73
Pitts, John, Association for the Study of Medical Education. Portfolios, personal development and reflective practice. Edinburgh: : ASME 2007.
74
Rees C. The use (and abuse) of the term ‘portfolio’. Medical Education 2005;39:436–436. doi:10.1111/j.1365-2929.2005.02119.x
75
Roberts C, Newble DI, O’Rourke AJ. Portfolio-based assessments in medical education: are they valid and reliable for summative purposes? Medical Education 2002;36:899–900. doi:10.1046/j.1365-2923.2002.01288.x
76
Schuwirth LWT, Vleuten CPM. A plea for new psychometric models in educational assessment. Medical Education 2006;40:296–300. doi:10.1111/j.1365-2929.2006.02405.x
77
Snadden D. Portfolios - attempting to measure the unmeasurable? Medical Education 1999;33:478–9. doi:10.1046/j.1365-2923.1999.00446.x
78
Snadden D, Thomas ML. Portfolio learning in general practice vocational training - does it work? MEDICAL EDUCATION 1998;32:401–6. doi:10.1046/j.1365-2923.1998.00245.x
79
Webb C, Endacott R, Gray M, et al. Models of portfolios. Medical Education 2002;36:897–8. doi:10.1046/j.1365-2923.2002.01318.x
80
Webb C, Endacott R, A Gray M, et al. Evaluating portfolio assessment systems: what are the appropriate criteria? Nurse Education Today 2003;23:600–9. doi:10.1016/S0260-6917(03)00098-4
81
Wilkinson TJ, Challis M, Hobma SO, et al. The use of portfolios for assessment of the competence and performance of doctors in practice. Medical Education 2002;36:918–24. doi:10.1046/j.1365-2923.2002.01312.x
82
Archer JC. Use of SPRAT for peer review of paediatricians in training. BMJ 2005;330:1251–3. doi:10.1136/bmj.38447.610451.8F
83
Archer J, Norcini J, Southgate L, et al. mini-PAT (Peer Assessment Tool): A Valid Component of a National Assessment Programme in the UK? Advances in Health Sciences Education 2008;13:181–92. doi:10.1007/s10459-006-9033-3
84
Campbell LM, Howie JG, Murray TS. Use of videotaped consultations in summative assessment of trainees in general practice. British Journal of General Practice 1995;45:137–41.http://0-bjgp.org.pugwash.lib.warwick.ac.uk/content/45/392/137
85
Crossley J, Eiser C, Davies HA. Children and their parents assessing the doctor-patient interaction: a rating system for doctors’ communication skills. Medical Education 2005;39:820–8. doi:10.1111/j.1365-2929.2005.02230.x
86
Daelmans HEM, van der Hem-Stokroos HH, Hoogenboom RJI, et al. Feasibility and reliability of an in-training assessment programme in an undergraduate clerkship. Medical Education 2004;38:1270–7. doi:10.1111/j.1365-2929.2004.02019.x
87
Evans R. Review of instruments for peer assessment of physicians. BMJ 2004;328. doi:10.1136/bmj.328.7450.1240
88
Govaerts MJB, Vleuten CPM, Schuwirth LWT, et al. Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment. Advances in Health Sciences Education 2007;12:239–60. doi:10.1007/s10459-006-9043-1
89
Murphy DJ, Bruce DA, Mercer SW, et al. The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Advances in Health Sciences Education 2009;14:219–32. doi:10.1007/s10459-008-9104-8
90
Norcini JJ. The Mini-CEX (Clinical Evaluation Exercise): A Preliminary Investigation. Annals of Internal Medicine 1995;123. doi:10.7326/0003-4819-123-10-199511150-00008
91
Norcini JJ. ABC of learning and teaching in medicine: Work based assessment. BMJ 2003;326:753–5. doi:10.1136/bmj.326.7392.753
92
Postgraduate Medical Education and Training Board. Developing and maintaining an assessment system - a PMETB guide to good practice. 2007.http://www.gmc-uk.org/Assessment_good_practice_v0207.pdf_31385949.pdf
93
Ramsey PG. Use of Peer Ratings to Evaluate Physician Performance. JAMA: The Journal of the American Medical Association 1993;269. doi:10.1001/jama.1993.03500130069034
94
Ringsted C, Henriksen AH, Skaarup AM, et al. Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice. Medical Education 2004;38:767–77. doi:10.1111/j.1365-2929.2004.01841.x
95
Whitehouse A, Hassell A, Bullock A, et al. 360 degree assessment (multisource feedback) of UK trainee doctors: Field testing of team assessment of behaviours (TAB). Medical Teacher 2007;29:171–6. doi:10.1080/01421590701302951
96
Moonen-van Loon JMW, Overeem K, Donkers HHLM, et al. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Advances in Health Sciences Education 2013;18:1087–102. doi:10.1007/s10459-013-9450-z
97
Bullock AD, Hassell A, Markham WA, et al. How ratings vary by staff group in multi-source feedback assessment of junior doctors. Medical Education 2009;43:516–20. doi:10.1111/j.1365-2923.2009.03333.x
98
Cleland JA, Knight LV, Rees CE, et al. Is it me or is it them? Factors that influence the passing of underperforming students. Medical Education 2008;42:800–9. doi:10.1111/j.1365-2923.2008.03113.x
99
Davies H, Archer J, Bateman A, et al. Specialty-specific multi-source feedback: assuring validity, informing training. Medical Education 2008;42:1014–20. doi:10.1111/j.1365-2923.2008.03162.x
100
Hill F, Kendall K, Galbraith K, et al. Implementing the undergraduate mini-CEX: a tailored approach at Southampton University. Medical Education 2009;43:326–34. doi:10.1111/j.1365-2923.2008.03275.x
101
Kogan JR, Holmboe ES, Hauer KE. Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees. JAMA 2009;302. doi:10.1001/jama.2009.1365
102
Postgraduate Medical Education and Training Board. Workplace Based Assessment: A Guide for Implementation. 2009.http://train-com.de/mediadb/9468/20532/eng41_workplace.pdf
103
Richards SH, Campbell JL, Walshaw E, et al. A multi-method analysis of free-text comments from the UK General Medical Council Colleague Questionnaires. Medical Education 2009;43:757–66.https://go.exlibris.link/dqJBlzww
