1.
Postgraduate Medical Education and Training Board: Developing and maintaining an assessment system - a PMETB guide to good practice, https://www.researchgate.net/profile/Christopher_Oliver/publication/264405740_Developing_and_Maintaining_an_Assessment_System-a_PMETB_Guide_to_Good_Practice/links/53dba0b50cf2cfac9928f468/Developing-and-Maintaining-an-Assessment-System-a-PMETB-Guide-to-Good-Practice.pdf, (2007).
2.
Association for the Study of Medical Education: Understanding medical education: evidence, theory and practice. Wiley Blackwell, Chichester, West Sussex, UK (2014).
3.
Cox, M., Irby, D.M., Epstein, R.M.: Assessment in Medical Education. New England Journal of Medicine. 356, 387–396 (2007). https://doi.org/10.1056/NEJMra054784.
4.
Jolly, Brian, Grant, Janet: The good assessment guide: a practical guide to assessment and appraisal for higher specialist training. Joint Centre for Education in Medicine, London (1997).
5.
Schuwirth, Lambert W. T., Vleuten, C. van der, Association for the Study of Medical Education: How to design a useful test: the principles of assessment. ASME, Edinburgh (2006).
6.
Black, H. D., Devine, Marion, Scottish Council for Research in Education: Assessment purposes: a study of the relationship between diagnostic  assessment and summative assessment for certification. Scottish Council for Research in Education, Edinburgh (1986).
7.
Bloom, Benjamin S.: Taxonomy of educational objectives: the classification of  educational goals, Handbook 1: Cognitive domain. Longman Group Ltd, London (1956).
8.
Cangelosi, J.S.: Designing tests for evaluating student achievement. Longman, New York.
9.
Hart, I.R.: Trends in clinical assessment. In: Approaches to the Assessment of Clinical Competence, Part 1 and 2 (1992).
10.
Livingston, S.A., Zieky, M.J.: Passing scores, http://www.ets.org/Media/Research/pdf/passing_scores.pdf, (1982).
11.
Samuel Messick: The Psychology of Educational Measurement. Journal of Educational Measurement. 21, 215–237.
12.
Miller, G.E.: The assessment of clinical skills/competence/performance. Academic Medicine. 65, 63–67 (1990).
13.
Ozuah, P.O., Reznik, M.: Using unannounced standardised patients to assess residents’ professionalism. Medical Education. 42, 532–533 (2008). https://doi.org/10.1111/j.1365-2923.2008.03083.x.
14.
Peile, E.: Knowing and knowing about. BMJ. 332, 645–645 (2006). https://doi.org/10.1136/bmj.332.7542.645.
15.
G. Rasch: Probabilistic models for some intelligence and attainment tests. University of Chicago Press, Chicago (1980).
16.
Rethans, J.-J., Norcini, J.J., Baron-Maldonado, M., Blackmore, D., Jolly, B.C., LaDuca, T., Lew, S., Page, G.G., Southgate, L.H.: The relationship between competence and performance: implications for assessing practice performance. Medical Education. 36, 901–909 (2002). https://doi.org/10.1046/j.1365-2923.2002.01316.x.
17.
Rethans, J.-J., Norcini, J.J., Barón-Maldonado, M., Blackmore, D., Jolly, B.C., LaDuca, T., Lew, S., Page, G.G., Southgate, L.H.: The relationship between competence and performance: implications for assessing practice performance. Medical Education. 36, 901–909 (2002). https://doi.org/10.1046/j.1365-2923.2002.01316.x.
18.
Rowntree, Derek: Assessing students: how shall we know them? Kogan Page, London (1987).
19.
Vleuten, C.P.M.: The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education. 1, 41–67 (1996). https://doi.org/10.1007/BF00596229.
20.
Research Methods - Validity and Reliability in AllPsych Online, https://allpsych.com/research-methods/variablesvalidityreliability/validityreliability/.
21.
Research Methods Knowledge Base, http://www.socialresearchmethods.net/kb/.
22.
Cronbach, L.J., Meehl, P.E.: Construct validity in psychological tests. Psychological Bulletin. 52, 281–302 (1955). https://doi.org/10.1037/h0040957.
23.
Messick, S.: Validity. In: Educational Measurement (The American Council on Education/Macmillan series on higher education). Macmillan USA.
24.
Developing and maintaining an assessment system, https://www.researchgate.net/publication/264405740_Developing_and_Maintaining_an_Assessment_System-a_PMETB_Guide_to_Good_Practice.
25.
Schuwirth, L.W.T., van der Vleuten, C.P.M.: Programmatic assessment and Kane’s validity perspective. Medical Education. 46, 38–48 (2012). https://doi.org/10.1111/j.1365-2923.2011.04098.x.
26.
Breakwell, Glynis M., Smith, Jonathan A., Wright, Daniel B.: Research methods in psychology. SAGE, Los Angeles (2012).
27.
Downing, S.M., Haladyna, T.M.: Validity threats: overcoming interference with proposed interpretations of assessment data. Medical Education. 38, 327–333 (2004). https://doi.org/10.1046/j.1365-2923.2004.01777.x.
28.
Schuwirth, L.W.: Assessing medical competence: finding the right answers. The Clinical Teacher. 1, 14–18 (2004). https://doi.org/10.1111/j.1743-498X.2004.00012.x.
29.
Schuwirth, L.W.: Assessing medical competence: finding the right answers. The Clinical Teacher. 1, 14–18 (2004). https://doi.org/10.1111/j.1743-498X.2004.00012.x.
30.
Downing, S.M.: Reliability: on the reproducibility of assessment data. Medical Education. 38, 1006–1012 (2004). https://doi.org/10.1111/j.1365-2929.2004.01932.x.
31.
Murphy, D.J., Bruce, D.A., Mercer, S.W., Eva, K.W.: The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Advances in Health Sciences Education. 14, 219–232 (2009). https://doi.org/10.1007/s10459-008-9104-8.
32.
Tighe, J., McManus, I., Dewhurst, N.G., Chis, L., Mucklow, J.: The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations. BMC Medical Education. 10, (2010). https://doi.org/10.1186/1472-6920-10-40.
33.
Schuwirth, L.W.T., van der Vleuten, C.P.M.: General overview of the theories used in assessment: AMEE Guide No. 57. Medical Teacher. 33, 783–797 (2011). https://doi.org/10.3109/0142159X.2011.611022.
34.
Nunnally, Jum C., Bernstein, Ira H.: Psychometric theory. McGraw-Hill, New York (1994).
35.
Ricketts, C.: A plea for the proper use of criterion-referenced tests in medical assessment. Medical Education. 43, 1141–1146 (2009). https://doi.org/10.1111/j.1365-2923.2009.03541.x.
36.
Friedman Ben-Davis, M.: AMEE Guide No. 18: Standard setting in student assessment. Medical Teacher. 22, 120–130 (2000). https://doi.org/10.1080/01421590078526.
37.
Norcini, J.J.: Setting standards on educational tests. Medical Education. 37, 464–469 (2003). https://doi.org/10.1046/j.1365-2923.2003.01495.x.
38.
Norcini, J.J.: Setting standards on educational tests. Medical Education. 37, 464–469 (2003). https://doi.org/10.1046/j.1365-2923.2003.01495.x.
39.
Downing, S.M., Tekian, A., Yudkowsky, R.: RESEARCH METHODOLOGY: Procedures for Establishing Defensible Absolute Passing Scores on Performance Examinations in Health Professions Education. Teaching and Learning in Medicine. 18, 50–57 (2006). https://doi.org/10.1207/s15328015tlm1801_11.
40.
Bandaranayake, R.C.: Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37. Medical Teacher. 30, 836–845 (2008). https://doi.org/10.1080/01421590802402247.
41.
Liu, M., Liu, K.-M.: Setting Pass Scores for Clinical Skills Assessment. The Kaohsiung Journal of Medical Sciences. 24, 656–663 (2008). https://doi.org/10.1016/S1607-551X(09)70032-4.
42.
Wood, T.J., Humphrey-Murto, S.M., Norman, G.R.: Standard Setting in a Small Scale OSCE: A Comparison of the Modified Borderline-Group Method and the Borderline Regression Method. Advances in Health Sciences Education. 11, 115–122 (2006). https://doi.org/10.1007/s10459-005-7853-1.
43.
Cohen-Schotanus, J., van der Vleuten, C.P.M.: A standard setting method with the best performing students as a point of reference: Practical and affordable. Medical Teacher. 32, 154–160 (2010). https://doi.org/10.3109/01421590903196979.
44.
Hurley, K.F.: OSCE and clinical skills handbook. Elsevier/Saunders, Toronto (2011).
45.
Epstein, R.M.: Assessment in Medical Education. New England Journal of Medicine. 356, 387–396. https://doi.org/10.1056/NEJMra054784.
46.
Schuwirth, Lambert W. T., Vleuten, C. van der, Association for the Study of Medical Education: How to design a useful test: the principles of assessment. ASME, Edinburgh (2006).
47.
Schuwirth, L.W.T.: ABC of learning and teaching in medicine: Written assessment. BMJ. 326, 643–645 (2003). https://doi.org/10.1136/bmj.326.7390.643.
48.
Schuwirth, L.W.T., van der Vleuten, C.P.M.: ABC Of Learning And Teaching In Medicine: Written Assessment. BMJ: British medical journal. 326, 643–645 (2003).
49.
Dory, V., Gagnon, R., Charlin, B.: Is case-specificity content-specificity? An analysis of data from extended-matching questions. Advances in Health Sciences Education. 15, 55–63 (2010). https://doi.org/10.1007/s10459-009-9169-z.
50.
Farmer, E.A., Page, G.: A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education. 39, 1188–1194 (2005). https://doi.org/10.1111/j.1365-2929.2005.02339.x.
51.
Farmer, E.A., Page, G.: A practical guide to assessing clinical decision-making skills using the key features approach. Medical education. 39, 1188–1194 (2005). https://doi.org/10.1111/j.1365-2929.2005.02339.x.
52.
Gagnon, R., Charlin, B., Roy, L., St-Martin, M., Sauve, E., Boshuizen, H.P.A., van der Vleuten, C.: The Cognitive Validity of the Script Concordance Test: A Processing Time Study. Teaching and Learning in Medicine. 18, 22–27 (2006). https://doi.org/10.1207/s15328015tlm1801_6.
53.
Tweed, M., Wilkinson, T.: A randomized controlled trial comparing instructions regarding unsafe response options in a MCQ examination. Medical Teacher. 31, 51–54 (2009). https://doi.org/10.1080/01421590802146018.
54.
National Board of Medical Examiners: Constructing Written Test Questions For the Basic and Clinical Sciences, http://www.nbme.org/publications/item-writing-manual.html.
55.
Brigden, D.: Constructing a learning portfolio. BMJ. 319, 2a–2a (1999). https://doi.org/10.1136/bmj.319.7201.2a.
56.
Challis, M.: AMEE Medical Education Guide No.11 (revised): Portfolio-based learning and assessment in medical education. Medical Teacher. 21, 370–386 (1999). https://doi.org/10.1080/01421599979310.
57.
Challis, M.: Portfolios and assessment: meeting the challenge. Medical Teacher. 23, 437–440 (2001). https://doi.org/10.1080/01421590120075643.
58.
Does a student log provide a means to better structure clinical education? Medical Education. 33, 89–94 (1999). https://doi.org/10.1046/j.1365-2923.1999.00285.x.
59.
Driessen, E., van Tartwijk, J., Vermunt, J., van der Vleuten, C.: Use of portfolios in early undergraduate medical training. Medical Teacher. 25, 18–23 (2003). https://doi.org/10.1080/0142159021000061378.
60.
Driessen, E., van der Vleuten, C., Schuwirth, L., van Tartwijk, J., Vermunt, J.: The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study. Medical Education. 39, 214–220 (2005). https://doi.org/10.1111/j.1365-2929.2004.02059.x.
61.
Driessen, E.W., Overeem, K., van Tartwijk, J., van der Vleuten, C.P.M., Muijtjens, A.M.M.: Validity of portfolio assessment: which qualities determine ratings? Medical Education. 40, 862–866 (2006). https://doi.org/10.1111/j.1365-2929.2006.02550.x.
62.
Driessen, E.W., Muijtjens, A.M.M., van Tartwijk, J., van der Vleuten, C.P.M.: Web- or paper-based portfolios: is there a difference? Medical Education. 41, 1067–1073 (2007). https://doi.org/10.1111/j.1365-2923.2007.02859.x.
63.
du Boulay, C.: From CME to CPD: getting better at getting better? BMJ. 320, 393–394 (2000). https://doi.org/10.1136/bmj.320.7232.393.
64.
Freeman, Richard T., Lewis, Roger: Planning and implementing assessment. Kogan Page, London (1998).
65.
David, M.F.B., Davis, M.H., Harden, R.M., Howie, P.W., Ker, J., Pippard, M.J.: AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical Teacher. 23, 535–551 (2001). https://doi.org/10.1080/01421590120090952.
66.
Hays, R.B.: Reflecting on learning portfolios. Medical Education. 38, 801–803 (2004). https://doi.org/10.1111/j.1365-2929.2004.01831.x.
67.
Brian Jolly: Clinical logbooks: recording clinical experiences may not be enough. Medical Education. 33, 86–88 (1999). https://doi.org/10.1046/j.1365-2923.1999.00386.x.
68.
Mathers, N.J., Challis, M.C., Howe, A.C., Field, N.J.: Portfolios in continuing medical education - effective and efficient? Medical Education. 33, 521–530 (1999). https://doi.org/10.1046/j.1365-2923.1999.00407.x.
69.
O’sullivan, P.S., Reckase, M.D., McClain, T., Savidge, M.A., Clardy, J.A.: Demonstration of Portfolios to Assess Competency of Residents. Advances In Health Sciences Education. 9, 309–323 (2004). https://doi.org/10.1007/s10459-004-0885-0.
70.
Pearson, D.J., Heywood, P.: Portfolio use in general practice vocational training: a survey of GP registrars. Medical Education. 38, 87–95 (2004). https://doi.org/10.1111/j.1365-2923.2004.01737.x.
71.
Pitts, J., Coles, C., Thomas, P.: Enhancing reliability in portfolio assessment: ‘shaping’ the portfolio. Medical Teacher. 23, 351–356 (2001). https://doi.org/10.1080/01421590120057021.
72.
Pitts, J., Coles, C., Thomas, P.: Educational portfolios in the assessment of general practice trainers: reliability of assessors. Medical Education. 33, 515–520 (1999). https://doi.org/10.1046/j.1365-2923.1999.00445.x.
73.
Pitts, John, Association for the Study of Medical Education: Portfolios, personal development and reflective practice. ASME, Edinburgh (2007).
74.
Rees, C.: The use (and abuse) of the term ‘portfolio’. Medical Education. 39, 436–436 (2005). https://doi.org/10.1111/j.1365-2929.2005.02119.x.
75.
Roberts, C., Newble, D.I., O’Rourke, A.J.: Portfolio-based assessments in medical education: are they valid and reliable for summative purposes? Medical Education. 36, 899–900 (2002). https://doi.org/10.1046/j.1365-2923.2002.01288.x.
76.
Schuwirth, L.W.T., Vleuten, C.P.M.: A plea for new psychometric models in educational assessment. Medical Education. 40, 296–300 (2006). https://doi.org/10.1111/j.1365-2929.2006.02405.x.
77.
Snadden, D.: Portfolios - attempting to measure the unmeasurable? Medical Education. 33, 478–479 (1999). https://doi.org/10.1046/j.1365-2923.1999.00446.x.
78.
Snadden, D., Thomas, M.L.: Portfolio learning in general practice vocational training - does it work? MEDICAL EDUCATION. 32, 401–406 (1998). https://doi.org/10.1046/j.1365-2923.1998.00245.x.
79.
Webb, C., Endacott, R., Gray, M., Jasper, M., Miller, C., McMullan, M., Scholes, J.: Models of portfolios. Medical Education. 36, 897–898 (2002). https://doi.org/10.1046/j.1365-2923.2002.01318.x.
80.
Webb, C., Endacott, R., A Gray, M., Jasper, M.A., McMullan, M., Scholes, J.: Evaluating portfolio assessment systems: what are the appropriate criteria? Nurse Education Today. 23, 600–609 (2003). https://doi.org/10.1016/S0260-6917(03)00098-4.
81.
Wilkinson, T.J., Challis, M., Hobma, S.O., Newble, D.I., Parboosingh, J.T., Sibbald, R.G., Wakeford, R.: The use of portfolios for assessment of the competence and performance of doctors in practice. Medical Education. 36, 918–924 (2002). https://doi.org/10.1046/j.1365-2923.2002.01312.x.
82.
Archer, J.C.: Use of SPRAT for peer review of paediatricians in training. BMJ. 330, 1251–1253 (2005). https://doi.org/10.1136/bmj.38447.610451.8F.
83.
Archer, J., Norcini, J., Southgate, L., Heard, S., Davies, H.: mini-PAT (Peer Assessment Tool): A Valid Component of a National Assessment Programme in the UK? Advances in Health Sciences Education. 13, 181–192 (2008). https://doi.org/10.1007/s10459-006-9033-3.
84.
Campbell, L.M., Howie, J.G., Murray, T.S.: Use of videotaped consultations in summative assessment of trainees in general practice. British Journal of General Practice. 45, 137–141 (1995).
85.
Crossley, J., Eiser, C., Davies, H.A.: Children and their parents assessing the doctor-patient interaction: a rating system for doctors’ communication skills. Medical Education. 39, 820–828 (2005). https://doi.org/10.1111/j.1365-2929.2005.02230.x.
86.
Daelmans, H.E.M., van der Hem-Stokroos, H.H., Hoogenboom, R.J.I., Scherpbier, A.J.J.A., Stehouwer, C.D.A., van der Vleuten, C.P.M.: Feasibility and reliability of an in-training assessment programme in an undergraduate clerkship. Medical Education. 38, 1270–1277 (2004). https://doi.org/10.1111/j.1365-2929.2004.02019.x.
87.
Evans, R.: Review of instruments for peer assessment of physicians. BMJ. 328, (2004). https://doi.org/10.1136/bmj.328.7450.1240.
88.
Govaerts, M.J.B., Vleuten, C.P.M., Schuwirth, L.W.T., Muijtjens, A.M.M.: Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment. Advances in Health Sciences Education. 12, 239–260 (2007). https://doi.org/10.1007/s10459-006-9043-1.
89.
Murphy, D.J., Bruce, D.A., Mercer, S.W., Eva, K.W.: The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Advances in Health Sciences Education. 14, 219–232 (2009). https://doi.org/10.1007/s10459-008-9104-8.
90.
Norcini, J.J.: The Mini-CEX (Clinical Evaluation Exercise): A Preliminary Investigation. Annals of Internal Medicine. 123, (1995). https://doi.org/10.7326/0003-4819-123-10-199511150-00008.
91.
Norcini, J.J.: ABC of learning and teaching in medicine: Work based assessment. BMJ. 326, 753–755 (2003). https://doi.org/10.1136/bmj.326.7392.753.
92.
Postgraduate Medical Education and Training Board: Developing and maintaining an assessment system - a PMETB guide to good practice., http://www.gmc-uk.org/Assessment_good_practice_v0207.pdf_31385949.pdf, (2007).
93.
Ramsey, P.G.: Use of Peer Ratings to Evaluate Physician Performance. JAMA: The Journal of the American Medical Association. 269, (1993). https://doi.org/10.1001/jama.1993.03500130069034.
94.
Ringsted, C., Henriksen, A.H., Skaarup, A.M., Van der Vleuten, C.P.M.: Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice. Medical Education. 38, 767–777 (2004). https://doi.org/10.1111/j.1365-2929.2004.01841.x.
95.
Whitehouse, A., Hassell, A., Bullock, A., Wood, L., Wall, D.: 360 degree assessment (multisource feedback) of UK trainee doctors: Field testing of team assessment of behaviours (TAB). Medical Teacher. 29, 171–176 (2007). https://doi.org/10.1080/01421590701302951.
96.
Moonen-van Loon, J.M.W., Overeem, K., Donkers, H.H.L.M., Vleuten, C.P.M., Driessen, E.W.: Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Advances in Health Sciences Education. 18, 1087–1102 (2013). https://doi.org/10.1007/s10459-013-9450-z.
97.
Bullock, A.D., Hassell, A., Markham, W.A., Wall, D.W., Whitehouse, A.B.: How ratings vary by staff group in multi-source feedback assessment of junior doctors. Medical Education. 43, 516–520 (2009). https://doi.org/10.1111/j.1365-2923.2009.03333.x.
98.
Cleland, J.A., Knight, L.V., Rees, C.E., Tracey, S., Bond, C.M.: Is it me or is it them? Factors that influence the passing of underperforming students. Medical Education. 42, 800–809 (2008). https://doi.org/10.1111/j.1365-2923.2008.03113.x.
99.
Davies, H., Archer, J., Bateman, A., Dewar, S., Crossley, J., Grant, J., Southgate, L.: Specialty-specific multi-source feedback: assuring validity, informing training. Medical Education. 42, 1014–1020 (2008). https://doi.org/10.1111/j.1365-2923.2008.03162.x.
100.
Hill, F., Kendall, K., Galbraith, K., Crossley, J.: Implementing the undergraduate mini-CEX: a tailored approach at Southampton University. Medical Education. 43, 326–334 (2009). https://doi.org/10.1111/j.1365-2923.2008.03275.x.
101.
Kogan, J.R., Holmboe, E.S., Hauer, K.E.: Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees. JAMA. 302, (2009). https://doi.org/10.1001/jama.2009.1365.
102.
Postgraduate Medical Education and Training Board: Workplace Based Assessment: A Guide for Implementation, http://train-com.de/mediadb/9468/20532/eng41_workplace.pdf, (2009).
103.
Richards, S.H., Campbell, J.L., Walshaw, E., Dickens, A., Greco, M.: A multi-method analysis of free-text comments from the UK General Medical Council Colleague Questionnaires. Medical Education. 43, 757–766 (2009).