Standard Setting for MRCP(UK) Part 2 Clinical Examination (PACES)
Before any PACES examination may start every examiner is required to examine the patients with whom they will be working to assess PACES candidates and agree with their fellow examiner the relative importance of the clinical features of each case and the performance criteria necessary for the award of a pass. This discussion is known as calibration.
The overall pass level is defined in part by the content of the patient encounter but also by the examiners' knowledge of:
- The curricula to which the assessment is mapped.
- The competencies defined in the curricula.
- The level of performance required to meet the curricular requirements in the workplace.
- The day-to-day workplace performance necessary to be “Receiving Medical Officer (RMO) Ready”. The traditional level at which this “RMO Ready” standard has been set has been that the candidate should be capable of managing patients (making accurate initial assessments and instituting appropriate initial treatment) admitted in a typical acute general medical take.
To ensure consistency in the examination, examiners are required to calibrate with reference to the published 'clinical skills'. A table listing the clinical skills is available here.
At all stations examiners must also complete 'calibration sheets', setting out what clinical signs or symptoms they have agreed to be present in the patient, and the specific criteria they have defined necessary to be met for the award of a Pass in that Station. View an example calibration sheet:
Where standardised scenarios are used during the PACES examination (with surrogates rather than real patients) a detailed scenario document is given to each examiner. This includes all the information given to the candidate immediately before the exam and also the guidance given to the surrogate themselves, along with explicit instructions to the examiners about what issues the candidate should explore or diagnoses they should make in order to pass. View an example of a scenario document:
Example scenario document
One of the strengths of the PACES examination is that each candidate is assessed by 10 different examiners in the carousel and by two examiners, marking independently, at each encounter. Although examiners assess patients and simulated patients themselves before each carousel, and agree on the criteria for award of a satisfactory judgement (the process of calibration), it is inevitable in any judgement-based system that examiner disagreement will sometimes occur. MRCP(UK) monitors the level of examiner disagreement overall, and for individual examiners, but does not regard the occurrence of disagreement as indicating that the assessment is in any way flawed. The benefits to individual candidates, and the assessment overall, of paired examiner observation at each encounter far outweigh any problem caused by inter-examiner disagreement.