Loading ...
Global Do...
News & Politics
6
0
Try Now
Log In
Pricing
DR SAIFUL’S NOTES ON MEDICAL & ALLIED HEALTH PROFESSION EDUCATION: STUDENTS’ ASSESSMENT Dr. Muhamad Saiful Bahri Yusoff MD, MScMEd Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 1 Content Assessment: what it means & what it means to me ...................2 evaluation and teaching process..............................................5 validity ..................................................................................7 characteristic of assessment – reliability ................................10 planning an assessment .......................................................13 overview of item formats .......................................................15 assessment of ‘knows’ and ‘knows how’..................................18 assessment of “show how” ....................................................28 assessment of “does” ............................................................33 performance assessment.......................................................38 assessment of professional behaviour ....................................42 standard setting & scoring procedures ...................................47 quality control in assessment ................................................54 Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 2 ASSESSMENT: WHAT IT MEANS & WHAT IT MEANS TO ME Introduction – give the similarity and differences between a ruler and a set of MCQ. • Similarity o Tool for assessment (measuring tool) Knowledge Skill Competency o Can be used for continuous assessment. • Differences o Ruler is more objective compare to MCQ. o Accuracy Validity? Reliability? 1. What is assessment? • Measurement o Process producing numbers • Evaluation (psychometric) o Adding values (meaning) to the numbers o E.g. number is 50%, meaning is who get marks above than 50% is considered pass and less than 50% is considered fail. o When we come to the meaning of value than we will talk about validity. • Test & Examination 2. Purposes of assessment: • To the teacher o How is the student doing? o How good are the students? o How am I doing? • To the student o How am I doing? o How should I learn? • Others Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 3 o How good are they • Purposes (from discussion in the class) o Steering student learning. o How far/good student doing. o To ensure standard of products. o Achievement of goals and objectives. o Continuous assessment of student progression. o Improvement of program. o To correct the weakness of student and teachers. o For change of plan of teaching. • Purposes (a handbook for medical teachers 4th edition David Newble & Robert Cannon) o Judging mastery of essential skills and knowledge. o Rank ordering students. o Measuring improvement overtime. o Diagnosing student difficulties. o Providing feedback for the students. o Evaluating the effectiveness of the course. o Motivating students to study. o Setting standard. o Quality control for the public. 3. Assessment in the instructional process • Identifying instructional goals. • Pre-assessing learners’ need. • Providing relevant instruction. • Assessing intended learning outcomes. • Using the results. 4. Types of assessment • In terms of functional roles in the instruction: • Placement (usually at the beginning). o Usually to determine students’ achievement in the beginning of study or course for example class determination according to their achievement. • Formative o Details and focus. Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 4 o Repeated through out the course. o Use as diagnostic assessment To students To teachers o To determine the specific items example weakness and strength. o Cannot be use as decisive assessment. o Detailed feedback on performance is high. • Diagnostic o To detect students’ difficulty during learning process. • Summative o Summary or overall the course. o At the end of instruction. o Use to determine the course of instruction (use as decisive assessment) for example pass or fail to progress to next year of study. o Detailed feedback on performance is less. • In terms of interpretation • “How good are the students?” o Compared to each other Norm-referenced (normative) Use usually for student selection Student compare with peers o Compared to a set standards (standard criteria) Criterion-referenced Usually use for professional course e.g. doctor, lawyer and etc. 5. Principles and limitations • Knowing what to assess has top priority. • Selection of procedure follows later. • Comprehensive assessment requires a variety of procedure. • Be aware of limitations of assessment procedures. • Assessment is a tool, not an objective. Formative Summative End of Instruction Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 5 EVALUATION AND TEACHING PROCESS 1. What to assess? • Knowledge, skill and attitude. • Taxonomy Bloom. o Cognitive domain Knowledge Comprehension Application Analysis Synthesis Evaluation o Psychomotor domain Perception Set Guided response Mechanism Complex overt response Adaptation Origination o Affective domain Receiving Responding Valuing Organization Characterized by value or value complex. • Miller’s Pyramid of clinical competence (1990) KNOWS KNOWS HOW SHOWS HOW DOES - Factual knowledge - Ability to use knowledge in a particular context - Clinical reasoning - Problem-solving - Ability to act appropriately in a practical session. - Hand-on behaviour in a simulated or practice situation - Actual performance in habitual (real) practice. Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 6 • Link between what and how KNOWS KNOWS HOW SHOWS HOW DOES Factual Tests: MCQ. Essay, oral Clinical-context based tests MCQ, essay, oral Performance ass. in vitro OSCE, SP based tests Performance ass. in vivo Undercover SPs, video, logs Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 7 VALIDITY 1. Construct: • Intangible collection of abstract concepts and principles • Inferred from behaviour • Explained by educational and psychological theory • For example o Driving ability o Policemen competency 2. Validity: • Evidence • Presented to support or refute the meaning/ interpretation assigned to assessment results. • Validity = construct validity • Approach as hypothesis • Unitary concept • More or less • Not an internal characteristic • The higher the stakes, the more critical is the determination of validity • On-going process • Face Validity? 3. Sources of evidence: • Content: o Examination blueprint/ test specifications o Sampling (adequate or not adequate) o Quality of items o Quality of test constructors E.g. policemen competency, the constructor should be people who are senior in police department. • Response process: o Examinee familiarity o Quality control Scanning/ scoring Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 8 Final scores Score report o Accuracy Combining different scores Applying pass/ fail decision rules Reporting to student/ faculty • Internal structure: o Item analysis Item difficulty/ discrimination Inter-item correlation Inter-total correlation o Reliability o Standard errors of measurement (SEM) • Relationship to other variables o Correlation to other relevant variables o Convergent correlation – similar measures o Divergent correlation – dissimilar measures o Test-criterion relationships Predictive (future) Concurrent (present) • Consequences: o Impact of scares/ results on student/ society o Consequences on learners/ future learning Positive consequences more than negative o Reasonableness of methods to determine pass/ fail cut-off scores o Pass/ fail consequences Classification accuracy False positive/ negative 4. Threats to validity: • Construct under-representation o Too few o Biased o Mismatch Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 9 o Low reliability • Construct irrelevant variance o Flawed item formats o Inappropriate difficulty o Cheating/ insecure o Indefensible passing-score methods Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 10 CHARACTERISTIC OF ASSESSMENT – RELIABILITY 1. Meaning of reliability • Reproducibility/consistency of scores/measurement over time/occasion. • Characteristic of result/outcome of the assessment, not the measurement instrument. • Necessary but not sufficient condition for validity 2. Relationship with validity, reliability and objectivity: • Validity: o Content Blueprinting Item quality Content expertise o Response process Student briefing Quality control at all steps o Internal structure Item analysis Reliability o Relationship to other variables Correlations with other relevant variables Convergent correlations Divergent correlations o Consequences On the student • Objectivity and reliability Valid and reliable Not valid but reliable Not valid and not reliable Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 11 o Increased objectivity result lower reliability E.g. essays, viva, rating o Strategies to reduce subjectivity Æ MCQs, checklist, etc: do they really improve reliability? Not much o Lessons More objective method is not necessarily more reliable or moe superior Choice of measurement method is firstly determined by educational context/purpose of the testing situation. 3. Types and determination of reliability: • Reliability of achievement o Stability Test – retest Test – retest with equivalent forms o Internal consistency Split-half Coefficient alpha Kuder-Richardson • Reliability of rater data o Inter-rater consistency o Intraclass correlation coefficient Æ analysis of variance (ANOVA) • Reliability of performance examinations o Inter-rater consistency 4. Factors affecting reliability • Number of assessment task o The larger the number of tasks, the higher the reliability o Overcoming ‘content specificity’ • Spread of scores o The larger the spread, the higher the reliability • Objectivity o Select procedures according to purpose, then make the procedures as objective as possible 5. Uses of reliability information Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 12 • Consequences of decision based on assessment o Very high stakes e.g licensure/certification: > 0.90 o End of course/year summative exam: 0.80 – 0.89 • Standard deviation (1-reliability) 6. Rigour vs. Practicality • Rigour o Validity o Reliability • Practicality o Feasibility Ease of administration Time required Cost Ease of interpretation and application o Acceptability • A mixture of methods is preferable to relying on a single method • Don’t forget educational impact Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 13 PLANNING AN ASSESSMENT 1. When we are planning an assessment the following step must be considered: • Defining the content to be assessed • Defining the purpose of the assessment • Choosing the most appropriate assessment methods and formats • Writing test items and scoring mechanism • Setting standards • Choosing the appropriate report format • Item banking In this note we will cover the first 3 step above. 2. Defining the content to be assessed • Miller’s Pyramid, other component? • Competence and performance • Authenticity across the levels of pyramid • Blueprinting o Simple or complex o Related to tasks/competencies o Ensures representativeness and adequate sampling o E.g. Blueprinting of OSCE question according to system and competency Component System Physical examination History taking CVS - chest pain - cyanosis 2 questions 2 questions RES - cough - SOB 1 question 2 questions 3. Defining the purpose of the assessment Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 14 • Formative vs. summative 4. Choosing the most appropriate assessment methods and formats Validity Reliability Feasibility Acceptability Educational Impact Purpose Summative +++ +++ +++ +++ +++ Formative +++ + + + ++ • Validity o Increasing authenticity o Move the assessment back to the workplace o Increase integration of competencies, resist atomization e.g. global rating over checklists o Multiple sources o General professional competencies o Use a qualitative, professional judgment • Reliability o Overcoming domain/content specificity Sample across content Sample across conditions o Any specific method can be sufficiently reliable, provided that sampling is adequate across content content/conditions – don’t throw away the old methods • Educational impact or consequential validity o Be aware of it • Planning assessment is designing instruction Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 15 OVERVIEW OF ITEM FORMATS 1. Item format according to Miller’s Pyramid • Pen – and – paper methods o Free response o Selected (fixed) response • Competence assessment o OSCE • Performance assessment o Observation, etc. 2. Principles • Response format vs. stimulus format • Authenticity (realistic) 3. Miller’s Pyramid (1990) DOES SHOWS HOW KNOWS HOW KNOWS SKILL CLINICAL AUTHENTICITY OBJECTIVITY Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 16 4. Stimulus and response formats • Stimulus format o A hypertensive 66-years-old obese lady complains of epigastric pain – CONTEXT RICH o Differential diagnoses of epigastric pain include – CONTEXT FREE o Stimulus format is the more important determinant of the thought process invoked o Don’t be fooled by the name, look at the stimulus and look at what you need to test • Response formats o Stimulus being equal, choose the most objective response formats. • Some consideration o Do not fall into ‘either/or’ thinking; frequently better to use both o Select the item type that provides the most direct measure of the intended learning outcome o An effective evaluation system contains several methods that reflects competence in identified areas of performance Performance assessment Competency assessment Written measurement (Vleuten et al, 2001) DOES SHOWS HOW KNOWS HOW KNOWS Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 17 o Final judgement on student progress should be made on the basis of multiple assessments, obtained on different occasions, using a variety of methods based on stated goals (Norman and Shannon, 1995) o Every formats has their advantages and disadvantages 5. Authenticity • Improved authenticity = improved validity • Trends o Include assessment day-to-day practice o Integrate competencies o Authenticity at all level of the pyramid Æ support multiple methods o Don’t ignore general professional competencies E.g. teamwork, professional behavior, reflection o Complex competencies Æ increased reliance on professional judgement Æ how to make vigorous (validity and reliability) Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 18 ASSESSMENT OF ‘KNOWS’ AND ‘KNOWS HOW’ 1. Revision of basic concepts: • Miller’s Pyramid (1990) • Response format & stimulus format • DOES SHOWS HOW KNOWS HOW KNOWS SKILL CLINICAL AUTHENTICITY OBJECTIVITY Performance assessment Competency assessment Written measurement (Vleuten et al, 2001) DOES SHOWS HOW KNOWS HOW KNOWS Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 19 Open-ended / Free response Select / multiple choice Context - Rich Context – Free 2. Measuring ‘thinking skill’ • Look at cognitive process required o E.g. recall/memory, interpretation/comprehension, problem- solving/reasoning. o Difficult to determine: Depends on the background of candidate as well as question content The same answer is arrived at using different processes. • Look at task of examinee o Simpler, more objective o Requires rote memory for isolated facts Æ recall item o Requires to reach a conclusion/make a prediction/select a course of action Æ application of knowledge item o E.g. to assess knowledge of the days of the weeks Context - free • List the days of the week Slightly context – rich • Work days in Kelantan include Context – rich STIMULUS FORMAT RESPONSE FORMAT Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 20 • A ruler in Manaland decided to have an 8-day week and inserted a new day, Fuadday, as the first day of the weekend. In Manaland people start work on Tuesday and now work 6 full days. The day after Fuadday is? • Promoting application of knowledge o Using novel situation o In context Æ context rich • As far as possible all items should be assess application of knowledge 3. Paper based methods • Select type o Type X (Multiple true-false) o Type A (Single best response) o Type R (extended matching) o The progress test • Supply type o Essay o Modified Essay Question • Mixture o Key Feature Problems/ Clinical Reasoning Exercise 4. General characteristics of select type (MCQ): • Strengths o Effective: • contributes to high validity and reliability per hour of testing time o Objective: • contribute to reliability o Cost-effective: • Administration and scoring o Higher-order thinking skills measurable if properly formatted • Weakness o Resource-intensive: Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 21 • Requires skill and time to write, especially context-rich questions o Often used to test recall of facts; trivilisation o Promotes surface learning in students if used to test primarily recall o Promotes trivilisation and avoidance of important areas and cognitive skills among item-writers if faced with difficulty in writing o Not able to assess summarizing and writing skills, critical thinking processes • Types available o Type X (MTF) • Strengths • Concise; most effective among types • Useful for evaluation of factual information • Weakness • Difficult to construct flawlessly; often leads item- writers to ask for pure recall • A student who knows something is false; does he know what the true answer is? • A rather complicated scoring procedure • Abandoned by NBME and use discouraged by most authorities • Visualization o Type A (Single Best Response) • Strengths • Highest reliability among the types • More suited to context-rich questions True False A C B D Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 22 • Often easier to construct than type X • Weakness • Often used to test fact • Visualization o Type R (Extended Matching) o The Progress Test • A strategy, not a specific method • Overcome negative steering effect • Regular MCQ examinations for all students • Aimed at graduation level • Upward progression expected • Visualization 4. Type X: uses and advantages • Efficient: 3 type X vs. 2 Type A in same time (Ebel & Frisbie, 1991) • Simple and direct measure of verbal knowledge • Most useful where there are 2 alternatives o Right/left, more/less • Distinguishing o Fact/option, cause/effect, relevant/irrelevant, valid/invalid • Identification of facts and definitions 5. Type X: requirement • Stem must be clear and unambiguous • Responses/ alternative/ branches/ options must be absolutely true or false. No shades of grey. 6. Type X: limitations (Linn & Gronlund, 1995) True False A B C D 1st year 2nd year 3rd year 4th year 5th year Repeated same MCQ test Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 23 • Difficult to construct unambiguous statement that measure complex learning outcomes (McAleer, 2001) • Most field has exceptions – unsuitable type X • Not especially useful beyond knowledge 7. Status of type X MCQ • Distinction between true or false not clear • To avoid ambiguity, writers are often pushed towards assessing recall of isolated facts • There is move away from this type (McAleer 2001, Anderson, 2004) • The NBME recommends not to use type X MCQs 8. Essay Question • Many variations • Traditional essay o Able to see ‘how a mind work’ o Problem in content specificity o Major problems in reliability Æ due to subjectivity o Overcoming Specify content (model answer) Multiple trained scorers Both difficult to achieve 9. The Modified Essay Question (MEQ) • Consist of a patient situation given successively • Students have to answer questions related to the situation • Presented as a booklet/ a series of sequential questions • Appealing in a problem-based learning • Features: o Students cannot go back to the previous questions/answers o Time guide provided o Assesses problem-solving and application of knowledge o In-depth knowledge can be assessed but loses in breadth of knowledge • Disadvantages Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 24 o Time-consuming to grade (hand-marked) o Content specificity o Greater number of invigilators needed to ensure compliance 10. Key-feature problem/ clinical reasoning exercise • Features o Short, multiple scenarios o Variable response formats • Promising research results for use to measure ‘clinical thinking’ of candidates 11. Extended Matching question (type R MCQ) • For example: Theme: Peripheral nerve supply to upper limb Options: a) Axillary nerve b) Radial nerve c) Ulnar nerve d) Median nerve e) Musculocutaneous nerve f) Circumflex nerve g) Suprascapular nerve h) Brachial plexus nerve i) Posterior interosseous nerve j) Spinal nerve C 8 Lead-in (instruction): For each of the given condition below, choose the most appropriate lesion to the nerve from the list above (stems) o Following an injury to the upper arm, the patient is unable to extend the elbow against resistance and has wrist drop. (d) Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 25 o A patient complains of paraesthesia of the thumb, index and middle finger of the right hand. On the examination the thenar eminence is wasted. (e) o A patient complains of progressive inability to extend his ring and little fingers on the left hand. On examination, these fingers are held in fixed flexion. (c) • Why choose EMQ? o Testing applied knowledge in contrast to recall knowledge o Better than MCQ because minimal cueing effect: recognitions, elimination, hints o Many options and options applicable to all vignettes or stems EMQ versus multiple true false MCQ EMQ MTF MCQ Applied knowledge Inclined on recall knowledge Testing matching and one best answer discriminatory ability Discriminatory ability contaminated by guessing Guessing has low correct probability Guessing has 50-50 probability No negative score penalty Require negative score penalty Can be scored by scanner Can be scored by scanner Wide and deep coverage based on question theme Usually wide coverage Integration well achieved Integration limited achieved New and hence difficult Established and hence easy Under one theme, 3 or more questions are developed Variable topics EMQ pros and cons Essay Short answer MCQ EMQ Application of knowledge Excellent Good Poor Good, can be improved with justification Assessment tool for higher order skill Excellent Good Poor Poor to good if justification is Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 26 required Coverage of topic Poor Good Excellent Excellent Reliability of score Poor Good Excellent Excellent Ease of scoring Poor Moderate Excellent Excellent Preparation time Minimal Moderate Maximal Moderate Total cost Large Moderate Low Low Cheating (sneak-a- peak) Most difficult Difficult Easy Easy unless justification required • How to write EMQ? o Determine the theme o Write the options o Write the lead-in o Write the stems as vignettes • Determine the theme o Diagnosis o Investigations o Causation o Management steps o Complications o Drugs o Clinical features o Prevention o Pathogenesis o Structure and function o Pathophysiology o Clinical correlation o Pharmacodynamics o Pathogens • Write the options o Short o No verbs o Each options clear entity Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 27 o 10 options required • Write the lead-in o Unambiguous o Relate to stem and options • Write the stem o Short, authentic description o Varying prototypically o Only one best answer o Use real life situation as a basis o Good distribution of presentation types • How popular is EMQ? o Sheffield University MBBS program o PLAB 1 – professional and linguistic Assessment Board o USMLE – US Medical Licensing Examination o PPSP Family Medicine MMed Program Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 28 ASSESSMENT OF “SHOW HOW” Miller’s Pyramid (1990) We will discuss more about OSCE (Objective, Structured, Clinical, Examination) OSCE DOES SHOWS HOW KNOWS HOW KNOWS SKILL CLINICAL AUTHENTICITY OBJECTIVITY Performance assessment Competency assessment OSCE, Long Case, Short Case, Written measurement (Vleuten et al, 2001) DOES SHOWS HOW KNOWS HOW KNOWS Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 29 1. Objective, Structured, Clinical (competence?) Examination • Developed in1979 (Harden & Gleeson, 1979) • Candidates rotate through a series of stations at which they carry out the task • All candidates are given the same task and judged by the same judges using the same preset standards • Not a method but a framework 2. Some Basic Ground Rules: • Multiple stations; each testing different competencies • Each station has a time limit • All candidates rotate through all stations • All candidates judged by the same preset standards e.g. o Checklist o Rating scale 3. The OSCE can test: • Skills o History-taking, physical examination, procedures, communication, interpersonal skills • Knowledge and understanding • Data interpretation • Problem-solving • Attitude (caution) 4. Using many methods • Short-case • Oral exam • Interpretation of laboratory results and images • Specimens • Diagrams • Standards patients…. Etc 5. OSCE strengths • Sampling o Higher than traditional methods Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 30 o Samples across all sources of variability 1 examiner for I station for all students better than 1 examiner for I student for all stations o Role of blueprinting and number of stations • Objective o Objectivity vs. objectification Objective not equal with reliable Global ratings vs. checklist • Authentic o Role of blueprinting • Educational impact o Caution: can be negative or positive impact 6. OSCE disadvantages • Compartmentalization of the skills • Sampling is critical • Resource intensive o Facilities o Training 7. OSCE blueprinting • Critical to ensure sampling of competencies • Can be simple list to complex grid • According to role/competencies expected Blueprint: determining competencies Take relevant history Perform relevant examination Communicates effectively with patient Able to interpret investigations CARDIOVASCULAR SYSTEM Bluish discoloration of lips SP of mother complaining about baby Chest pain Examine for signs of risk factors for heart disease on SP/patient Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 31 Shortness of breath Advises SP on taking sublingual GTN Fever and joint pain Reading CXR with cardiomegaly Calf pain on walking √ RESPIRATORY SYSTEM GASTROINTESTINAL SYSTEM 8. Developing stations • Station types o Marker stations Students answer written questions, recode findings or interpret patient data Does not need observer o Examiner stations Observer scores student performance Usually involves SP or mannequin Global rating scales • Broad categories for marking • Use of expert raters • Discriminates better than using checklist Checklist • Danger of trivilisation/atomization • Use of non-expert • Stations o Problem of content specificity Large number of stations required (around 20) o Time per station: 5 – 30 minutes o Achieving acceptable reliability of 0.8 4 to 8 hours of testing needed Overcoming Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 32 • Combination with other formats • Even with written tests 9. End notes: • It needs to emphasized that o The OSCE does not replace all other examination formats o Knowledge testing and writing skills are still more efficiently tested in written examinations o There is still a place for other ‘traditional formats’ Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 33 ASSESSMENT OF “DOES” MILLER’S PYRAMID (1990) 1. A competent health professional • A competent doctor DOES SHOWS HOW KNOWS HOW KNOWS SKILL CLINICAL AUTHENTICITY OBJECTIVITY Performance assessment In vivo: everyday practice Competency assessment OSCE, Long Case, Short Case, Written measurement (Vleuten et al, 2001) DOES SHOWS HOW KNOWS HOW KNOWS Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 34 • A competent nurse • A competent medical assistant • Competent? o Knowledge? o Skill? o Attitude? o What they do? 2. It is also said that “DOES” is for competency assessment… some questions regarding correlation between “SHOWS HOW” and “HOW” • Does performance in “shows how” correlate with “does”? o Majority research indicates no correlation. • Does performance in “does” correlate with postgraduate performance? o More towards no correlation • How can we assess performance at work? • How can we assess the non-clinical competencies? • How can we design a balanced assessment regarding their utility? 3. Clerkship and Apprenticeship model in “DOES” • Research finding of clerkship: o Far fewer patient contact than assumed o Considerable time spent on non-educational activities o More educational contact with HO, registrars and fellow students; less with senior staff o Rarely being observed during patient contact o Little consensus on clerkship objectives o Wide gap between previous theoretical learning and clerkship teaching. • All of this happen due to lack of educational structure 4. Utility of assessment procedures • Reliability o Content specificity o Importance of sampling; need increased sampling o Need increased observers Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 35 o Observation Increased numbers Concrete behaviour Specific time • Validity o Competency: difficult to define o What is asked is more important (stimulus format) o No single methods is ideal • Educational impact o Assessment influences learning o Deliberate choice and use of methods o Importance of feedback • Acceptability o Role of education, training and staff development • Cost o Quality assurance Format related Content related Standard setting – standard for fitness of practice Statistical o Better done by central o Expensive but important 5. Clerkship assessment: • Why? o Summative? Formative? Program evaluation? Emphasis: formative • High quality needed for summative • Selection/summative should be at earlier stage • Lack of educational structure of clerkship Needs high feedback component Final weight (summative/formative) to be agreed upon • What? o Focus on content Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 36 Which medical problems students would encounter during clerkship rotation Benefits of clarifying content: guidelines for both students and teachers Æ enhancing educational structure Importance of core content identification and blueprinting o Evaluation of attitude Attitude: propensity/tendency for certain behaviour Minimize ‘psychologising’, maximize stating concrete and clinically meaningful actions Who defines ‘appropriate’ attitude/behaviour? • When? o At the end? Logical for summative purposes But no time for students to remedy their patient encounters o At the beginning? o Midway? o At both ends? • How? o Do present methods test what we really want to test? Patient-based examinations • Short and long case Observations Relatively newer methods • OSCE using SPs • Longitudinal video taping of undercover SPs Again Æ role of blueprinting o Observations More direct More often More observer • Who? o All partners Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 37 Nurses, patient and etc o Separate role of teacher and assessor as much as possible Especially in assessment more summative in nature o Need for more collaboration Systematic and planned • Central planning and coordination • Saves cost 6. Some review: • Does performance in “shows how” correlate with “does”? o Majority research indicates no correlation. • Does performance in “does” correlate with postgraduate performance? o More towards no correlation • How can we assess performance at work? • How can we assess the non-clinical competencies? • How can we design a balanced assessment regarding their utility? • Professional behaviour and its assessment • Assessment of performance at work Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 38 PERFORMANCE ASSESSMENT 1. MILLER’S PYRAMID (1990) 1. Performance of assessment at work: • Basis o Patient outcomes Mortality and morbidity • Ultimate measure of accountability • Reassurance to public • Fairest measure of individual competence • Basis for identification of effective doctor • For patient Æ basis for choosing doctor DOES SHOWS HOW KNOWS HOW KNOWS Competence assessment Performance in vivo: everyday work Performance assessment Graduation: HERE?? Graduation: OR HERE?? Assumption: 1. Doctors are competent enough to work with patient immediately after graduate 2. They would remain competent by taking postgraduate course NOT SO!! At the same time: 1. Public demands for accountability and quality Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 39 Other outcomes • Physiological measures (e.g. BP) • Clinical event (e.g. stroke) • Symptoms (e.g. shortness of breath) • Patient satisfaction • Cost-effectiveness o Process of care Processes • Screening (e.g. lipid disorder) • Others: o Preventive services o Immunization o Patient education o Counseling • Disease-specific processes (e.g. foot and eye examination in Diabetes) Advantages • Within doctor’s control • Fits in well with quality improvement programs Disadvantages • The right process does not guarantee the best outcomes for patients o Volume Volume: how often service is provided Advantages • Data easy to obtain • Comparison among doctor easy Disadvantages • High volume does not mean doing the right things • Sources of information o Clinical practice records From patient records Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 40 Disadvantages • Expensive • Time consuming • Records often incomplete and eligible Role of electronics record system o Administrative databases Advantages • Information regarding outcomes, process and volume • Easily available, inexpensive and based on large population of patient and doctor Disadvantages • Information limited to demographics, diagnosis and procedures Best used as part of screening process of doctor o Dairies or case logs E.g. doctor logbook • Which procedures, who observed, complications, patient information and adequacy of procedure Reasonable for volume data, not for process and outcomes Doctors sometimes record only chosen patients o Observation 4 issues • Experience of observer should match the judgment expected • The more observer the better • Training observers necessary • Relationship between the observer and the observed may influence validity (Halo effect) Best suited for information regarding process • Threats to validity and reliability o Patient mix Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 41 Different composition of patients • Between specialties/departments • Within specialties/departments Focus on certain conditions (tracer conditions) • Common • Doctors can make a difference Disadvantages • Incomprehensive picture • Some conditions important but infrequent o Patient complexity Variability of patients within same condition • Severity of illness • Co-morbidity • Other undocumented problems Overcoming • Exclusion criteria • Risk adjustment o Attribution More patient are being managed by a team Overcoming • Inclusion criteria • Focus on process known to correlate with outcome (e.g. Diabetes Æ routine monitoring HbA1C) o Number of patient Related to case specificity Some studies recommend 100 patients Limits to common conditions Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 42 ASSESSMENT OF PROFESSIONAL BEHAVIOUR 1. The competent physicians? • Negligence • Malpractice • Public complaint – getting more and more nowadays o Disrecpect, mistrust and miscommunication are the major elements relating to the complaints. • The question is WHY? o Doctors not clear of their roles? o Doctors are not taught their roles? • Professional behaviour o Should it be taught? o Attitude: can it be taught? • Assessment of professional behaviour o What is the role of assessment? o Issues in assessment? 2. Definition of Professional Behaviour • ABIM (1990) o 3 commitments To the highest standards of excellence in the practice of medicine To sustain the interests and welfare of patients To be responsive to the health needs pf the society o 6 elements Altruism, accountability, excellence, honor, integrity and respect for others • Epstein and Hundert’s dimensions (2002) o Cognitive o Technical o Integrative o Context o Relationship Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 43 o Affective/moral o Habits of minds • Medical Professional in the new millennium – a physician charter (2002) ABIM, ACP-ASIM, EFIM: o Professional responsibilities Commitment to professional competence Honesty with patient Patient confidentiality Maintaining appropriate relations with patients Improving qualities of care Improving access to care Just distribution of finite resources Scientific knowledge Maintaining trust • Themes o Identifying concrete behaviour o Developing educational plans and activities in teaching and assessing o Recognizing the role of relationships in the systemic dimension of professionalism 3. Assessment in Professional Behaviour • Instruments to track aspects of attitudinal change over time • Early detection of unprofessional behaviour o Are there predictive factors? • Role of assessment o Positive educational impact o Meaningful o Accurate o High feedback value • Most direct Æ observation • Problems related with observation method • Problem in giving feedback 4. Methods used Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 44 • Observation o Behavioral elements: How student deals with other persons How student deals with himself Communication skills Certain personal qualities o Observations over long time, regular, in different situation o Training for observers o Observation – condition for assessment Reliability • Situation o Frequency and duration (intra-observer) • Judge o Informed and trained; multiple sampling • Rating scale o Clear and easy to mark Validity • Situation o Relevant for future practice • Judge o Qualified and personally observe • Rating scale o Focus on professional behaviour; steers students positively, discriminate between adequate/not adequate Acceptability • Student o Criteria of assessment also useful for feedback; time to change given • School o Obtain useful information to support/sanction (block) student; efficiently o 360o evaluation Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 45 Rating form filled by different raters who interact with the assessee during the course of work/education Raters • Clinician, therapies, social worker, case manager, psychologist and etc. • OSCE and simulated patients o Communication skills, humanism, empathy o Some instruments e.g. communication skills, extensively researched o Pros and cons? (please refer to assessment of “shows how”) • Peer review o Teamwork, evaluation/feedback skills o Favorable study results • Portfolio assessment o Evidence of student progress offered by student o Includes clinical experiences, reflective journals, etc. 5. Issues in assessment • Content specificity o Cannot generalized one task to decide professional/unprofessional behavior • Idiosyncrasy o There are many ways/methods that give the same result • Amount of structuring • Behaviour: competence or performance? o Continuity of behaviour important Æ performance assessment • Assessment form a ‘competence’ approach o Often single/infrequent sampling o Increased structuring Possible overstressing of knowledge Induction of test-taking strategies o Increasing reliability of subjective methods Sample all possible sources of bias Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 46 • Multiple occasion • Multiple judges Judgment based on specific observations • Global judgment from specific observation (ok) • Global judgment from memory over the long time (not ok) • Important for good feedback o Professional behaviour A goal by itself, or a means to achieve an end? • It is not a goal but it is merely as a means to achieve the end which the medical professional eventually understand what are their roles as a medical professional… Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 47 STANDARD SETTING & SCORING PROCEDURES 1. Determining the cut-off score. • How? • Any objective criteria? • The ‘truth’ is not ‘out there’ o “…even the most rigorous standard-setting method, followed meticulously, will be somewhat arbitrary… however, they be should be credible.” – Schnidler, Corcoran & DaRosa, 2006 – o Credible standards share 3 important characteristics: Set by appropriate numbers and types of judges Utilize appropriate method Produce reasonable outcomes - Norcini & Guille, 2002 - 2. Credible standards: • Judges 100% Able to do all tasks that sample competency 0% Unable to do tasks that sample competency PERFORMANCE KNOW CAN DO SAFE DON’T KNOW CAN’T DO UNSAFE Cut-off score BORDERLINE STUDENT Has a 50:50 probability of passing or failing the test Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 48 o Content experts o Know the target population o Understand the task and assessment tool o Fair-minded o Willing to follow directions o Give full attention to the process o Demographically diverse to avoid bias o 5 to 6 considered minimum • Appropriate method o Produces standards that are consistent with the final purpose of the test o Relies on informed expert judgment o Demonstrates due diligence (demonstrates standard of method) o Is supported by a body of evidence o Easy to explain and implement (Norcini & Guille, 2002) • Reasonable outcomes o Compare with historical standard/external measure o Consider stakeholder opinion (Norcini & Guille, 2002) 3. Methods available • Angoff o Judges estimate performance of borderline student on each item o Judges try to answer the question for e.g. MCQ like borderline student answer the question (judges mimic the borderline student) o Mean estimate are combined to produce passing score o The passing score is determined before the actual performance (predetermined passing score) • Ebel o Matrix created for test items Item difficulty (actual performance data) Item relevance (judged) o Judges borderline performance for each category o Passing score calculated Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 49 o Matrix created before the actual performance and passing score calculated after actual performance • Hofstee o Judges decide Min and max acceptable pass score Min and max acceptable fail rate o Plotted and compare with actual performance (fail rate vs. pass score) o Passing score set where cumulative freq dist crosses this bracketing triangle • Borderline o Judges observe performance o Rate using checklist as well as pass/borderline/fail o Mean score of ‘borderline’ is passing mark Difficult Medium Easy Essential Important Acceptable score Fail rate Passing Score Junction between frequency curve and triangle line bracket Triangle line Frequency curve line Min & Max passing score line Min & Max fail rate line Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 50 • Contrasting groups o Judges observe performance o Rate using checklist as well as pass/fail o Intersection of ‘pass’ and ‘fail’ score distribution is the passing mark. Miller’s Pyramid (1990) 4. Scoring Strategies • Combination of test means test battery • Interpretation result means o Scoring strategies o Related to validity (meaningfulness of score interpretation) • Validity evidence o Content o Reliability o Consequential validity • Types of scoring strategies o Compensatory Sum of scores over battery of tests compared against a standard Æ pass or fail • Multiple measures required OR Borderline/contrasting group Angoff Intersection of ‘pass’ and ‘fail’ score distribution Fail distribution score Passl distribution score Knows Shows How Knows How Does Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 51 • Minimum performance on ANY measure Æ complementary Can fail in 1 or more tests in battery but still pass Usage of strategy • Content of experts decide that low performance in any single trait can be tolerated • Total score meaningfully reflects the construct Reliability of total score high Risk of false positive Less diagnostic value o Conjunctive Nonsequential • Each test in battery important Æ must pass separately • Less sampling of each trait Æ lower reliability o Enough for high-stakes decision? • Risk of false negative • More rigorous and demanding o Effect on student o Appealing from legislator/authority Sequential • A series of conjunctive decisions in the test battery for example FRCP examination must pass part I then only can proceed with part II • If fail any test Æ does not proceed • Often combined with opportunity to repeat test • Strategy useful if battery of test is o Time-consuming o Expensive • May minimize false-negative decisions (failing those who deserve to pass) o Disjunctive @ complementary Provides a series of equivalent testing alternatives Any of parallel tests valid for making pass/fail decision Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 52 Emphasizes equivalence of test forms across testing sessions e.g. annual licensing body Allows using initial test as basis for planning improvements for later test Æ disjunctive test should also provide diagnostic information Can be quite expensive Can be combined with compensatory/conjunctive strategies 5. Summary of strategies Validity evidence Compensatory Conjunctive Content Implies overall performance Emphasizes distinct traits/components Reliability Score High due to increased sampling Low due to reduced sampling Rater Same Same Decision - High - Risk of false positive - Lower - Risk of false negative Consequential - Less demanding; may induce negative learning behaviour - Loss of diagnostic information - Highly demanding; may induce negative stress - More diagnostic information 6. Summary by Situation Situation Compensatory Conjunctive Disjunctive/ Complementary Measure of different construct - Increased rigor - low Reliability - False negative Different measures for same construct - Different exams/ item/ types/ times - false positive - Validating / confirming inferences - //////////// Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 53 Multiple opportunities - Minimizing false negative Accommodation and alternate assessments - ///////////// 7. Principles • The manner of combining multiple measures is as important as the measures themselves Should be driven by values to be promoted • Multiple measures does not necessarily mean higher reliability Especially for conjunctive strategy Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 54 QUALITY CONTROL IN ASSESSMENT The Assessment Loop 1. Guiding Principles • Assessment is harmonious with overall principles and intended outcomes • Design an overall assessment strategy: o Assessment of clinical competence and professional behaviour o Mix of assessment procedures o Utility of assessment tools Reliability Validity Acceptability Educational impact Cost o Balance of summative and formative methods o Utilizes appropriate scoring strategies o Involves students Self-assessment/peer-assessment, etc o Broad assessment time frame Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 55 2. Pre-assessment procedures • Blueprinting • Pre-established guidelines for question formation • Feedback loop • Interdisciplinary review o Clarity/ambiguity o Factual accuracy o Etc • Formal faculty training • Item banking • Standard setting • Student briefing • Administrator 3. Intra-assessment procedures • Examinations security • Fairness • Administration of questions • External examiner 4. Post-assessment procedures • Item analysis • Student participation o Flagging of faulty items o Decision affecting faulty items • Marking and scoring • Appeal procedures • Reporting o To student o To school o To departments Student Assessment in Medical & Allied Health Schools Dr Saiful’s notes on Medical Education 56 • Decisions o Affecting policy, objectives, aims, etc 5. Research agenda: • Assessment as a programmes o What constitutes good assessment programme? o Which building block should be included? o Who should do what? o How should information be collated (combined): qualitatively/quantitatively? • Quality elements of multi-modal assessment o How to “score” qualitative judgments? o Identifier for profile scores o What influences the thinking of judges? o Fraud/plagiarism (internet) o Flexible adaptation of teaching to assessment outcomes • Consequential validity/educational impact o Roles of format, content, scheduling and regulations o What influences teacher behaviour • Research into assessment programmes o How to change a programme? o Stakeholder involvement? o How are the instruments combined? o How to use parts instead of whole instruments?
