- Paul Di Capua (paul.dicapua) [Forum Discussion]

Introduction

In a cohort of 80 undiagnosed patients presenting to a primary care center, the medical interview alone yielded the correct diagnosis in 76% of visits.¹ Another study found that while diagnostic imaging was accurate in only 35% of cases, the combination of the medical history and physical examination produced the right diagnosis in 70% of cases.² Physicians facing malpractice claims are less likely to use patient‐centered interviewing than are physicians without malpractice claims.³ Moreover, patients are significantly less satisfied with their visits when physicians ignore psychosocial aspects of their care.⁴ As Holmboe argues, clinical skills, defined as the medical interview, physical examination and counseling, “remain the most important and effective diagnostic and therapeutic tools.” ⁵ The most important change required of health care education is ensuring all providers are equipped with high quality clinical skills; this can be achieved by the implementation of a quality improvement mechanism in health care education.

Quality improvement is the fundamental driver of this proposal. Lessons from industrial manufacturing on quality teach us the value of observation and feedback in continually improving quality. However, the current system of health care education lacks structure and yields graduates with low quality clinical skills. This essay proposes the implementation of a system of observation and feedback as means of assessing and improving providers’ clinical skills. For medical education, this can be achieved through the implementation of the mini‐CEX, a validated evaluation tool discussed in greater detail below. I support my argument by describing my own experience of implementing the mini‐CEX at a student‐run clinic at my medical school. A significant shift towards quality improvement in education may result in a cultural change in medicine in which providers continually help each other improve their care of patients.

Quality: A brief history with instrumental lessons

Quality in health care has taken on its own definition, focusing on technical outcomes such as hemoglobin A1C levels, door‐to‐cath times, and so on. However, “quality” was defined for industrial purposes by Joseph Juran as “fitness for use” and by Philip Crosby as “conformance to requirements.” In industry, these definitions lay the foundation for the theory of quality, which seeks to minimize errors and to create systems of production or service which conform to set standards. The notion of high quality, for which firms such as Toyota or the Ritz Carlton have become famous, set narrow standards with minimal variation to consistently achieve remarkable results.

Every process exhibits variation. The quality gurus of the post‐WWII era such as Dr. W.E. Deming and Joseph Juran sought to minimize variation to achieve higher quality. In the process of helping Japanese auto makers establish their superiority in producing high quality cars, Dr. Deming summarized his expectations of his clients in his “Fourteen Points for Transforming Management Effectiveness” (see Table 1).

Quality had to be built in to the system. A rigorous system could be achieved through “modern methods of supervising,” which allowed for constant observation and feedback on how to improve the process. Although a high quality system produced minimal variation, the goal to “constantly and forever improve the systems of production and service” continually yielded better results.
By 1986 Deming predicted that it would take 30 years for Americans to match the progress of the Japanese and that the United States was still falling behind. Today, in 2009, Deming might be smirking if he could see the United States auto manufacturers facing the possibility of bankruptcy as Americans continue buy more imported cars.

The implications for the American health care system, and for medical education in particular, mandate a more rigorous quality control in our system of education. High quality health care implies setting rigorous standard requirements for graduating health care providers such that future patient care providers are reliably fit to provide the high quality care demanded from the American public.  Purposeful quality improvement processes are necessary in order to reliably produce high quality
physicians, lest we too fall 30 years behind and face a future comparable to the Big Three.

Quality in medical education: filling the educational gap

As the national discussion on health care continues to call for higher quality of care, attention has turned to medical education in determining whether future physicians are adequately prepared to provide the care the public deserves. Sadly, this is currently not the case.

Graduating medical students entering residency have shown an alarmingly wide variability in clinical skills,⁶ and residents are commonly assumed to possess skills they do not have.^{7, 8} The Association of American Medical Colleges (AAMC) visited 97 medical schools and found that faculty rarely observed medical student patient encounters; instead evaluators’ impressions of the students’ presentation skills and knowledge formed the basis of their evaluations.9 And despite the widespread call for higher quality health care, lower costs and greater patient safety, the medical literature evidences a general paucity of studies examining the effectiveness of clinical teaching or how to measure its long‐term outcomes.^10‐12

Moreover, residents often point out that as medical students they were told that they would learn certain skills as residents, but as residents they are expected to have learned them in medical school.⁶ Without a well‐organized infrastructure to ensure that all trainees are well‐equipped with the necessary clinical skills to distinguish them as high quality providers, improvised clinical teaching instead results in haphazard results revealed by wide variability, and assessments based on impressions and indirect
evaluations. The widely variable results of the medical education process evidence a poor‐quality system.

One of the fourteen points of Dr. Deming’s program states, “Cease dependence on mass inspection.” Inspection implies planning on defects, at which point it is too late and ineffective. Instead, Deming insisted, processes must be improved so that the product or service can be guaranteed to be of high quality and render inspection superfluous. Nonetheless, the only current quality control mechanism of clinical skills in medical education is the USMLE Step 2 Clinical Skills Exam taken medical students shortly before graduation.

In the clinical years of medical school as with residency, a large part of learning the critical skills which distinguish an average physician from a high quality physician are taught ad hoc, with scant processes in place to assess and improve trainees’ clinical skills. While the science of medicine redefines the frontier of human knowledge each day, the education of patient care has hardly budged from the centuries‐old apprenticeship model.

Without a structured mechanism of supervision to continually assess and evaluate the quality of providers’ clinical skills, it is impossible to ensure any level of clinical competency until the “final inspection” of the national boards before graduation. And without a purposeful mechanism to teach trainees, students and teachers depend on the “right situation” to arise to provide a learning opportunity.

Handfield‐Jones, et al propose a model that links assessment with practice performance and learning.¹³ In this model, positive feedback validates actual practice, driving the trainee closer to the “desired practice”; discrepancies between desired and actual practice are identified as an “educational gap” to be addressed through contemplation and “educational action.” Filling this “educational gap” begins by identifying it, which is possible only by observing current actual practice. Once identified, teachers can fill in this gap through “educational action”: feedback to students. Without both observation and feedback, the educational gap will persist, and clinical skills will never improve.

The miniCEX: the mechanism for quality improvement

The mini‐CEX is a tool that uses observation and feedback to assess and teach trainees clinical skills (see Figure 1). The American Board of Internal Medicine developed the mini‐CEX as a means to efficiently evaluate competence in clinical skills of residents over a number of cases of variable complexity in various settings.¹⁴ Over the span of the training period, the trainee has multiple opportunities for observation and feedback, and should therefore accumulate several evaluations from different faculty with a variety of patients with a broader spectrum of clinical problems and required skills. The scores produced by the mini‐CEX have been noted to be more reliable than those based on the traditional CEX as an evaluation tool.¹⁴

Previous studies have established the reliability, validity and educational impact of the mini‐CEX.^14‐20 The mini‐CEX has been previously used in the instruction of medical students.^{21, 22} However, its use on short, individual medical student clerkships limits the ability of students to take advantage of serial feedback to continually hone their skills over months or a year. In order to fully harness the power of serial feedback on quality improvement, the mini‐CEX should be implemented longitudinally, ideally over a student’s entire medical education, if not at least over an entire year. The mini‐CEX therefore provides a mechanism for achieving the desired high quality standards of clinical skills which all physicians should have upon entering their own practice.

Implementation of the miniCEX

In order to test the potential utility of this proposal to use the mini‐CEX to improve the quality of clinical skills in medical education, I took the initiative to implement the mini‐CEX at a student‐run clinic at my medical school in which I participated as a provider.

Upon agreeing that the development of clinical skills was a primary objective which was not sufficiently addressed in the operation of the clinic, the students agreed to incorporate the mini‐CEX. This was not a symbolic agreement: students agreed to change the normal operations of the clinic to incorporate the mini‐CEX. We then ran a training session for faculty members to learn the art of observation, evaluation and feedback. Although the details of the study are beyond the scope of this essay (manuscript in progress), over the summer of 2008 we demonstrated the acceptability and feasibility of implementing
this validated and reliable tool with proven educational impact in our clinic.^14‐20

A variety of previous studies have examined the feasibility of implementing the mini‐CEX in a variety of contexts.^{14, 15, 17, 18, 20‐25} However, this study was the first known implementation of the mini‐CEX in a longitudinal setting with medical students. Furthermore, this was the first reported implementation of a mini‐CEX implemented by students to assess themselves. Its success should largely be attributed to students, who were invested in maintaining this feedback mechanism throughout the trial. At our clinic we are starting to see a cultural change in the approach to observation and feedback.

Self-regulated quality control

Physicians are remarkably inaccurate in self‐assessing their strengths and weaknesses.²⁶ Handfield‐Jones et al suggest a paradigm for learning over a career that acknowledges the importance of continuous improvement, dependent on a systematic approach to “practice generated identification of learning needs.”¹³ A model that allows for self‐directed continuous improvement requires clinical assessment and correction must begin during training.¹³

Creating a culture where observation and feedback are routine could have drastic changes on the practice of medicine. M&M rounds in surgery are opportunities for feedback from colleagues. But these cases are typically the most drastic, and each surgeon only gets a few opportunities to present her mistakes each year. Moreover, by only reviewing their most egregious mistakes, surgeons rarely get feedback on what they are doing well, or smaller adjustments that may improve skills but would not result in an M&M presentation. Nonetheless, M&M rounds provide a narrow window into what selfregulating quality improvement can look like. Systemic implementation of the quality improvement measures such as the mini‐CEX may result in a next generation of physicians who are accustomed to routine assessment and evaluation of their clinical skills by their colleagues.

The implications of quality improvement on patient care

Evidence suggests that any feedback is better than no feedback at all, and management literature suggests that supervision, which implies both observation and feedback, can be seen as a form of quality assurance.10 The direct benefits of effective supervision in clinical settings are well‐documented, as reviewed by Kilminster and Jolly.10 For example, supervision has been shown to result in better patient outcomes, while lack of supervision is harmful to patients. Supervision seems to facilitate trainees learning new skills, and trainees are able to identify many gains from supervision.

Multiple studies have documented a perceived gap by residency graduates between the skills required in clinical practice and the training they received several years after they had graduated from residency.^27‐29 Simply stated, this implies that what they did not learn during their training, they learned on the job. Undoubtedly learning under supervision is safer for patients; what mistakes were made along the way while clinicians fill those gaps? The implications of better clinical skills development during training have longer‐term implications for patient safety beyond training.

Furthermore, as health care becomes increasingly standardized and regulated, providers will be assessed on their clinical competence. An assessment system created by health care providers will translate to an assessment system that best serves providers and their patients, as opposed to the many other powerful stakeholders in the health care industry. Acting now to implement this change may ensure that providers retain the power of dictating their own standards of competency.

Conclusion

Providing high quality patient care depends on providers with strong clinical skills. The literature evidences that clinical education in US medical schools lacks structure, with a high degree of variance in the clinical skills of graduating physicians. All providers interact with patients, and so each provider must master a set of clinical skills to add value to their team. The change proposed in this essay provides a mechanism for achieving these higher quality standards of clinical skills among physicians.

This proposal has three corollaries which suggested an even greater upside from making this change. First, observation and feedback will not only ensure a minimum quality of clinical skills, but moreover allows for continuous improvement. Through continuous structured feedback, each individual will be able to realize his fullest potential as a clinician; a paradigm shift from the embarrassingly low minimum bar set by the USMLE Step 2 CS exam for physicians. Second, teams of providers will be able to rely on each other, thereby increasing coordination of patient care. Finally, systemic implementation of quality improvement feedback may result in a cultural shift in which health care providers seek to help each other provide the best patient care. Dr. Deming would proudly admire teamwork that seeks to constantly and forever improve our patient care.

Table 1: Deming’s 14 Points¹

1. Create constancy of purpose for improvement of product and service. Management
must change from a preoccupation with the short run to building for the long run. This
requires dedication to innovation in all areas to best meet the needs of customers.
2. Adopt the new philosophy. Shoddy materials, poor workmanship, defective products,
and lax service must become unacceptable.
3. Cease dependence on mass inspection. Inspection is equivalent to planning for defects;
it comes too late and is ineffective and costly. Instead, processes must be improved.
4. End the practice of awarding business on price tag alone.
5. Constantly and forever improve the systems of production and service. Waste must be reduced and quality improved in every activity: procurement, transportation, engineering, methods, maintenance, sales, distribution, accounting, payroll, customer service, and manufacturing. Improvement, however, does not come from studying the defects produced by a process that is in control but from studying the process itself. Most of the responsibility for process improvement rests with management.
6. Institute modern methods of training on the job. Training must be restructured and centered on clearly defined concepts of acceptable work. Statistical methods must be used for deciding when training has been completed successfully.
7. Institute modern methods of supervising.
8. Drive out fear.
9. Break down barriers between departments.
10. Eliminate numerical goals for the work force. Targets, slogans, pictures, and posters urging people to increase productivity must be eliminated. Most of the necessary changes are out of workers’ control, so such exhortations merely cause resentment. Although workers should not be given numerical goals, the company itself must have a goal: never‐ending improvement.
11. Eliminate work standards and numerical quotas.
12. Remove barriers that hinder the hourly workers.
13. Institute a vigorous program of education and training.
14. Create a structure in top management that will push every day on the above 13 points.

¹ Deming’s original words in bold, summary explanation of relevant points taking from “A Note on Quality.” 30.
A Note on Quality: The Views of Deming, Juran and Crosby”. Harvard Business School Case Study. Revised
February 28, 1990 1990(Case #: 9‐687‐011).

Figure 1: The mini-CEX

Works Cited

1. Peterson MC, Holbrook JH, Von Hales D, Smith NL, Staker LV. Contributions of the history, physical examination, and laboratory investigation in making medical diagnoses. West J Med. Feb 1992;156(2):163‐165.
2. Kirch W, Schafii C. Misdiagnosis at a university hospital in 4 medical eras. Medicine (Baltimore). Jan 1996;75(1):29‐40.
3. Levinson W, Roter DL, Mullooly JP, Dull VT, Frankel RM. Physician‐patient communication. The relationship with malpractice claims among primary care physicians and surgeons. Jama. Feb 19 1997;277(7):553‐559.
4. Roter DL, Stewart M, Putnam SM, Lipkin M, Jr., Stiles W, Inui TS. Communication patterns of primary care physicians. Jama. Jan 22‐29 1997;277(4):350‐356.
5. Holmboe ES. Faculty and the Observation of Trainees’ Clinical Skills: Problems and Opportunities. Academic Medicine. 2004;79(1):16‐22.
6. Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO. Assessing residents’ competencies at baseline: identifying the gaps. Acad Med. Jun 2004;79(6):564‐570.
7. Learning objectives for medical student education‐guidelines for medical schools: report I of the Medical School Objectives Project. [Article]: Academic Medicine January 1999;74(1):13‐8.
8. Moercke AM, Eika B. What are the clinical skills levels of newly graduated physicians? Self-assessment study of an intended curriculum identified by a Delphi process. Med Educ. May 2002;36(5):472‐478.
9. Pulito AR, Donnelly MB, Plymale M, Mentzer RM, Jr. What do faculty observe of medical students’ clinical performance? Teach Learn Med. Spring 2006;18(2):99‐104.
10. Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: a literature review. Med Educ. Oct 2000;34(10):827‐840.
11. Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. Sep 2006;28(6):497‐526.
12. Reilly BM. Inconvenient truths about effective clinical teaching. Lancet. Aug 25 2007;370(9588):705‐711.
13. Handfield‐Jones RS, Mann KV, Challis ME, et al. Linking assessment to learning: a new route to quality assurance in medical practice. Med Educ. Oct 2002;36(10):949‐958.
14. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini‐CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. Nov 15 1995;123(10):795‐799.
15. Alves de Lima A, Barrero C, Baratta S, et al. Validity, reliability, feasibility and satisfaction of the Mini‐Clinical Evaluation Exercise (Mini‐CEX) for cardiology residency training. Med Teach. Oct 2007;29(8):785‐790.
16. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. Aug 2003;78(8):826‐830.
17. Malhotra S, Hatala R, Courneya CA. Internal medicine residents’ perceptions of the Mini‐Clinical Evaluation Exercise. Med Teach. 2008;30(4):414‐419.
18. Nair BR, Alexander HG, McGrath BP, et al. The mini clinical evaluation exercise (mini‐CEX) for assessing clinical performance of international medical graduates. Medical Journal of Australia. Aug 4 2008;189(3):159‐161.
19. Norcini JJ, Blank LL, Arnold GK, Kimball HR. Examiner differences in the mini‐CEX. Adv Health Sci Educ Theory Pract. 1997;2(1):27‐33.
20. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini‐CEX: a method for assessing clinical skills. Ann Intern Med. Mar 18 2003;138(6):476‐481.
21. Hauer KE. Enhancing feedback to students using the mini‐CEX (Clinical Evaluation Exercise). Acad Med. May 2000;75(5):524.
22. Kogan JR, Bellini LM, Shea JA. Implementation of the mini‐CEX to evaluate medical students’ clinical skills. Acad Med. Nov 2002;77(11):1156‐1157.
23. Howley LD, Wilson WG. Direct observation of students during clerkship rotations: a multiyear descriptive study. Acad Med. Mar 2004;79(3):276‐280.
24. Wilkinson J, Crossley JGM, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace‐based assessment across the medical specialties in the United Kingdom. Medical Education. 2008;42(4):364‐373.
25. Torre DM, Simpson DE, Elnicki DM, Sebastian JL, Holmboe ES. Feasibility, reliability and user satisfaction with a PDA‐based mini‐CEX to evaluate the clinical skills of third‐year medical students. Teach Learn Med. Summer 2007;19(3):271‐277.
26. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self‐assessment compared with observed measures of competence: a systematic review. Jama. Sep 6 2006;296(9):1094‐1102.
27. Card SE, Snell L, O’Brien B. Are Canadian General Internal Medicine training program graduates well prepared for their future careers? BMC Med Educ. 2006;6:56.
28. Mandel JH, Rich EC, Luxenberg MG, Spilane MT, Kern DC, Parrino TA. Preparation for practice in internal medicine. A study of ten years of residency graduates. Arch Intern Med. Apr 1988;148(4):853‐856.
29. McPhee SJ, Mitchell TF, Schroeder SA, Perez‐Stable EJ, Bindman AB. Training in a primary care internal medicine residency program. The first ten years. Jama. Sep 18 1987;258(11):1491‐1495.
30. A Note on Quality: The Views of Deming, Juran and Crosby”. Harvard Business School Case Study. Revised February 28, 1990 1990(Case #: 9‐687‐011).