➢ The apprenticeship model has been the basis of orthopaedic training for decades. New expectations regarding the demonstration of abilities, knowledge, and professionalism require better methods of education and evaluation.
➢ There is a concern that operative and physical examination skills are not being measured objectively. The Surgical Skills Modules for PGY-1 residents, The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE), Objective Structured Assessment of Technical Skills (OSATS), and musculoskeletal Objective Structured Clinical Examination (OSCE) all appear to have potential for training and evaluating residents and should be further evaluated.
➢ The Orthopaedic Surgery Milestone Project will require a marked increase in the documentation of medical knowledge and surgical skill among residents, corresponding with the Accreditation Council of Graduate Medical Education (ACGME) core competencies.
➢ Although more research is needed, the technology of simulation is improving rapidly and is approaching very meaningful use.
Orthopaedic surgeons traditionally have been trained with use of an apprenticeship model, with experience being measured on the basis of case log documentation and with competency being determined by senior mentors. Over the past decade, medical education has undergone a major paradigm shift, with increasing emphasis on teaching toward competence as a specialist, including operative skills, essential knowledge for practice, professionalism, and the use of evaluation methods thought to be credible, accurate, reproducible, and transparent. The primary use of an apprenticeship model may result in graduates never having performed some procedures or never having managed patients with certain diagnoses. Alternatively, the competency model may result in graduates performing well on certain standard procedures but perhaps not demonstrating competency on relatively rare entities. While the provision and assessment of medical education have become more complex in proportion to advanced technologies and novel procedures, delivering excellence in education itself has become increasingly difficult as a result of duty-hour restrictions, greater attention to patient safety, increased requirements for direct supervision, and a negative economic climate1.
The Accreditation Council for Graduate Medical Education (ACGME) has assumed the pivotal role of influencing the balance between competing expectations while simultaneously attempting to facilitate modernization of medical education. The ACGME has fostered innovation at the specialty and program levels and has provided direction and tools to improve the assessment and transparency of performance2. Currently, residents are required to demonstrate appropriate knowledge and performance in six core competencies, defined as patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. Residency programs are required to measure and document the acquisition of these skills as per the guidelines set forth by the ACGME and the American Board of Medical Specialties (ABMS).
As of July 1, 2013, the ACGME implemented The Orthopaedic Surgery Milestone Project3, with the support of the American Board of Orthopaedic Surgery (ABOS) and the Residency Review Committee for Orthopaedic Surgery (RRC-OS), which includes new requirements for training and the assessment of structured motor skills during basic orthopaedic surgery education. The milestones, which include knowledge, skills, attitudes, and other attributes for each core competency, were developed to evaluate the competency and progress of resident physicians in ACGME-accredited programs. The Next Accreditation System (NAS) will utilize the milestones to determine whether residents appear to be progressing as expected. This is truly an important foundational tool that will allow us to move toward competency-based orthopaedic education, which is distinct from our current system of time-defined educational goals.
Evidence in support of current methods to teach and assess knowledge of musculoskeletal medicine and operative skills remains largely anecdotal. Most of the available research on surgical education stems from the general surgery literature emphasizing “away from patient” learning involving simulation situations4-8. A handful of studies of orthopaedic surgical training have demonstrated the effectiveness of using simulations, mostly focused on arthroscopy skills9-17.
Without a strong research foundation built on educational research, orthopaedic education will continue to be based on untested assumptions and personal preferences, and assessment will continue to be subjective rather than objective. To our knowledge, no comprehensive literature review has been performed to highlight best educational practices in orthopaedic training. The American Academy of Orthopaedic Surgeons (AAOS) created the Orthopaedic Education Study Group with the goal of performing a comprehensive review of published research literature about orthopaedic education. The study group was charged with determining current status, recommending best practices, and identifying potential research questions. Within the study group, a resident/fellow workgroup was created with the aim of determining gaps in orthopaedic graduate medical education and suggesting an agenda of research initiatives to improve the quality of programs, educational practices, and assessment methods.
The AAOS invited orthopaedic surgeon members to participate in the Orthopaedic Education Study Group. A non-surgeon faculty member from the Department of Medical Education at the University of Illinois College of Medicine guided the literature review with the aid of a research assistant and librarian.
The study group worked from March 2012 through September 2013. All articles published in peer-reviewed journals involving orthopaedic education research in North America (United States and Canada) and written in English (or, at the minimum, including an abstract in English) were considered for inclusion. Articles about orthopaedic education but not about programs in North America were later reviewed and incorporated into the final literature analysis but were identified as to geographic origin.
The search of journals had the following steps. First, medical bibliographic databases were searched for relevant articles. MEDLINE and Embase database searches were done for all articles cited up to April 2013 whether the article was electronically available or not. The search terms were (“Orthopaedics”[Mesh]) AND (“Education”[Mesh] OR “Education, Medical, Undergraduate”[Mesh] OR “Education, Medical, Graduate”[Mesh] OR “Education, Medical, Continuing”[Mesh] OR “Competency-Based Education”[Mesh] OR “Health Education, Dental”[Mesh] OR “Education, Public Health Professional”[Mesh] OR “Education, Distance”[Mesh] OR “Education, Professional”[Mesh] OR “Education, Premedical”[Mesh] OR “Education, Continuing”[Mesh] OR “Education, Professional, Retraining”[Mesh]). Full copies of all selected articles were retrieved and electronically stored with use of Refworks bibliographic software (http://www.refworks.com/). Study group members individually performed an initial pilot screening with use of the title and abstracts, when available.
Second, a coding form, building on the standards for performing literature reviews developed by Best Evidence in Medical and Health Professions Education (BEME collaboration) (http://www.bemecollaboration.org/), was developed and pilot tested. The coding sheet was mounted on Qualtrics survey software to generate summary statistics for further analysis. Each article was judged with use of the coding questions outlined in Table I.
Third, the final set of selected articles was analyzed by individual study group members, and findings were recorded on the survey coding form. Analyzed articles were grouped into four categories on the basis of the level of training (medical students, residents and fellows, orthopaedic surgeons, others or mixed subjects). Teams of three or four study group members reanalyzed all relevant articles after the studies had been grouped according to training level. The present report summarizes the findings from articles about training of residents and fellows. Once the final articles were selected for inclusion, all of the citations from these articles were then fully reviewed, and relevant articles were included in the final analysis.
The search of the PubMed and Embase databases produced 2027 articles that were tagged with the keywords “orthopaedic education” and related phrases. Once the Study Group had screened the titles and abstracts of the articles and had removed duplicates and irrelevant articles, there were 306 relevant articles overall and ninety-eight articles about orthopaedic residency and fellowship education1,4,6-14,16-102.
Supportive evidence in most (sixty-eight) of the studies was classified as participant perceptions or shifts in knowledge or attitudes. Nearly half (forty-eight) of the studies involved programmatic and/or curricular activities. Twenty-three studies concerned simulation and seven articles addressed or utilized Orthopaedic In-Training Examination (OITE) data. The vast majority of studies used questionnaires and/or surveys (thirty-three) or reported opinions and/or observations (twenty-nine) when assessing residents. Eighty-eight studies were from United States-based investigators, and ten were international.
Very few of the experimental studies included controlled randomization or comparison groups. Most were single-group prospective studies. The quality of evidence among the articles about orthopaedic education was classified as Level 1 (participant perception) for forty-two studies, Level 2 (knowledge gained or attitudes influenced) for twenty-six studies (Level 2a for two studies and Level 2b for twenty-four), Level 3 (change behavior or adopt in practice) for twelve studies (Level 3a for nine and Level 3b for three), Level 4 (change practice function) for four studies, and Level 5 (influence patient health status) for zero studies.
The present report represents a comprehensive review of the literature pertaining to orthopaedic graduate medical education from the early 1900s through 2013. In general, there were surprisingly few articles addressing orthopaedic education over this extensive time frame. There were only nineteen prospective studies and eleven case-control studies, with the majority of studies having no comparison group. With the recent focus on competency-based education, performance assessment has come to the forefront. Simulation and other technical teaching methods are being used on a trial basis with the aim of creating a proficiency-based approach that is structured, objective, efficient, and safe. There are some promising investigations and interventions, all of which will require further research and validation.
Several articles addressed individual ACGME competencies62,80,100,103. A survey of program directors and residents demonstrated that there was a universal commitment by programs to teach the core competencies, but perhaps a more understandable definition of each competency may further enable programs to effectively educate their residents100. Whereas 50% of program directors devote more time and resources to duty hours, only 17% commit more time to the core competencies100.
Although communication skills are rarely taught formally in residency programs, program directors and residents value communication as an important skill62. A combination of questionnaires, focus groups, and semi-structured interviews revealed that communication skills are taught implicitly rather than explicitly62. Some institutions teach systems-based practice with a combination of methods, such as morbidity and mortality conferences, simulation, didactic lecture series, monthly workshops, web-based modules, group projects, and experiential systems-based practice learning assignments80,103. However, Roberts et al. found that systems-based practice is inconsistently taught in most orthopaedic residency programs and is still poorly understood by most orthopaedic educators80.
Both residency program directors and residents believe that patient care is the most important core competency100. However, only one article focused on patient safety education, with the conclusion that issues surrounding patient safety are not addressed formally50.
The ACGME duty-hour guidelines (implemented in 2003 and modified in 2011) were created to mitigate resident fatigue, to improve education and academic performance, to enhance supervision, and to increase patient safety. Several articles addressed the effects of the ACGME duty-hour standards on resident education38,76,102,104,105.
A survey that was conducted by the American Orthopaedic Association (AOA) indicated that 74% of the program directors found the 2003 work-hour guidelines to be negative, mostly because of lost educational opportunity and a compromise in patient care77. Conversely, 38% of residents believed that the work-hour restrictions were negative and 34% found them to be positive. The AAOS surveyed residents one year after the 2003 ACGME guidelines were implemented and found that 37% of the respondents believed that their learning experience had decreased37. Another survey, conducted in 2011, demonstrated conflicting opinions between orthopaedic residents and program directors regarding the 2003 ACGME duty-hour regulations on education and patient care, although there was consensus that the 2011 rules may be detrimental to both resident education and patient care106.
Residents are concerned with decreased continuity of care brought about by the duty-hours requirements102. Residency programs have utilized various methods to ensure that the regulations are followed appropriately while trying to maintain patient care and educational endeavors. Many programs have moved to a night float system and have hired physician extenders. One large Canadian study demonstrated no difference in terms of educational benefit between a night float service and a standard one-night-in-four call service, although the health-related quality of life was substantially worse in association with the night float service101.
It is still unclear how the duty-hour changes have affected either the resident educational experience or patient safety. Whereas some investigators have demonstrated a decrease (as much as 20%) in operative experience102,104, others have demonstrated no differences in total operative experience after the implementation of the eighty-hour work week38,76. One institution demonstrated an increase in total resident publications and publications per resident-year since the work-hour changes were implemented105. It does not appear that there have been changes in OITE scores since the duty-hours implementation38.
Residents must receive an education in basic research skills and demonstrate aptitude in formal literature analysis and the fundamentals of evidence-based medicine. The ACGME requires research to be included in the curriculum, and residency programs must provide resources and space such that residents can participate in scholarly activities. Having good preceptor support was thought to be the single most important factor impacting the research experience for residents27. One program developed a multidisciplinary group of researchers and support personnel and demonstrated substantial increases in scholarly activity54. There is a correlation between protected block research time and greater research success27. Wolf et al. studied basic knowledge of medical research and found that the Level of Evidence criteria that are utilized in clinical articles do not appear to be regularly taught to or understood by residents99.
Orthopaedic In-Training Examination (OITE)
The OITE is a standardized examination that is designed to assess and compare the knowledge of orthopaedic residents nationally as well as to serve as a tool for educational improvement64. A number of articles focused on preparation for the OITE30,57,69,107. Monthly review of The Journal of Bone and Joint Surgery (American Volume), frequent review of the Journal of the American Academy of Orthopaedic Surgeons, an earlier start of preparation for the examination, review of prior OITEs, and a higher total number of hours devoted to studying had a strong correlation with OITE success69. Krueger et al. recently analyzed all questions from the 2006 to 2010 OITEs to determine which of three commonly used review sources would best aid in preparation56. The authors found that significantly (p < 0.0001) more answers were found on the Orthobullets web site (99.4%) as compared with Miller’s Review of Orthopaedics and the American Academy of Orthopaedic Surgeons AAOS Comprehensive Orthopaedic Review56. This finding suggests that perhaps our traditional methods of learning may not provide residents with all of the information necessary to pass required standardized tests. A few articles30,34,35 analyzed United States Medical Licensing Examination (USMLE) and OITE scores and correlated them with the ABOS pass rate. In general, there does appear to be some correlation, particularly between lower OITE scores and the ABOS fail rate. Percentile rank scores on the OITE during the third postgraduate year (PGY-3) do help to predict eventual ABOS-I and II pass rates30.
Assessment of Physical Examination Skills and Operative Ability
Oral and written examinations are often used to assess understanding, but performance on knowledge testing does not correlate with physical examination or technical skills21. Very few articles addressed how to best teach and assess physical examination and technical skills. In a national survey of orthopaedic residency program directors and orthopaedic surgery residents, 58% of the eighty-six program directors who responded and 83% of the 687 residents who responded believed that improvement in surgical skill among orthopaedic residents was not being objectively measured52. A survey of department chairs and residents underscored that there is insufficient time in the clinical setting to observe and critique a resident’s patient examination, even though physical examination skills are deemed important21.
The importance of operative skill development during resident education is clearly evident when reviewing the ACGME core competencies2. In the general surgery literature, the Global Rating Index for Technical Skills (GRITS) is a nine-item global rating scale assessing resident technical ability that has demonstrated good construct validity and reliability4. The American Board of Internal Medicine utilizes the Mini-Clinical Evaluation Exercise (Mini-CEX) to assess resident clinical assessment skills; the Mini-CEX has been found to have good construct validity and reliability108. In a survey of the Association of Residency Coordinators in Orthopaedic Surgery (ARCOS), Beran et al.21 found that only 2.6% of residencies offered Objective Structured Clinical Examination (OSCE)-type physical examination training and assessment.
A so-called Learning Assessment Toolkit, which is a relatively facile method for highlighting objective as well as self-assessed gaps in performance of key competencies before and after an educational program, has been developed for orthopaedic educators32. One program developed a musculoskeletal OSCE and found that residents, despite achieving high OITE scores, perform poorly on physical examination testing21. A novel Canadian article piloted the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE), which is a succinct tool for measuring competence in any surgical procedure14. This tool demonstrated validity and accurately differentiated among senior, mid-level, and junior residents. This instrument may well prove to be an extremely valuable assessment tool in the future. The first attempt by the ACGME to develop a more objective assessment tool for orthopaedics is the Milestone Project3, which is too early in its implementation to determine efficacy at this time.
Sonnadara et al.16 demonstrated that a thirty-day intensive course in surgical skills can be an effective means for first-year orthopaedic residents to develop targeted basic surgical skills. The authors compared a group of residents who participated in an intensive surgical skills laboratory course for thirty days with a group who were exposed to standard residency training. The scores for the laboratory-trained group were significantly better than those for the standard residency group on both the Objective Structured Assessment of Technical Skills (OSATS) standard checklist and a global rating scale (GRS) (p < 0.01). In a follow-up study, the same investigators demonstrated excellent retention rates15. These studies support the initiation of the ACGME/ABOS surgical skills modules for the first postgraduate year (PGY-1).
Cadaver and Computer Software Tools
Bergeson et al., in a study of free-hand insertion of thoracic pedicle screws, found that there was a learning curve for all residents, with 34% of cortical violations occurring on the first cadaver and only 15% occurring on the last cadaver10. Shanedling et al. found that two online instruments designed to assess residents’ perception of preparedness and cognitive skills were both valid and reliable predictors of their readiness to successfully pass a cadaveric motor skills test of carpal tunnel release86.
Because cadaveric models have substantial financial costs, lengthy set-up requirements, and ethical implications, computer software programs, including virtual reality technology, offer a tremendous advantage. The AAOS Council on Education established a Task Force on Virtual Reality in 1998 to investigate and recommend current technologies109. Blyth et al.9 reported that a personal computer-based virtual-reality training system that was designed to evaluate and improve the reduction and internal fixation of hip fractures demonstrated good face validity. Thomas and Allen employed a computer-assisted learning module in the form of a CD-ROM with an interactive tutorial on anatomy, video demonstrations, and basic treatment of nonoperative problems of the foot and ankle90. Residents scored higher on post-test examinations after spending only one hour with this computer program.
The aviation industry has utilized simulators for decades, dating back to the late 1920s with use of the Link trainer110. General surgeons are much further ahead of orthopaedic surgeons with respect to the use of simulation in training and have created unbiased, feasible, reliable, and validated laboratory-based and virtual reality simulators that have demonstrated face and construct validity4-8,42.
The AAOS Council on Education convened a simulation summit in 2011 with representatives from the AAOS, ABOS, RRC-OS, and the Council of Orthopaedic Residency Directors of the AOA (AOA/CORD). The discussion focused on how other surgical specialties have incorporated simulation into their educational programs. The RRCs of general surgery and obstetrics-gynecology have instituted program requirements for residency training, including structured motor skills training and dedicated simulation facilities. In 2004, the American College of Surgeons (ACS) and the Society of American Gastrointestinal and Endoscopic Surgeons created the Fundamentals of Laparoscopic Surgery (FLS) program, which was the first validated simulation module to be standardized and made universally available. By 2008, there was industry support to provide simulation and didactic materials to every general surgery residency program in the United States. The FLS examination measures cognitive knowledge, case and problem-management skills, and manual dexterity. The ACS has now mandated that surgical residents must pass an FLS simulator test in order to become board-certified surgeons111.
Simulation provides the opportunity to learn in a controlled environment without fear of harming patients, increases the exposure to surgical procedures, provides uniformity, improves the rate of skill acquisition, and provides objective assessment of surgical skill111,112. Karam et al. reported that 80% of orthopaedic program directors and 86% of residents believe that surgical skills simulation should become a required part of training52. Korndorffer et al.113 demonstrated that only 55% of programs had surgical skills laboratories, with 87% of program directors identifying a lack of funding as the most substantial barrier to development of a formal surgical skills program.
We found a tremendous increase in the number of orthopaedic articles published over the past decade with a focus on surgical simulation. Atesok et al.18 recently performed a thorough review of surgical simulation in orthopaedic training. The authors concluded that although simulation appears promising, current evidence to support its use in orthopaedic education is limited and validation studies must be performed. Many studies have utilized arthroscopic simulation, often demonstrating facilitation of psychomotor skills and construct validity11-13,65. Gomoll et al. found that simulator-based arthroscopy training demonstrated significant (p < 0.05) differences between participants on the basis of experience in all parameters41 and that subjects demonstrated significant (p < 0.05) improvement on simulator retesting three years after the initial evaluation1.
The Knee Arthroscopy Surgical Trainer (KAST), developed by the AAOS Task Force on Virtual Reality and supported by the Arthroscopy Association of North America and the ABOS, includes high-fidelity haptic rendering, creates three-dimensional models of structures within the knee, and includes the Mentor, which is the didactic aspect of the KAST63. The Sheffield Knee Arthroscopy Training System (SKATS) is a cost-effective virtual reality arthroscopic training system that has demonstrated both construct and face validity as well as predictive validity66.
A high-fidelity porcine dynamic compression plate model that was developed in the United Kingdom represents a validated, low-cost model that utilizes movement analysis and video rating scores for skills assessment58. Interestingly, Howells et al. reported that skills learned on a simulator were lost following a six-month hiatus during which the surgeons did not perform the procedure47.
The OSATS, which includes global ratings and checklist evaluations, may demonstrate value in orthopaedic education17. This assessment requires trainees to perform standardized surgical tasks on models under the direct observation of an instructor, with the performance being assessed with two scoring methods: (1) a checklist consisting of specific surgical tasks that are essential aspects of a particular procedure and (2) a global rating form114. The validity and reliability of the OSATS are similar to those of the OSCE6,115,116. Van Heest et al.17 assessed the knowledge of residents about carpal tunnel release on the basis of a knowledge test, the OSATS, a pass-fail assessment, and time to completion. They found that the knowledge test, the global rating scale, the detailed checklist, and pass-fail assessment positively correlated with the level of orthopaedic training.
Simulation may decrease operative costs as a result of the acquisition of surgical proficiency in a laboratory environment as opposed to an operating theater. Farnworth et al. found that the differences in operative and anesthetic times were significantly greater for residents compared with attending surgeons (p < 0.0001). The resident-led procedures cost an average of roughly $900 more than those performed by staff members alone117.
Future Directions and Recommendations
The Orthopaedic Surgery Milestone Project3 will provide an enhanced framework for assessing resident competence in the orthopaedic subspecialties in terms of both medical knowledge and surgical skill. This project requires a marked increase in documentation of teaching processes and evaluating proficiency in the ACGME core competencies. More research is needed to determine best practices of instruction and assessment of technical skills within the framework.
A Surgical Skills Task Force (SSTF) was convened by the ABOS with support from the AAOS and AOA/CORD to develop a curriculum of simulation exercises with goals, objectives, and assessment tools for first-year residents. The Surgical Skills Modules for PGY-1 residents was introduced in 2012 to teach first-year residents a wide range of skills, such as sterile technique, suturing, soft-tissue handling, casting, fluoroscopy, arthroscopy, arthroplasty, and patient safety. These modules, the O-SCORE, OSATS, and musculoskeletal OSCE, all appear to have substantial potential for training residents and should be further evaluated.
While funding for graduate medical education has remained flat or has decreased, costs have increased dramatically and will continue to do so as additional resources are required to teach and also to comply with increasingly complex and demanding documentation standards. This is particularly true with regard to simulation, which clearly will secure a prominent role in the educational process, but at great expense. Centralized and collaborative efforts across the orthopaedic education community should be encouraged in an effort to provide economies of scale and to minimize duplication, while still fostering innovation at the program level. Participants of the AOA 2005 Resident Leadership Forum believed that a national curriculum with more centralization and formalization would improve orthopaedic education78. Incorporation of the Surgical Skills Modules into the intern year may provide an ideal opportunity for regional or national collaboration. A centralized effort potentially will result in an improved and more uniform educational product, allowing for better allocation of resources, an improved platform for fund-raising, and a tremendous opportunity to perform education research. Our general surgery colleagues have created a viable method to integrate simulation into their programs, which should be studied and implemented into orthopaedics.
Our review of the current literature in orthopaedic education revealed a decades-long conversation about how to provide the best training environment. Although challenges abound, the orthopaedic community must collaborate to ensure that we deliver the best educational product possible. Advances in educational philosophy and technology should be incorporated into the education process in the most efficient and effective method possible. Resources should be optimally focused in an effort to develop, measure, and report best practices in orthopaedic education.
The AAOS Orthopaedic Study Group includes the authors as well as Edward Akelman, MD, James Heckman, MD, Dawn LaPorte, MD, Valerae Lewis, MD, Alfonso Mejia, MD, Robert Murphy, MD, Denise Santomauro, and Veronica Wadey, MD.
Source of Funding: No external funds were received in support of this study.
Investigation performed at The Ohio State University, Columbus, Ohio; the University of Illinois at Chicago, Chicago, Illinois; Chapel Hill Orthopaedic Surgery and Sports Medicine, Chapel Hill, North Carolina; the University of Kentucky Shriners Hospital for Children, Lexington, Kentucky; The University of Connecticut Health Center, Farmington, Connecticut; and The University of Texas Health Science Center at San Antonio, San Antonio, Texas
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. One or more of the authors, or his or her institution, has had a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. No author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
- Copyright © 2014 by The Journal of Bone and Joint Surgery, Incorporated