Clin Med Rev Case Rep ISSN: 2378-3656

• Introduction
• Self-Regulated Learning
• Materials and Methods
• Results
• Discussion
• Conclusions
• Figure 1
• Table 1
• Table 2
• Table 3
• Table 4
• References

Download PDF
    

Clinical Medical Reviews and Case Reports

Department of Surgery, Yale New Haven Hospital, USA

Introduction

The concept of lifelong learning has received widespread attention in the healthcare field and in numerous other industries due to the sheer volume of new and expanding knowledge and information continuously created by one hand and rendered obsolete by the other. As a result, physicians need to be engaged in lifelong learning to improve, expand, and maintain mastery of their knowledge, skills, and competencies.

For physicians, lifelong learning is a process that begins in medical school, extends into residency, and continues throughout one's professional life [1]. Its importance is seen in a statement from the American Medical Association's Principles of Medical Ethics, which states "a physician must continue to study, apply, and advance scientific knowledge [2]. This is underscored by the Accreditation Council for Graduate Medical Education (ACGME) in its Common Program Requirements document, which states: "Residents and faculty members must demonstrate an understanding and acceptance of their personal role in...attention to lifelong learning" [3].

In her article on portfolios in radiology residency education, Deitte points to changes in training over the past ten years, which include more focus on self-directed learning, evidence-based practice, and outcomes-based assessment. She states that these changes are the result of demand for physician accountability and documentation of a commitment to lifelong learning [4]. As a result, the medical education system is expected to prepare physicians to participate in lifelong learning activities. However, engaging in lifelong learning is based on successfully developing and applying two skill areas: self-directed learning and metacognition [5].

Self-Regulated Learning

Self-regulated learning and self-directed learning share two critical components related to the learning experience - internal monitoring and external management. Both concepts address issues of responsibility and control where the learner is master of his/her own learning [6]. Self-regulated learning refers to learning that results from learners' self-generated thoughts and behaviors oriented systematically toward achieving their goals. It represents a proactive approach where learners exert control over their learning processes and environments. It is learning that is guided by metacognition (awareness and knowledge about one's own thinking), strategic action (planning, implementing, monitoring, and evaluating strategies), and an emphasis on motivation to learn (commitment to their goals, beliefs about outcomes of their actions, and personal beliefs (self-efficacy) about their learning or performance). Learners who are self-regulated learners are metacognitively active participants in their own learning. Both metacognition and self-regulated learning are two influential types of cognitive control processes that converge and diverge along select dimensions.

Specifically, self-regulated learners are cognizant of their academic strengths and weaknesses and are aware of when they know a concept or have mastered a skill and when they do not. These individuals monitor and reflect upon the effectiveness of their learning strategies, which further motivates them to improve their learning methodologies. As a result, they are more likely to succeed academically and are better prepared to engage in lifelong learning [7].

Metacognition

Metacognition (thinking about one's thinking) is the ability to reflect on, understand, and control one's learning. It is central to self-regulated learning and helps learners to be successful. Metacognition involves two major functions: knowledge and regulation of the cognitive processes involved in learning. Metacognitive knowledge is knowledge learners have about their cognitive abilities, strategies, and tasks. Metacognitive regulation refers to processes that coordinate cognition. These include monitoring (detecting errors) and control (error correction, planning, and resource allocation) strategies [8].

The purpose of this study is to assess the relationship between self-regulated learning (measured by the Self-Regulated Learning Perception Scale (SRLPS), metacognition (measured by the Metacognitive Awareness Inventory (MAI)), and academic achievement, measured by performance on the American Board of Surgery In-Training Examination. The American Board of Surgery website describes the ABSITE as "an evaluation measurement to assess residents' progress" [9]. However, ABSITE results are now incorporated into milestones, arguably rendering the exam as "high-stakes". Likewise, ABSITE performance is more commonly being used as one measure to select candidates for fellowships.

If the linkages between self-regulated learning, metacognition, and ABSITE performance are confirmed, program leaders can have confidence using instruments like the SRLPS and MAI early on during residents' training experience to quickly and reliably identify who will likely struggle with academic performance. Program leaders can then design and initiate intervention plans to help residents develop better self-regulation and metacognitive skills and strategies to help ensure academic success. As such, this study represents another step in this important area of research.

Materials and Methods

Self-regulated learning

The Self-Regulated Learning Perception Scale (SRLPS) was used to measure resident self-regulation (Table 1). It consists of 41 items that are answered on a five-point Likert scale from Strongly Disagree (1) to Strongly Agree (5). Total scores can range from 41 to 205. Four dimensions of self-regulation are assessed:
1. Motivation and action to learning (seven items)
2. Planning and goal setting (eight items)
3. Strategies for learning and assessment (nineteen items)
4. Lack of self-directedness (seven items)

Scale items were developed based upon literature, expert comments, and narratives from medical students about their learning strategies. Seven experts evaluated the items to provide content validity and factor analysis was performed for construct validity. Factor analysis yielded 41 test items from an original bank of 66 items resulting in the above four factors [5].

Metacognition

The Metacognitive Awareness Inventory (MAI) was used to measure resident metacognition (Table 2). The MAI consists of 52 items answered on a five-point Likert scale ranging from Always False (1) to Always True (5). Total scores can range from 52 to 260. Two major components associated with metacognition are assessed: knowledge about cognition and regulation of cognition.

Items related to knowledge about cognition include three sub-processes that facilitate reflection of metacognition:
1. Declarative knowledge: knowledge about learning and one's cognitive skills and abilities
2. Procedural knowledge: knowledge about how to use strategies
3. Conditional knowledge: knowledge about when and why to use strategies

Items related to regulation of cognition include five sub-processes that facilitate the control aspect of learning:
Planning: planning, goal setting, and allocating resources
Information management strategies
a. Organizing: implementing strategies and heuristics that help manage information
b. Information Management: organizing, elaborating, summarizing, and selectively focusing on important information
Monitoring: assessing one's learning or strategy use
Debugging: strategies to correct performance errors or assumptions about the task or strategy use
Evaluation: post-hoc analysis of performance and strategy effectiveness [9].

These instruments were selected since their items were deemed more appropriate for trainees participating in the learning structure and environment of a residency program than similar instruments used at the secondary school or college level.

Resident performance

The American Board of Surgery In-Training Examination (ABSITE) was used to measure resident performance. This high stakes examination is a five hour test consisting of 250 multiple choice questions developed by ABS directors and expert consultants from across the country. All questions are vetted through a multi-tiered review process by several ABS committees. Questions that survive this process are then incorporated into the examinations.

In a further assessment of their validity, all examination questions undergo psychometric analysis. This analysis includes a review of the percentage of examinees who answered a question correctly and the percentage of correct answers attained by both high and low scorers on the examination. Any question whose psychometric performance does not meet well-accepted standards is deleted from final scoring. In addition, all questions are periodically reviewed for their continued relevance.

The content is based on topics covered in a five-year general surgery residency training program as identified in the publication "The SCORE® Curriculum Outline for General Surgery". The primary focus of the ABSITE is on Clinical Management. Approximately 80% of the exam's questions address Clinical Management topics and 20% Applied Science topics. Residents take this exam during each year of their residency. Exams are scored compared to other residents at the same PGY (Post Graduate Year) level.

IRB approval, setting, and participants

This study was reviewed by the Human Subjects Committee of Yale University and qualified for exemption under 45 CFR 46.101(b) [2]. The study took place at Yale New Haven Hospital, Saint Raphael Campus (SRC), with general surgery residents on this campus. Testing dates were announced approximately one week prior and residents reported to a conference room to complete the instruments.

Statistical procedures

Descriptive statistics on both demographic variables and outcome variables of interest were first obtained. Correlation analysis between Question Scores and Factor Scores was then performed to identify the relationship between these variables. Non-parametric techniques were used to compare MAI and SRLPS individual factor and total scores by a number of baseline characteristics (gender, medical school (U.S /Foreign) and PGY level (PGY 1/ PGY 2-3/ PGY 4-5)). Univariate and multivariable regression analyses were carried out to determine whether MAI and SRLPS individual factor and total scores could predict the main outcome variable (ABSITE score). All analyses were performed using SAS© version 9.2.

Results

Data collected from 22 residents (18 categorical, four preliminary) were analyzed. Categorical residents remain in a program for his/her entire residency training. Preliminary residents are in a given residency program for one year. This usually precedes training in another specialty. The median ABSITE score was 64% (IQR 35-77) and the majority (n=13; 62%) of study participants scored above the 50th percentile. Table 3 summarizes the component and total scores for MAI and SRLPS and table 4 displays similar component and total scores by PGY level.

As anticipated, all four SRLPS Factor scores (SRLPS Factor 1/2/3/4) were highly correlated with the SRLPS Total Score (p=0.003; p<0.0001; p<0.0001; p<0.0001 respectively). Likewise, all 10 MAI Factor scores (MAI F1-10) were highly correlated with MAI Total Score (MAI Factor 6 score p<0.0004; all others p<0.0001). In addition, SRLPS Total Score was significantly correlated with MAI Total Score (p=0.0001).

SRLPS Factor 1 Score was not correlated with the ABSITE Score (p=0.91); the rest of the factor scores were all significantly correlated (SRLPS Factor 2; p=0.05; SRLPS Factor 3; p=0.04; SRLPS Factor 4; p=0.02). In addition, the SRLPS Total score was significantly correlated with ABSITE Score (p=0.03)

MAI Factors 2, 3 and 8 did not appear to be significantly correlated with ABSITE Score (p>0.05); the rest of the MAI Factors were significantly correlated with ABSITE Score as follows: MAI Factor 1 p=0.045; MAI Factor 4 p=0.047; MAI Factor 5 p=0.04; MAI Factor 6 p=0.0198; MAI Factor 7 p=0.016; MAI Factor 9 p=0.01; MAI Factor 10 p=0.007. The MAI Total Score was significantly correlated with ABSITE Score (p=0.011).

Analyses to assess differences in the factor scores by some demographic variables revealed some interesting findings:
• There were no differences in SRLPS and MAI individual Factor scores and Total scores by gender.
• There were no differences in SRLPS Factor and Total scores when comparing PGY Level 1 vs. PGY Level 2-3 vs. PGY Level 4-5. With the exception of MAI Factor 6 score (p=0.03), there were no differences in the remaining MAI Factors and Total Scores between PGY Level 1, Level 2-3 and Level 4-5 residents.
• When comparing results of residents who attended U.S. vs. foreign medical schools, there were no differences in SRLPS Individual Factor scores or Total Scores. However, there were differences in MAI Factor Scores 1, 3, 4, 6, 7, 9, 10 and MAI Total Score (all p<0.05) between residents from U.S. vs.foreign medical schools (for all scores, residents from foreign medical schools scored higher than their counterparts from U.S. medical schools).

Regression analyses suggest that both SRLPS Total and MAI Total Scores can be used to predict ABSITE Score (Figure 1a,1b).

Discussion

Resident performance on ABSITE examinations (and other specialty in-training examinations) is one of the most vexing issues faced by Program Directors and educators. In spite of resident and faculty efforts to improve performance, continued poor scores on in-training examinations persist on being a significant problem.

In the past several years this problem has become greatly magnified since importance of the in-training examination has changed. Where scores once represented "low stakes" markers to help residents and Program Directors identify and improve knowledge and skills and to enhance residents' chances of passing Board examinations, they have now become "high stakes" tools used to screen residents for fellowship positions as well as to assess their attainment of milestones. In this era of specialization where many residents are vying for fellowships, good scores on in-training examinations have become an imperative for securing prime positions. Yet, despite the best efforts to improve resident success, for some there is little change. By demonstrating connections between self-regulated learning, metacognition, and in-training examination scores, the outcomes of this study can provide direction to help improve resident performance.

One interesting finding was that PGY 4-5 level residents performed significantly better on the MAI Factor 6 score (monitoring) than residents at the other PGY levels. This may be a result of senior residents' experience and becoming more adept than their less experienced colleagues in knowing how to identify and master important data and effectively separate out relevant from irrelevant data (fewer encoding errors when monitoring their learning).This is not surprising since the sheer volume of information to learn in residency programs is formidable and may take several years for residents to discriminate between important and unimportant variables. This difference is often observed in presentations by junior-level residents. Unlike more senior residents, they tend to present all aspects of patient cases, not just critical data points. However, it is worth noting that PGY 1 residents displayed higher MAI Factor 6 scores (monitoring) than PGY 2-3 level residents. One quarter of the PGY 1 residents in the study were graduates from foreign medical schools. These graduates face greater challenges applying and getting accepted into U.S. medical schools and may have been taught monitoring strategies to become efficient and effective learners and more favorably compete.MAI Factor 6 scores for PGY 1 residents in this study were among the highest of all residents and may have contributed to them outperforming PGY 2-3 level residents.

Another finding of interest is foreign medical school graduates scored higher on almost all MAI factors and MAI Total scores compared to graduates from U.S. medical schools. As previously mentioned, foreign medical school graduates face greater challenges as their U.S. counterparts in applying to U.S. residency programs. Two studies referenced by Desbiens and Vidaillet [10], showed differences in responses to letters of inquiry and requests for applications from U.S. medical school graduates (USMGs) and graduates from foreign medical schools (International Medical Graduates -(IMGs)) that were statistically significantly biased in favor of USMGs over IMGs by a margin of 50 to 100 percent.10In addition, according to a 2010 National Resident Matching Program® (NRMP) Program Director Survey, one of the top four factors in resident selection is the United States Medical Licensing Examination® (USMLE) Step 1 scores. [11] Recognizing bias against their graduates and the high importance U.S. residency programs place on standardized examinations, foreign medical schools may incorporate metacognitive teaching strategies into the curriculum to teach students how to be more effective and efficient learners and to better prepare them for achieving a high level of success on standardized examinations. Although we could find no reports or studies in this area, our years of experience reviewing residency applications indicates that foreign medical school applicants have among the highest USMLE examinations scores. In addition, they seem to enjoy a high first-time pass rate on board examinations. Residents and Program Directors can use this information to improve resident performance that is not only evidence-based as shown by the results of this study, but also in similar studies described below that preceded this one.

In his research on assessing metacognitive awareness, Scraw cited several studies that indicated learners who are more metacognitively aware are more strategic and perform better than those who are unaware [12]. He offered the explanation that metacogntively aware learners plan, sequence, and monitors their learning in a way that directly improves their performance [12]. This is reinforced in Quirk's work on the development of expertise in medical education [13]. He argued for teaching physicians metacognitive skills to help them optimize their learning experiences during training and throughout their careers. Metacognition is currently viewed as one part of the more general construct of self-regulated learning. Although self-regulated learning and its relationship to academic success has been studied extensively over the past fifteen years, the impact of metacognition has not been comprehensively examined [14].

Numerous studies have been conducted on various aspects of self-regulation and metacognition with students at the elementary school, high school, and college levels showing students who use these processes demonstrate better learning and achievement outcomes. In their research on metacognitive development with professional educators, Cooper and Stewart also cited a number of studies which consistently demonstrated a relationship between students' metacognitive knowledge, self-regulation abilities, and academic performance. Although the authors indicated these studies have shown the positive influence of metacognition on performance and cited one study with nurses and electronics technicians as an example of research with adults in the workforce, there have been very few studies examining these linkages with adults after college or in the workforce [15].

In the medical field, evidence reported from studies with medical school students examining linkages between self-regulated learning, metacognition, and learning and performance further confirms the relationship between these constructs and achievement [5,16-20]. In graduate medical education one study conducted in Brazil with anesthesiology residents examined linkages to academic performance on several variables including their learning and study strategies. Their results revealed that three factors associated with strategic learning (anxiety, motivation, and ability to properly select main ideas) were found to independently predict academic performance [21]. Other than the above, there is a lack of such research exploring these relationships with residents and none in general surgery. It is critically important to investigate self-regulated learning in graduate medical education programs since, unlike medical school where instruction is well defined and arranged, residents are virtually on their own to study and must master a significant body of knowledge from diverse sources. It is estimated that as much as 95% of what is learned in surgical residency training occurs in the clinical setting or at home. Because of the clinical and operative demands placed on surgical residents, it is difficult to find large blocks of study time; more often these come in interrupted slivers of time. As a result, residents must become more proactive and skillful in their approach to master the body of knowledge associated with this discipline [22].

Conclusions

The small sample size used in this study limited the ability to carry out formal hypothesis testing. However, the results provide preliminary data that could potentially be used to power a study with a large enough sample size to allow confirmation of linkages between self-regulated learning and high stakes examinations. Additionally, this study was conducted at a single site, thus limiting generalizability. This study can be undertaken in more or larger academic centers and/or in other disciplines where hypothesis can be formulated and tested to definitively affirm the relationships between self-regulated learning and high stakes examinations. In this preliminary study, we were able to show the relationship between self-regulated learning, metacognition and academic achievement. In fact, a resident's score on the SLRP and MAI could potentially predict subsequent ABSITE scores. Identifying those residents at increased risk for academic performance challenges at the start of residency training may be of significant benefit to Surgical Educators. Our results suggest that poor scores on the ABSITE may not purely reflect lack of effort, but rather, expose major issues of how residents approach the act of learning. With the SLRPS and MAI tools, surgical educators can identify those residents with poor insight into their cognitive skills and abilities and deficiencies in being able to plan, organize and implement information management strategies. Creating an organized, feasible study plan from day one may prevent academic shortcomings and failure during and, more importantly, after residency training.

References

1. Hojat M, Veloski J, Nasca TJ, Erdmann JB, Gonnella JS (2006) Assessing physicians' orientation toward lifelong learning. J Gen Intern Med 21: 931-936.
   


2. American Medical Association (AMA) Principles of medical ethics.
   


3. Long TR, Poe JD, Zimmerman RS, Rose SH (2012) A Citation Tracking System to Facilitate Sponsoring Institution Oversight of ACGME-Accredited Programs. J Grad Med Educ 4: 500-504.
   


4. Deitte L (2008) Learning portfolios in radiology residency education: how do I get started? J Am Coll Radiol 5: 664-669.
   


5. Turan S, Demirel O, Sayek I (2009) Metacognitive awareness and self-regulated learning skills of medical students in different medical curricula. Med Teach 31: e477-483.
   


6. Pilling-Cormick J, Garrison D (2007) Self-directed and self-regulated learning: conceptual links. Canadian Journal of University Continuing Education 33:2:13-33.
   


7. Zimmerman B (2002) Becoming a self-regulated learner: an overview. Theory into Practice 41: 64-70.
   


8. Fernandez-Duque D, Baird JA, Posner MI (2000) Executive attention and metacognitive regulation. Conscious Cogn 9: 288-307.
   


9. ABS In-Training Examination (ABSITE).
   


10. Desbiens NA, Vidaillet HJ Jr (2010) Discrimination against international medical graduates in the United States residency program selection process. BMC Med Educ 10: 5.
    


11. Sohi, Sajeet (2014) International medical graduates -a possible solution to the expected physician shortage
    


12. Schraw G, Dennison R (1994) Assessing metacognitive awareness. Contemporary Educational Psychology 19:460-475.
    


13. Nothnagle M, Goldman R, Quirk M, Reis S (2010) Promoting self-directed learning skills in residency: a case study in program development. Acad Med 85: 1874-1879.
    


14. Isaacson R, Fujita F (2006) Metacognitive knowledge monitoring and self-regulated learning: academic success and reflections on learning. Journal of the Scholarship of Teaching and Learning.
    


15. Cooper S, Stewart P (2006) Metacognitive development in professional educators, Paper presented at the annual meeting of the Northern Rocky Mountain Educational Research Association; October 2005;Jackson Hole, Wyoming.Paper presented at the annual convention of the American Educational Research Association.
    


16. Turan S, Demirel O (2010) The relationship between self-regulated learning skills and achievement: a case study from Hacettepe university medical school, H.U. Journal of Education 38:279-291.
    


17. Turan S, Konan A (2012) Self-regulated learning strategies used in surgical clerkship and the relationship with clinical achievement. J Surg Educ 69: 218-225.
    


18. Agrawal S, Norman GR, Eva KW (2012) Influences on medical students' self-regulated learning after test completion. Med Educ 46: 326-335.
    


19. White CB (2007) Smoothing out transitions: how pedagogy influences medical students' achievement of self-regulated learning goals. Adv Health Sci Educ Theory Pract 12: 279-297.
    


20. Artino AR Jr, Hemmer PA, Durning SJ (2011) Using self-regulated learning theory to understand the beliefs, emotions, and behaviors of struggling medical students. Acad Med 86: S35-38.
    


21. de Oliveira Filho GR, Vieira JE (2007) The relationship of learning environment, quality of life, and study strategies measures to anesthesiology resident academic performance. Anesth Analg 104: 1467-1472, table of contents.
    


22. Schmitz C, D'Cunha J, Antonoff M (2010) Developing self-regulated learning habits can help residents be better learners.