Does Assessment of Self-Regulated Learning and Metacognition in Surgical Residents Provide Insight to Performance on High Stakes Standardized Examinations ? A Pilot Study

C l i n M e d International Library Citation: Contessa J, Kyriakides TC, Nadzam G (2015) Does Assessment of SelfRegulated Learning and Metacognition in Surgical Residents Provide Insight to Performance on High Stakes Standardized Examinations? A Pilot Study. Clin Med Rev Case Rep 2:047. doi.org/10.23937/2378-3656/1410047 Received: February 10, 2015: Accepted: August 11, 2015: Published: August 14, 2015


Introduction
The concept of lifelong learning has received widespread attention in the healthcare field and in numerous other industries due to the sheer volume of new and expanding knowledge and information continuously created by one hand and rendered obsolete by the other.As a result, physicians need to be engaged in lifelong learning to improve, expand, and maintain mastery of their knowledge, skills, and competencies.
For physicians, lifelong learning is a process that begins in medical school, extends into residency, and continues throughout one's professional life [1].Its importance is seen in a statement from the American Medical Association's Principles of Medical Ethics, which states "a physician must continue to study, apply, and advance scientific knowledge [2].This is underscored by the Accreditation Council for Graduate Medical Education (ACGME) in its Common Program Requirements document, which states: "Residents and faculty members must demonstrate an understanding and acceptance of their personal role in…attention to lifelong learning" [3].
In her article on portfolios in radiology residency education, Deitte points to changes in training over the past ten years, which include more focus on self-directed learning, evidence-based practice, and outcomes-based assessment.She states that these changes are the result of demand for physician accountability and documentation of a commitment to lifelong learning [4].As a result, the medical education system is expected to prepare physicians to participate in lifelong learning activities.However, engaging in lifelong learning is based on successfully developing and applying two skill areas: selfdirected learning and metacognition [5].

Self-Regulated Learning
Self-regulated learning and self-directed learning share two critical components related to the learning experience -internal monitoring and external management.Both concepts address issues of responsibility and control where the learner is master of his/ her own learning [6].Self-regulated learning refers to learning that results from learners' self-generated thoughts and behaviors oriented assess residents' progress" [9].However, ABSITE results are now incorporated into milestones, arguably rendering the exam as "highstakes".Likewise, ABSITE performance is more commonly being used as one measure to select candidates for fellowships.
If the linkages between self-regulated learning, metacognition, and ABSITE performance are confirmed, program leaders can have confidence using instruments like the SRLPS and MAI early on during residents' training experience to quickly and reliably identify who will likely struggle with academic performance.Program leaders can then design and initiate intervention plans to help residents develop better self-regulation and metacognitive skills and strategies to help ensure academic success.As such, this study represents another step in this important area of research.

Self-regulated learning
The Self-Regulated Learning Perception Scale (SRLPS) was used to measure resident self-regulation (Table 1).It consists of 41 items that are answered on a five-point Likert scale from Strongly Disagree (1) to Strongly Agree (5).Total scores can range from 41 to 205.Four dimensions of self-regulation are assessed:

1.
Motivation and action to learning (seven items)

2.
Planning and goal setting (eight items)

3.
Strategies for learning and assessment (nineteen items)

4.
Lack of self-directedness (seven items) Scale items were developed based upon literature, expert comments, and narratives from medical students about their learning strategies.Seven experts evaluated the items to provide content validity and factor analysis was performed for construct validity.Factor analysis yielded 41 test items from an original bank of 66 items resulting in the above four factors [5].

Metacognition
The Metacognitive Awareness Inventory (MAI) was used to measure resident metacognition (Table 2).The MAI consists of 52 items answered on a five-point Likert scale ranging from Always False (1) to Always True (5).Total scores can range from 52 to 260.Two major components associated with metacognition are assessed: knowledge about cognition and regulation of cognition.
Items related to knowledge about cognition include three subprocesses that facilitate reflection of metacognition: 1. Declarative knowledge: knowledge about learning and one's cognitive skills and abilities 2. Procedural knowledge: knowledge about how to use strategies

3.
Conditional knowledge: knowledge about when and why to use strategies Items related to regulation of cognition include five sub-processes that facilitate the control aspect of learning:

1.
Planning: planning, goal setting, and allocating resources

Information management strategies:
a. Organizing: implementing strategies and heuristics that help manage information b.Information Management: organizing, elaborating, summarizing, and selectively focusing on important information

3.
Monitoring: assessing one's learning or strategy use

4.
Debugging: strategies to correct performance errors or assumptions about the task or strategy use

5.
Evaluation: post-hoc analysis of performance and strategy effectiveness [9].These instruments were selected since their items were deemed more appropriate for trainees participating in the learning structure and environment of a residency program than similar instruments used at the secondary school or college level.

Resident performance
The American Board of Surgery In-Training Examination (ABSITE) was used to measure resident performance.This high stakes examination is a five hour test consisting of 250 multiple choice questions developed by ABS directors and expert consultants from across the country.All questions are vetted through a multi-tiered review process by several ABS committees.Questions that survive this process are then incorporated into the examinations.
In a further assessment of their validity, all examination questions undergo psychometric analysis.This analysis includes a review of the percentage of examinees who answered a question correctly and the percentage of correct answers attained by both high and low scorers on the examination.Any question whose psychometric performance does not meet well-accepted standards is deleted from final scoring.In addition, all questions are periodically reviewed for their continued relevance.
The content is based on topics covered in a five-year general surgery residency training program as identified in the publication "The SCORE® Curriculum Outline for General Surgery".The primary focus of the ABSITE is on Clinical Management.Approximately 80% of the exam's questions address Clinical Management topics and 20% Applied Science topics.Residents take this exam during each year of their residency.Exams are scored compared to other residents at the same PGY (Post Graduate Year) level.

1.
I ask myself periodically if I am meeting my goals 1 2 3 4 5

2.
I consider several alternatives to a problem before I answer 1 2 3 4 5

3.
I try to use strategies that have worked in the past 1 2 3 4 5

4.
I pace myself while learning in order to have enough time

IRB approval, setting, and participants
This study was reviewed by the Human Subjects Committee of Yale University and qualified for exemption under 45 CFR 46.101(b) [2].The study took place at Yale New Haven Hospital, Saint Raphael Campus (SRC), with general surgery residents on this campus.Testing dates were announced approximately one week prior and residents reported to a conference room to complete the instruments.

Statistical procedures
Descriptive statistics on both demographic variables and outcome variables of interest were first obtained.Correlation analysis between Question Scores and Factor Scores was then performed to identify the relationship between these variables.Non-parametric techniques were used to compare MAI and SRLPS individual factor and total scores by a number of baseline characteristics (gender, medical school (U.S /Foreign) and PGY level (PGY 1/ PGY 2-3/ PGY 4-5)).Univariate and multivariable regression analyses were carried out to determine whether MAI and SRLPS individual factor and total scores could predict the main outcome variable (ABSITE score).
All analyses were performed using SAS© version 9.2.

Results
Data collected from 22 residents (18 categorical, four preliminary) were analyzed.Categorical residents remain in a program for his/ her entire residency training.Preliminary residents are in a given residency program for one year.This usually precedes training in another specialty.The median ABSITE score was 64% (IQR 35-77) and the majority (n=13; 62%) of study participants scored above the 50th percentile.Table 3 summarizes the component and total scores for MAI and SRLPS and table 4 displays similar component and total scores by PGY level.
Analyses to assess differences in the factor scores by some demographic variables revealed some interesting findings:

•
There were no differences in SRLPS and MAI individual Factor scores and Total scores by gender.

•
There were no differences in SRLPS Factor and Total scores when comparing PGY Level 1 vs. PGY Level 2-3 vs. PGY Level 4-5.With the exception of MAI Factor 6 score (p=0.03),there were no differences in the remaining MAI Factors and Total Scores between PGY Level 1, Level 2-3 and Level 4-5 residents.

•
When comparing results of residents who attended U.S. vs. foreign medical schools, there were no differences in SRLPS Individual Factor scores or Total Scores.However, there were differences in MAI Factor Scores 1, 3, 4, 6, 7, 9, 10 and MAI Total Score (all p<0.05) between residents from U.S. vs.foreign medical schools (for all scores, residents from foreign medical schools scored higher than their counterparts from U.S. medical schools).
Regression analyses suggest that both SRLPS Total and MAI Total Scores can be used to predict ABSITE Score (Figure 1a,1b).

Discussion
Resident performance on ABSITE examinations (and other specialty in-training examinations) one of the most vexing issues faced by Program Directors and educators.In spite of resident and faculty efforts to improve performance, continued poor scores on intraining examinations persist on being a significant problem.
In the past several years this problem has become greatly magnified since importance of the in-training examination has changed.Where scores once represented "low stakes" markers to help residents and Program Directors identify and improve knowledge and skills and to enhance residents' chances of passing Board examinations, they have now become "high stakes" tools used to screen residents for fellowship positions as well as to assess their attainment of milestones.In this era of specialization where many residents are vying for fellowships, good scores on in-training examinations have become an imperative for securing prime positions.Yet, despite the best efforts to improve resident success, for some there is little change.By demonstrating connections between self-regulated learning, metacognition, and intraining examination scores, the outcomes of this study can provide direction to help improve resident performance.One interesting finding was that PGY 4-5 level residents performed significantly better on the MAI Factor 6 score (monitoring) than residents at the other PGY levels.This may be a result of senior residents' experience and becoming more adept than their less experienced colleagues in knowing how to identify and master important data and effectively separate out relevant from irrelevant data (fewer encoding errors when monitoring their learning).This is not surprising since the sheer volume of information to learn in residency programs is formidable and may take several years for residents to discriminate between important and unimportant variables.This difference is often observed in presentations by juniorlevel residents.Unlike more senior residents, they tend to present all aspects of patient cases, not just critical data points.However, it is worth noting that PGY 1 residents displayed higher MAI Factor 6 scores (monitoring) than PGY 2-3 level residents.One quarter of the PGY 1 residents in the study were graduates from foreign medical schools.These graduates face greater challenges applying and getting accepted into U.S. medical schools and may have been taught monitoring strategies to become efficient and effective learners and more favorably compete.MAI Factor 6 scores for PGY 1 residents in this study were among the highest of all residents and may have contributed to them outperforming PGY 2-3 level residents.
Another finding of interest is foreign medical school graduates  In his research on assessing metacognitive awareness, Scraw cited several studies that indicated learners who are more metacognitively aware are more strategic and perform better than those who are unaware [12].He offered the explanation that metacogntively aware learners plan, sequence, and monitors their learning in a way that directly improves their performance [12].This is reinforced in Quirk's work on the development of expertise in medical education [13].He argued for teaching physicians metacognitive skills to help them optimize their learning experiences during training and throughout their careers.Metacognition is currently viewed as one part of the more general construct of self-regulated learning.Although selfregulated learning and its relationship to academic success has been studied extensively over the past fifteen years, the impact of metacognition has not been comprehensively examined [14].
Numerous studies have been conducted on various aspects of self-regulation and metacognition with students at the elementary school, high school, and college levels showing students who use these processes demonstrate better learning and achievement outcomes.In their research on metacognitive development with professional educators, Cooper and Stewart also cited a number of studies which consistently demonstrated a relationship between students' metacognitive knowledge, self-regulation abilities, and academic performance.Although the authors indicated these studies have shown the positive influence of metacognition on performance and cited one study with nurses and electronics technicians as an example of research with adults in the workforce, there have been very few studies examining these linkages with adults after college or in the workforce [15].
In the medical field, evidence reported from studies with medical school students examining linkages between self-regulated learning, metacognition, and learning and performance further confirms the relationship between these constructs and achievement [5,[16][17][18][19][20].
In graduate medical education one study conducted in Brazil with anesthesiology residents examined linkages to academic performance on several variables including their learning and study strategies.Their results revealed that three factors associated with strategic learning (anxiety, motivation, and ability to properly select main ideas) were found to independently predict academic performance [21].Other than the above, there is a lack of such research exploring these relationships with residents and none in general surgery.It is critically important to investigate self-regulated learning in graduate medical education programs since, unlike medical school where instruction is well defined and arranged, residents are virtually on their own to study and must master a significant body of knowledge from diverse sources.It is estimated that as much as 95% of what is learned in surgical residency training occurs in the clinical setting or at home.Because of the clinical and operative demands placed on surgical residents, it is difficult to find large blocks of study time; more often these come in interrupted slivers of time.As a result, residents must become more proactive and skillful in their approach to master the body of knowledge associated with this discipline [22].

Conclusions
The small sample size used in this study limited the ability to carry out formal hypothesis testing.However, the results provide preliminary data that could potentially be used to power a study with a large enough sample size to allow confirmation of linkages between self-regulated learning and high stakes examinations.Additionally, this study was conducted at a single site, thus limiting generalizability.This study can be undertaken in more or larger academic centers and/or in other disciplines where hypothesis can be formulated and tested to definitively affirm the relationships between self-regulated learning and high stakes examinations.In this preliminary study, we were able to show the relationship between self-regulated learning, metacognition and academic achievement.In fact, a resident's score on the SLRP and MAI could potentially predict subsequent ABSITE scores.Identifying those residents at increased risk for academic performance challenges at the start of residency training may be of significant benefit to Surgical Educators.Our results suggest that poor scores on the ABSITE may not purely reflect lack of effort, but rather, expose major issues of how residents approach the act of learning.With the SLRPS and MAI tools, surgical educators can identify those residents with poor insight into their cognitive skills and abilities and deficiencies in being able to plan, organize and implement information management strategies.Creating an organized, feasible study plan from day one may prevent academic shortcomings and failure during and, more importantly, after residency training.

2 3 4 5 47. I try to break studying down into smaller steps 1 2 3 4 5 48. I focus on overall meaning rather than specifics 1 2 3 4 5 49.2 3 4 5 50.
when I encounter important information 1 2 3 4 5 10.I know what kind of information is most important to learn 1 2 3 4 5 11.I ask myself if I have considered all options when solving a problem 1 2 3 4 5 12.I am good at organizing information 1 2 3 4 5 13.I consciously focus my attention on important information 1 2 3 4 5 14.I have a specific purpose for each strategy I use 1 2 3 4 5 15.I learn best when I know something about the topic 1 2 3 4 5 16.I know what an attending expects me to learn 1 2 3 4 5 17.I am good at remembering information 1 2 3 4 5 18.I use different learning strategies depending on the situation 1 2 3 4 5 19.I ask myself if there was an easier way to do things after I finish a task 1 2 3 4 5 20.I have control over how well I learn 1 2 3 4 5 21.I periodically review to help me understand important relationships 1 2 3 4 5 22.I ask myself about the material before I begin 1 2 3 4 5 23.I think of several ways to solve a problem and choose the best one 1 2 3 4 5 24.I summarize what I've learned after I finish 1 2 3 4 5 25.I ask others for help when I don't understand something 1 2 3 4 5 26.I can motivate myself to learn when I need to 1 2 3 4 5 27.I am aware of what strategies I use when I study 1 2 3 4 5 28.I find myself analyzing the usefulness of strategies while draw pictures or diagrams to help me understand while learning 1 2 3 4 5 38.I ask myself if I have considered all options after I solve a problem 1 2 3 4 5 39.I try to translate new information into my own words 1 2 3 4 5 40.I change strategies when I fail to understand 1 2 3 4 5 41.I used the organizational structure of the text to help ask myself if what I'm reading is related to what learn more when I am interested in the topic 1 I ask myself questions about how well I am doing while I am learning something new 1 I ask myself if I learned as much as I could have once I finish a task ABSITE Score= -70.21 + 0.78 (SRLPS Total Score) ABSITE Score = -87.4+ 0.697 (MAI Total Score)

Table 3 :
SRLPS and MAI Component and Total Scores

Table 4 :
SRLPS, MAI Subtest Mean Scores, Total Mean Test Scores, ABSITE Mean Scores, by PGY levels [11]ession analyses suggest that both SRLPS Total and MAI Total Scores can be used to predict ABSITE Score scored higher on almost all MAI factors and MAI Total scores compared to graduates from U.S. medical schools.As previously mentioned, foreign medical school graduates face greater challenges as their U.S. counterparts in applying to U.S. residency programs.Two studies referenced by Desbiens and Vidaillet[10], showed differences in responses to letters of inquiry and requests for applications from U.S. medical school graduates (USMGs) and graduates from foreign medical schools (International Medical Graduates -(IMGs)) that were statistically significantly biased in favor of USMGs over IMGs by a margin of 50 to 100 percent.10Inaddition,according to a 2010 National Resident Matching Program® (NRMP) Program Director Survey, one of the top four factors in resident selection is the United States Medical Licensing Examination® (USMLE) Step 1 scores.[11]Recognizingbias against their graduates and the high importance U.S. residency programs place on standardized examinations, foreign medical schools may incorporate metacognitive teaching strategies into the curriculum to teach students how to be more effective and efficient learners and to better prepare them for achieving a high level of success on standardized examinations.Although we could find no reports or studies in this area, our years of experience reviewing residency applications indicates that foreign medical school applicants have among the highest USMLE examinations scores.In addition, they seem to enjoy a high first-time pass rate on board examinations.Residents and Program Directors can use this information to improve resident performance that is not only evidence-based as shown by the results of this study, but also in similar studies described below that preceded this one.