ACGME has put great emphasis on scholarly activity during residency, yet ACGME has not been able to identify gaps in achieving these goals. We possess longitudinal data for the last 10 years across our residency training programs. With this, we have identified missing gaps as well as propose interventions to address gaps.
The research design employs utilizing a standardized research instrument (questionnaire), answered by every resident at the onset and completion of residency training. This questionnaire assesses resident's comfort level, knowledge to critically appraise research and literature and knowledge of biostatistics, epidemiology etc. It also covers questions related to career goals, opinions on research environment, and opinions regarding the effect of research on future practice, time constraints etc.
Less than half (44%) of the residents who took the pre-test passed the 11 basic statistical questions (e.g. null hypothesis, p-value). Preliminary analysis showed more than 8/10 residents wish to learn more about research and statistical methods at the onset of their residency training. This proportion drops to 7/10 at the end of residency. The mean overall score for basic concepts regarding epidemiology and biostatistics for pre and post-test was unchanged across different residency programs across years.
Our study showed current training structure has minimal impact on helping residents to achieve research goals per ACGME standards. A more thorough evaluation helped us identify more specific trends as well as propose interventions to address it.
Resident, Research, Internal medicine, Scholarly activity
It has been consistently noted and realized that there is a paucity of physician-researchers in the United States [1,2]. A research mind ideally should flourish in medical school. However, most students wait till they step into their practical lives to harness this vital aspect of medicine as course work and other priorities make it hard for them to focus on research. Residency programs, especially community programs, provide physicians who have little interest or exposure to research with a conducive environment to acquire this additional yet essential skill.
Conducting research during residency benefits both the resident and the program. On an individual level, it is associated with choosing an academic career [3-6], for an institution or program it has a different impact, i.e. residents with more research would be more likely to pursue academic careers and hence more physician researchers added to the work force [7]. Most importantly, research residents are more comfortable applying the principles of "evidence-based medicine," enhancing their analytical and critical thinking skills, an essential component to improving patient care quality [8]. Unfortunately, improving resident research skills is often overlooked and neglected in many training programs.
The Accreditation Council for Graduate Medical Education (ACGME) recognizes this need and has placed great emphasis on the importance of residents engaging in clinical research during residency. It requires residency programs to provide curricula that propagate the resident's knowledge of research, allocate resources to achieve this goal and report resident scholarly activity in different forums. However, several missing components are pivotal to achieving this goal. Some of them worth highlighting are, ACGME was not able to identify gaps in achieving the goals as mentioned above. Despite studies conducted by different institutions to identify these gaps, no longitudinal data is available. In the absence of any longitudinal data identifying gaps across different institutions, ACGME has been unable to provide residency programs with specific recommendations to bridge these gaps. Lack of resident scholarly activity is one of the most frequent citations by the ACGME Resident Review Committee.
Marshfield Clinic is in a unique position to address these issues. With the resources at our disposal, we were in a pivotal position to address this essential but yet often neglected issue. We possessed longitudinal data for the last ten years across our residency training programs. With this, we have identified missing gaps in resident research, based on which we can propose interventions to address this. We believe that by identifying these gaps and addressing them, we will be able to enhance scholarly activity among residents, which will be reflected by the increased number of abstracts and posters at state and national meetings, increase in number and better quality improvement projects and increased number of manuscripts.
Our goal was to assess and analyze data (pre- & post-residency) that has been collected from the last ten cohorts of residents (2008-2017) across all offered residency training programs at Marshfield Clinic. Our long term goal was to identify areas of need unique to an individual program, common denominators across several programs, and to propose interventions that can help rectify and improve these areas. This improvement would not only be reflected in post-test assessment questionnaires but also in research output by our residents. Our unique questionnaire has been distributed to residents longitudinally over the years for a decade and also across multiple specialties. We were therefore uniquely poised to compare and contrast multiple specialties with one another as well as to capture trends across the same specialty over the years. This questionnaire also helped us identify "need assessments" critical to our project. We felt the need to assess the entire resident research program in addition to individual improvements in research knowledge and attitudes of residents. The assessment questionnaire was created combining questions from three disciplines: Survey/test from JAMA article (Yale University) [9]; survey/test from Pharmacy (University of Michigan) [10]; and survey/test from nursing (Royal Hospital - England) [11]. The final questionnaire includes demographics, attitudes about research and biostatistics, basic study design, and biostatistics.
Program Directors discussed the need for baseline markers for all outputs and a reliable way to evaluate the program in general and individual resident progress. The short and medium term outputs must be easily measured. The long term outputs are directly tied to the vision statement of the curriculum and will be more difficult to measure.
Our approach was to identify critical elements we believed were integral to a successful research curriculum for residents. One key element of this questionnaire was to assess the resident's comfort level and knowledge to appraise research and literature critically. Besides, this questionnaire assessed the resident's knowledge of biostatistics, epidemiology and research design. Our questionnaire gave a general impression of a resident's willingness to get involved in research projects and to assess their intellectual curiosity. The questionnaire also assessed their involvement in collecting and analyzing data, formulating a research question, training in evidence based medicine. Also this questionnaire gives an idea if residents have the tools to interpret medical literature and if they can practice evidence-based medicine.
We had a total of 158 pre-test and 117 post-test questionnaires from different specialties. Respondents included over 200 MDs and 17 DOs across different specialties including Internal Medicine, Surgery, Dermatology, Pediatrics and Med-Pediatrics. That gave us a unique opportunity to compare and analyze data across as well as among different specialties, to identify trends and gaps, and as a second part of our project would help to propose interventions. Hopefully, continued data collection will allow evaluation of any interventions over time.
Standard descriptive statistics are presented to summarize resident characteristics and responses to the assessment survey questions, pre-test, and post-test. Responses to individual, ordinal-scale questions were compared between test times using chi-square procedures (both general and trend tests). Questions relating to study design, biostatistics, and epidemiology, were scored relative to the most appropriate response. Overall scores for the 21 questions were then compared between test times and among groups defined by the resident program and/or resident characteristics using nonparametric procedures (Wilcoxon and Kruskal-Wallis tests).
Table 1 gives a breakdown of different residency programs in pre and post-test questionnaires. Internal medicine had the highest number of participants both in pre and post-test followed by pediatrics, surgery, med/pediatrics, and dermatology. Questionnaires were started collecting in 2008 with the first post-test questionnaire in 2010. Data collection is an ongoing process, and here we include results till the batch year 2015 with post-test questionnaires till 2018. Simple concepts tested were the null hypothesis, p-value, outliers, sensitivity, specificity, types of research studies including cohort, case-control, incidence, prevalence; confidence intervals (Appendix 1A and Appendix 1B).
Table 1: Participants across residency programs completing questionnaires. View Table 1
The first sections of the questionnaire address the respondents' interest in, and comfort level with, statistics and utilizing research literature. Preliminary analysis of our aggregated data shows that most residents wished to learn about research and statistical methods at the onset of their residency training. Some of the questions asked in particular to their interest were, "exploring evidence using Cochrane Database, finding clinical practice guidelines for patient care questions". Interestingly but not surprisingly there was a decrease in the interest of residents from 50 to 38% (p = 0.018) at the end of their residency. However, despite their loss of interest when asked about evidence-based medicine like finding clinical practice guidelines for patient care questions and exploring evidence using Cochrane database among other questions, more residents acknowledged the importance and need to do that (p < 0.001 for both). Similar trends were noted across all residency programs p < 0.001 for both evidence-based medicine and Cochrane database. This trend probably illustrates the inefficiency of the current structure of improving resident clinical research skills.
While 9 in 10 residents acknowledge that possessing research skills is essential to interpreting clinical research, less than 30 percent of residents are comfortable with interpreting research findings in clinical journals at the end of their residency. Less than half (44%) of the residents who took the pre-test passed the 11 simple/basic statistical questions (for example, what is the null hypothesis, what is P-value) (Table 2).
Table 2: Resident assessment study - Pre-(2008-2016) and Post-(2010-2018) test comparisons by percentage of correct responses on 11 statistical related questions. View Table 2
The mean overall score for basic concepts regarding epidemiology and biostatistics for pre and post-test was 11.8 and 11.6 respectively, p = 0.804 (Table 3). The trend was not much different for the internal medicine residency program where pre and the post-test score was 11.7 and 11.1, p = 0.190 (Table 4). Surgery residents had the highest scores compared to other residency programs with 12.8 for both pre and post-test, (p = 0.797, results not shown). None of the results across residency programs on post-test questionnaires were statistically significant.
Table 3: Scored results by test time. Overall Score*: All Residency Programs. View Table 3
Table 4: Scored results by test time. Overall Score: Internal Medicine Residency Program. View Table 4
We then looked at scores by gender, age group, years since graduation, foreign versus US graduates, prior courses are taken in epidemiology or biostatistics or training in evidence-based medicine, none of which were statistically significant (results not shown).
The number of posters or abstracts for the annual meeting of the state chapter of the American College of Physicians across years for the Internal Medicine residency program were unchanged across all these years.
Very little research has been done on the resident scholarly activity and most of what has been done is sporadic in nature with no longitudinal data available over the years. Moreover, most of these studies have been done across one specialty only at an institution.
With leadership consensus there was a clear understanding that residents and other physicians need to be fluent in current literature, which leads to the underlying need to know basic statistics or at least know about basic statistical methods - when and why used, know their baseline, identify some impediments and to propose interventions to overcomes this reflected by their research output.
Our preliminary analysis identified exciting trends. We had identified a very high interest of residents in research at the onset of training. We also saw that the current training structure has minimal impact on helping residents to achieve research goals per ACGME standards. A more thorough evaluation helped us identify more specific trends as well as propose interventions to address it.
In this study, we examined factors influencing resident research in our residency training program over a 10-year period. We observe that within and across our residency training program, there were no significant differences in pre and post test scores at the beginning and end of residency training. This finding indicates that there was minimal improvement in research knowledge and aptitude of our residents during residency training.
As reported in previous studies, we observe that interest in research was high at the beginning of residency, however this gradually declined as residency training progressed [12,13]. This finding may be explained by the impact of the challenging nature of residency training on research interests. In a survey by Pawar, et al. lack of time was cited by over 70% of residents as a major obstacle to engaging in research [12]. According to the current ACGME guidelines, residents can work for up to 80 hours on a weekly basis. Such a heavy work load definitely reduces the opportunities to engage in research opportunities.
Past studies suggest that the perceived importance of research on clinical practice may impact resident engagement in research. Our study shows that majority of residents acknowledge that evidence based medicine significantly influences clinical practice. We observe 9 of 10 residents understand that interpreting medical literature is dependent on acquiring adequate clinical research skills. Regardless, a sizeable proportion of residents were comfortable with interpreting medical research and many of them were not actively engaged in research activities.
There are also program specific factors that impact resident research. For example, some residency training programs lack faculty who are trained in clinical research [13]. This inherently limits mentorship opportunities for residents who want to engage in clinical research. More so, while some programs have research faculty, residents also indicate that faculty lack time to mentor them in research or they lack specific curriculums in clinical research for their residents. Pawar, et al. also noted that nearly half of residents indicated that a lack of research training curriculum impeded their ability to engage in research activities [12]. These finding illustrates opportunities for residency programs to provide mechanisms to facilitate resident engagement in research during residency training.
In a recent study by Oliver, et al. which evaluated the role of a dedicated research curriculum on resident scholarly output, residents who were exposed to a dedicated research curriculum had significantly higher number of scholarly activities as evidenced by PubMed indexed publications, online publications and conference presentations [14]. Beyond having a research curriculum, it appears mandating residents to submit scholarly work prior to graduation increased scholarly activity. A national study by Geyer, et al. demonstrated that programs with a research requirement for residents were more likely to have residents with oral or poster presentations compared to those that did not [15]. These programs were also more likely to have residents with successful fellowship applications.
However, while many residency programs may have a research curriculum, it is unclear if these curricula are effective towards fostering resident research activity. As observed in our study, our current research curriculum does not appear to influence the outcomes of resident scholarly activity. We observed no difference in pretest and post-test questionnaires administered to residents at the beginning and conclusion of their training. We also observed no association between resident results and other resident characteristics including years from graduation, age group, foreign versus US trained and gender. Similarly, we saw no differences in the scholarly output in form of conference presentations at the state American College of Physician conference over the study period. Similar trends have been noted in other residency programs in the United States [14].
However, it appears that residency programs are willing to make changes and adopt a more concise research curriculum. Amrhein, et al. evaluated their residency training program's willingness to institute changes to enhance resident research and observed many residents and faculty were strongly in support of instituting research curriculums [16]. These findings likely reflect the attitude of residency programs across the United States towards improving resident research activities.
Based on the data we collect we will propose interventions. These will mostly be included in the second part of the project. Interventions proposed include but not limited to hiring a dedicated research scientist, developing an organized research elective, provide resources for writing skills, incorporating research lectures as part of didactics throughout residency training, mentorship and promoting research by implementing a resident reward system to name a few.
Outcomes for this study as phase II of the project will be measured by the number of presentations at local meetings, regional and national meetings. Outcomes will also include the number of papers submitted to journals and articles published. We will include both oral and poster presentations. Papers submitted include case reports as well as traditional manuscripts.
Actively participating in scholarly projects helps enhance self-esteem and satisfaction with residency training [17] and at the same time also enhances the trainee's capacity to practice evidence-based medicine [18]. Besides, this would also greatly enhance faculty experience who are supervising these physicians, especially in community programs.
As we know any intervention without needs assessments would be ineffective or inefficient or both. That would also help prevent misallocation of resources including designation of resources to areas residents have previously mastered or allocation of insufficient funds to areas in need. To our knowledge, no such prospective data is available in any other institution in any other residency program across the US. We, therefore, are in a unique position not only to address this issue but also provide a model for other residency programs especially community programs to follow suit.
There are no relevant financial disclosures.
There are no potential conflicts of interest for any authors listed.