Research in Emergency Medicine (EM) serves as the basis for informed, reasonable, effective medical practice and knowledge of research methods is a requirement of residency training in EM.
Our objective was to catalog the statistical methods used in four EM journals to find the most commonly employed techniques. We examined what types of studies were reported.
This was a cross-sectional observational study. We examined original research articles and meta-analyses published from July 2016 to June 2017. The journals included were Academic Emergency Medicine, the American Journal of Emergency Medicine, Annals of Emergency Medicine, and the Journal of Emergency Medicine. For each article, we recorded the type of study performed and collected the number and type of statistical tests used. The use of statistical software packages was recorded.
We evaluated 545 articles. Almost 60% of them were cohort studies; 17% were randomized controlled trials, 15% were cross-sectional studies and 4% were meta-analyses. The mean number of statistical tests per article was 4.16 (SD = 1.98), with a median of 4 (IQR = 2). The top ten statistical techniques applied (descriptive statistics, confidence intervals, contingency tables, t-tests, epidemiologic statistics, non-parametric tests, regression analysis, power analysis, multiway tables, and non-parametric correlation) comprised 84% of all the methods employed.
An extensive array of statistical methods is utilized in EM research. The number and sophistication of statistical methods employed in the EM literature presents challenges for both practicing emergency physicians and residency educators and underscores the need for curriculum development.
Statistics, Education, Curriculum, Residency, Emergency medicine
Research in emergency medicine (EM) serves as the basis for informed, reasonable, and effective medical practice. Knowledge of research methods and statistical analyses is essential in interpreting the abundance of research in EM. Moreover, knowledge of the basic principles of research, including how research is conducted, evaluated, and applied to patient care is a requirement of residency training in EM [1].
Previous studies have examined the frequency of use of various statistical techniques in a variety of medical journals [2-7]. This has not been studied in an emergency medicine journal in a number of decades [8,9]. In an effort to ascertain the current level of statistical literacy required to have a reasonable understanding of published studies in EM, we undertook a study of the methods and statistical analyses used in various emergency medicine journals (Table 1).
Table 1: Categories of statistical procedures in emergency medicine. View Table 1
The objective of the study was to catalog the statistical methods used in four emergency medicine journals to find the most commonly employed techniques. This data can provide some indication of the level of statistical literacy required to have a good understanding of published studies; moreover, it will serve to guide residency educators in developing and refining their research curriculum. We also examined what types of studies were reported and which software packages, if any, were employed by the authors.
This was a cross-sectional observational study of the statistical methods employed in four peer-reviewed EM journals. We examined original research articles, systematic reviews, and meta-analyses published from July 2016 to June 2017. We did not examine summaries of extant research such as commentaries, opinion pieces, reports, letters, or editorials. The journals reviewed were Academic Emergency Medicine, the American Journal of Emergency Medicine, Annals of Emergency Medicine, and the Journal of Emergency Medicine.
For each journal, issues published from July 2016 to June 2017 were included. Each investigator was assigned one journal and identified articles in each issue as fitting inclusion criteria or not. After identifying those that fit criteria, for each article we recorded the type of study performed and the number and type of statistical techniques used. If no statistical tests were employed, that was recorded. Individual statistical tests were grouped into larger categories using a modification of the methodology of Sato, et al. [7]. For example, the categories for simple linear regression and multiple regression were combined into one category entitled "regression analysis". We also added two categories for recording the use of confidence intervals and for other statistical tests not already covered in one of the existing categories. We did not comment on the appropriateness of using a particular statistical test, since the purpose of this investigation was to ascertain which specific tests are being used the most, and not whether their use is justified or correct. If the use of a statistical software package was mentioned, it was recorded.
Data entry and interrater reliability was assessed by having 55 of the articles (10%) randomly selected and checked by another researcher for agreement. Discrepancies were resolved by consensus. Interrater reliability was determined using Cohen's kappa. This study was reviewed by our institutional IRB and found to be exempt.
We evaluated 545 articles. The types of research studies found are depicted in Figure 1. Three hundred twenty (59%) were cohort studies; 90 (17%) were randomized controlled trials, 83 (15%) were cross-sectional studies, and 23 (4%) were meta-analyses. The remaining 29 were a variety of other types of studies.
Figure 1: The types of research studies found.
View Figure 1
The individual and cumulative frequency distributions of the statistical methods employed are presented in Figure 2. The bars represent the absolute number of times a technique was utilized, while the line is a graph of the cumulative frequency. The mean number of statistical tests per article was 4.16 (SD = 1.98), with a median of 4 (IQR = 2). Only 33 articles (6%) employed descriptive statistics alone. The top ten statistical techniques (descriptive statistics, confidence intervals, contingency tables, t-tests, epidemiologic statistics, non-parametric tests, regression analysis, power analysis, multiway tables, and non-parametric correlation) comprised 84% of all the methods employed (Table 2).
Figure 2: The individual and cumulative frequency distribution of the statistical methods employed.
The bars represent the absolute number of times a technique was utilized, while the line is a graph of the cumulative frequency. ROC: Receiver Operating Characteristic.
View Figure 2
Table 2: Statistical Methods by Journal. View Table 2
The five most common statistical software packages utilized were SPSS (35%), SAS (20%), Stata (19%), Excel (9%), and R (5%). In 23% of articles no software package or application was mentioned.
The interrater reliability was almost perfect with a kappa of 0.92.
A multiplicity of statistical techniques was employed by researchers in four prominent peer-reviewed emergency medicine journals during a one-year period. Based on our data, having an understanding of the ten most utilized methods would allow a reader to be familiar with 84% of all the statistical techniques applied.
Compared to earlier studies, it is apparent that the number and variety of statistical methods utilized by EM researchers is increasing and becoming more diverse. Compared to findings by Menegazzi, et al. [9] in 1991, our study shows a substantial decrease in the number of articles with only descriptive statistics (31% versus 6%); while increased use of confidence intervals (2% versus 48%), nonparametric tests (7% versus 32%), regression analysis (8% versus 20%), and nonparametric correlation (1% versus 12%) was noted.
In 2003, Reed and colleagues cataloged the statistical methods used in six journals from the fields of family practice, emergency medicine, and obstetrics and gynecology [10]. If we focus only on the results from the two EM journals, we find additional methods being employed, epidemiologic statistics, receiver operating characteristic (ROC) curves, and survival methods. Our study reveals that currently researchers are employing even more sophisticated methods, such as power analysis, multiway tables, and sensitivity analysis.
The increasing variety and complexity of statistical techniques employed by EM researchers presents challenges for both residency training programs and practicing emergency physicians. The scope of expected statistical literacy is expanding and it benefits the resident and the practicing physician to have an understanding of the principles underlying statistical analysis and some familiarity with the actual techniques employed by investigators so that they may better evaluate the results of research and how it may be applied to their practice environment. This investigation will help to inform educators in residency training programs in their research curriculum development and refinement.
Although experience with statistical software is not required for the practicing emergency physician, knowledge of which statistical packages are most commonly used would be useful for the EM researcher. In our study, the most utilized statistical software packages reported were SPSS, SAS, and Stata.
Our study has several limitations. First, it reflects only one-year snapshot of four journals and did not attempt to track trends over time. Second, only EM-specific journals were included. EM related research is also presented in a broad range of other journals and they may include methods not identified in our survey. Finally, we did not attempt to ascertain whether the methods used were the most appropriate for the research hypothesis studied.
An extensive array of statistical methods is utilized in emergency medicine research. Identification of these methods provides a structure for developing a research curriculum for residents and training for practicing physicians. The increased number and sophistication of statistical methods employed in the EM literature may present challenges for both practicing emergency physicians and residency educators.
None of the authors have any competing interests.
We received no outside support for this project.
All of the authors contributed equally to this project.