In this study, the objective was to compare different estimation methods in diagnostic meta-analysis. In this scope, DerSimonian and Laird (DL), Restricted Maximum Likelihood (REML), Sidik and Jonkman (SJ), Hedges and Olkin (HO), Maximum Likelihood (ML), Paule and Mandel (PM) estimation methods were examined. In the implementation part, effectiveness of Clinical Oral Examination (COE) in predicting the diagnosis of histological dysplasia or Oral Squamous Cell Carcinoma (OSCC) was studied. Meta analysis was performed for the data set obtained from 24 studies in accordance with the criteria. Odds Ratio (OR) was used as the effect size. In meta analysis of the random effect model, according to the DerSimonian and Laird (DL) method, the pooled sensitivity value of COE was calculated as 0.953 (95% CI: 0.895-0.979), pooled selectivity was 0.25 (95% CI: 0.124-0.44), and pooled odds ratio was OR = 6.031 (95% CI: 2.208-16.471). According to these results, it can be concluded that COE was not effective in diagnosis. Among the other estimation methods, DerSimonian and Laird (DL) presented the lowest value for I2 and τ2 (I2 = 66.63%, τ2 = 3.489).

Meta-analysis, Diagnostic test, DerSimonian and Laird, Odds ratio

In order to acquire trustworthy findings from a scientific research, it is essential to design a comprehensive study plan, to appropriately collect data, to select adequate statistical methods for evaluation and to interpret the results accurately. Therefore, both the insecurity arising from impractical data, and the inconsistent and contradictory results can be eliminated by combining the previous studies on the same topic. This approach is defined as "meta-analysis" which provides a joint and accurate decision-making opportunity [1]. Meta-analysis aims to predict the related parameters more accurately by increasing the sample size via statistical analysis of the results obtained from the published or unpublished individual studies which are related to a special topic [2,3]. Meta-analysis has been used in 1980s mostly to assess the clinical efficacy of individual medical interventions and since then, it has been a required and advocated statistical analysis in various disciplines [4].

Today, a large number of diseases can be diagnosed and treated. Diagnostic tests which confirm the presence or absence of a disease, give information about the prognosis of the disease and in certain situations, determine the response to treatment have an essential role in medical field [5].

The estimation methods used in meta-analysis have been investigated by numerous studies. Viechtbauer, et al. showed that the Paule and Mandel (PM) estimator is the same as the so-called empirical Bayes estimator [6]. The PM estimation method retains many of the advantages of the method of moments, because it is semiparametric and requires no convergence diagnostics [7]. The moment-based method proposed by DerSimonian and Laird (DL) is most commonly used to estimate the heterogeneity variance. DL method is the default approach in many software routines. Simulation studies have found that the method can be biased and thus, other methods have been introduced [3]. The maximum likelihood (ML) method is asymptotically efficient, but requires an iterative solution. The Sidikve Jonkman (SJ) estimator has methodological similarities with the PM estimator. Although the Hedges ve Olkin (HO) estimator is simple to compute and does not require an iterative numerical solution, it is not widely used [8,9]. On the other hand, restricted maximum likelihood (REML) estimation is a generally well-known estimation technique in the statistical literature [10]. Bowden, et al. conducted an empirical study by comparing DL and PM estimation methods and stated that as the variance between studies increased, τ2 value of PM was greater than τ2 obtained from DL [11].

Diagnostic tests are utilized to identify the presence or absence of a condition in order to develop an appropriate treatment plan [12]. Many performance measures are used to evaluate a diagnostic test. These measures include sensitivity, specificity, false positive rate, false negative rate, Positive Predictive Value (PPV), Negative Predictive Value (NPV), positive likelihood ratio (LR+), negative likelihood ratio (LR-), accuracy, Youden Index (YI) and Diagnostic Odds Ratio (DOR) (Table 1 and Table 2) [13-15].

Meta-analysis is a statistical method that aims to provide more reliable and accurate findings via combining and summarizing the results from previous individual studies [3,16,17]. In 1954, Cochran developed a method for parameter estimation by bringing together researches made in different places, times and areas in an appropriate form [18]. Meta-analysis has been used in 1980s mostly to assess the clinical efficacy of individual medical interventions and since then, it has been a required and advocated statistical analysis in various disciplines [4].

In meta-analysis, different estimations are provided depending upon the contents of the study and these estimations are essential for determining the combined effect and assigning study weights. One of the models utilized in the meta-analysis is the fixed-effects model and the other is the random-effects model [9,16,17].

The fixed-effects model is based on the ground of the assumption that all studies included in the analysis predict the same effect size. In other words, it is assumed that if a trial has an effect, this effect does not interact with the study criteria and it remains constant. In the fixed-effects model, it is assumed that the differences between the effect sizes are the results of the sampling error. In this model, relatively narrower confidence intervals are obtained, accurate information about the homogeneity of the studies cannot be estimated since the between-study variance is not taken into account, and studies with small sample size may not be as sensitive as the ones with large samples [1,9,17,19,20].

The random-effects model makes calculations taking into consideration both the variances between the studies and within each study. The random-effects model assumes that the heterogeneity of all effect sizes arises both from the sampling error and the variations within the study population. Since the between-study variances are taken into account with this model, the homogeneity of the studies can be assessed, and it is more sensitive in small sample sized studies [1,9,17,19,20].

Methods to estimate between-study variance

In the meta-analysis, there are various methods to estimate the between-study variance. Some of those are DerSimonian-Laird (DL), Restricted Maximum Likelihood (REML), Sidik and Jonkman (SJ), Hedges and Olkin (HO), Maximum Likelihood (ML), and Paule and Mandel (PM) estimation methods (Table 3) [3,9,10,21].

DerSimonian and Laird (DL) method: DL estimator is a non-iterative method that is frequently used as the default approach in many softwares [3,9]. τ2 which is the between-study variance for random effect size model, and wi which is the reverse of fixed effect variance for each study are used to calculate the new weights as

w∗i=1[vi+τˆ2]

From here τˆ2,

τˆ2=⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪Q−(k−1)∑ki=1wi−∑ki=1wi2∑ki=1wi0,,Q≥dfQ < df

When τˆ2 equals to zero, it transforms from the random-effects model to the fixed-effects model.

Q=∑ki=1wi(lnORi)2−(∑ki=1wilnORi)2∑ki=1wi

The above-mentioned Q value is calculated as

lnT¯¯¯DL=∑ki=1w∗ilnORi∑ki=1w∗i

The combined effect size is calculated as

lnT¯¯¯DL=∑ki=1w∗ilnORi∑ki=1w∗i

The variance of the combined estimation is calculated as

Var(T¯¯¯DL)=1∑ki=1w∗i

and % (1 – α) the confidence interval is calculated as stated below

exp[ln(T¯¯¯DL)−zα2Var(T¯¯¯DL)−−−−−−−−−√]≤θ≤exp[ln(T¯¯¯DL)+zα2Var(T¯¯¯DL)−−−−−−−−−√] [17,22,23].

Restricted Maximum Likelihood (REML) method: REML estimation method is a well-known technique in the statistical literature and in this estimation method, the between-study variance (τ2) is calculated via double-iterative process. The first iteration includes the estimation of maximum likelihood estimator of τˆ2ML [3,8,10]. The estimate of τˆ2REML is obtained by the derivative of the restricted log-likelihood function.

lnL(τ2)=−k2ln(2π)−12∑i=1kln(vi+τ2)−12∑i=1k((yi−μˆRE(τˆ2ML))2vi+τ2−12ln(∑i=1k1vi+τ2)

with respect to τ2 equals to zero and the resulting solution of the equation for τ2 is,

τˆ2REML=max{0,∑ki=1(wi,RE)2((yi−μˆRE(τˆ2ML))2−vi∑ki=1(wi,RE)2+1∑ki=1wi,RE}

From here, it is provided via

wi,RE=1vi+τˆ2REML

[9]

Sidik and Jonkman (SJ) method: This estimation method is proposed by Sidik and Jonkman and it is a non-iterative technique based on weighted least squares method [24]. To obtain the SJ estimator rˆi=viτˆ20 (assuming τˆ20 ≠ 0) and with this equation qˆi=rˆi+1 values are calculated.

Here, τˆ20=∑ki=1(yi−y¯)2k is the initial estimate of the between-study variance. Then, the SJ estimator is obtained by setting the quantity ∑qˆi−1(yi−μˆqˆ,RE)2 equal to its expected value τˆ2SJ=1k−1∑ki=1qˆi−1(yi−μˆqˆ,RE)2 [9].

Hedges and Olkin (HO) method: Hedges and Olkin estimation method was first defined by Cochran [18]. Hedges (1983) discussed the estimation method for the between-study variance component in the meta-analytic context. The estimator is obtained by setting the sample variance

S2y=1k−1∑i=1k(yi−y¯)2

equal to its expected value and solving τ2, which yields

τˆ2HO=max{0,1k−1∑i=1k(yi−y¯)2−1k∑i=1kvi} (9)

Maximum Likelihood (ML) method: ML estimator is asymptotically efficient but it requires an iterative solution. According to the marginal distribution of yi ~ N(μ,vi + τ2), the estimation of τˆ2ML is obtained by maximizing the log-likelihood function.

lnL(μ,τ2)=−k2ln(2π)−12∑i=1kln(vi+τ2)−12∑i=1k(yi−μ)2(vi+τ2)

Setting partial derivatives with respect to µ and τ2 which are equal to zero, and solving the likelihood equations for the two parameters to be estimated, the ML estimators for µ and τ2 can be obtained as follows

μˆRE(τˆ2ML)=∑ki=1wi,REyi∑ki=1wi,RE

τˆ2ML=max{0,∑ki=1(wi,RE)2((yi−μˆRE(τˆ2ML))2−vi)∑ki=1(wi,RE)2}

From here, it is provided via

wi,RE=1vi+τˆ2ML [9]

Paule and Mandel (PM) method: The Paule and Mandel estimation method has most of the advantages of the method of moments due to its'' semiparametric characteristics and the lack of requirement of convergence diagnostics [7]. This method is essentially equivalent to the Emprical Bayes estimator discussed by Morris [9,25]. Using the random effect weights, this method is equivalent to empirical Bayes method. Paule and Mandel, proposed a special form of Q with ai equation [26].

τˆ2PM=Qgen−(∑ki=1aivi−∑ki=1a2ivi∑ki=1ai)∑ki=1ai−∑ki=1a2i∑ki=1ai

ai=wi,RE=1vi+τ2.

The generalized Q statistic is

Qgen=∑wi,RE(yi−μˆRE(τ2))2∼χ2k−1 [9]

In the application section of the study, we conducted a meta-analysis in order to evaluate the effectiveness of Clinical Oral Examination (COE) for predicting the diagnosis of oral dysplasia or Oral Squamous Cell Carcinoma (OSCC) of mucosal lesions that were submitted for biopsy and were diagnosed histologically. A Clinical Oral Examination (COE) is the principal strategy used to detect abnormal oral mucosal changes including OSCC and oral dysplasia, which is the initial stage of cellular transformation to malignancy [27]. Automated literature searches of articles published from Jan 1, 1966, through Jan 20, 2010, was completed by using the PubMed, Web of Knowledge and the Cochrane Library databases via using the search terms "oral mucosal lesion screening" and "oral lesions". A total of 1,252 articles have met the inclusion criteria (1,195 studies in PubMed, 38 in the Cochrane Library and 19 in Web of Knowledge). Additional articles which included clinically detected lesions that were identified by means of visual examination and other visual techniques were also entered as subsets of data. In all enrolled studies, the main inclusion criterion was the presence of histological diagnoses which were obtained after tissue biopsy of clinically detected oral mucosal lesions.

In conclusion, twenty-four observational studies which included 7,079 patients and 1,956 biopsies met the inclusion criteria [27]. The analyses for diagnostic test and meta-analysis of the data were performed by using Open Meta-Analyst, R Packages, Meta Essential 1.4, STATA 13.0 statistical software.

First of all, the sensitivity, specificity, odds ratio, accuracy, Positive Likelihood Ratio (PLR), Negative Likelihood Ratio (NLR), Positive Predictive Value (PPV), Negative Predictive Value (NPV), and Youden Index (YI) were calculated (Table 4). When the Odds Ratio values (OR) of the studies were considered, both very high (OR = 4815) and very low OR values (OR = 0.068) were observed. The accuracy value, which is expected to be high in a favorable diagnostic test, has varied between 0.065 and 0.995 among the studies.

The Q test was utilized to evaluate the heterogeneity between studies. As a result, it was assessed that the studies were heterogeneous (Q = 68.94, p = 0.00 < 0.05). Thus, random effect model was used for meta-analysis.

Using Open Meta Analyst statistical software, the meta-analysis of the random effect model that was performed according to the DerSimonian and Laird (DL) estimation method revealed that the pooled sensitivity value of COE was high [0.953 (95% CI: 0.895-0.979)] and the pooled specificity was low [0.25 (95% CI: 0.124-0.44)] (Table 5).

When the PLR and NLR were considered, the pooled PLR value was 1.053 (95% CI: 1.00-1.11) and the pooled NLR was 0.469 (95% CI: 0.341-0.645). In general, a PLR value above 10.0 indicates that the test makes a significant contribution to the diagnostic process and a NLR below 0.2 indicates that the test is good at ruling out diseases [15,28]. Additionally, PLR and NLR values of 1 demonstrate that the test provides no information about the likelihood of the disease. In our study, the pooled odds ratio (OR) was 6.031 (95% CI: 2.208-16.471), revealing the ineffectiveness of the COE in prediction of oral dysplasia or OSCC.

The results of the analyses obtained with DL, REML, ML, PM, HO, and SJ estimation methods using R, Open Meta Analyst, Meta Essential softwares are presented in Table 6. The DL estimation method was present in all the software programs used in the study. The Q statistic value that was calculated for evaluation of the homogeneity by using the DL method in R, Open Meta Analyst and Meta Essential softwares yielded to 68.943 (p < 0.0001), and the lowest I2 and τ2 values were obtained. Based on these results, it can be concluded that a moderate level of heterogeneity was present. In R and Open Meta Analyst softwares, Restricted Maximum Likelihood (REML), Maximum Likelihood (ML), Paule and Mandel (PM), Hedges and Olkin (HO) and Sidik and Jonkman (SJ) estimation methods were utilized and similar results were obtained with both softwares. According to the results, the highest I2 and τ2 values were obtained using the non-iterative SJ estimation method. The analysis with the PM estimation method which is simple and does not require distributional assumption, the lowest I2 value was obtained following REML and ML estimation methods (I2 = 72.80%).

The publication bias was investigated by using the Egger weighted regression method and a funnel plot chart was prepared (Table 7 and Figure 1). Egger regression method and the funnel plot chart showed that, with 95% confidence intervals, publication bias was not present (p = 0.087 > 0.05).

The results of our study indicate that Clinical Oral Examination (COE) is not a sufficient technique for the diagnosis. Except Der Simonian and Laird (DL) estimation methods, analyses could be performed with other estimation methods in ready-made softwares. Furthermore, it can be concluded that the appropriate software program for meta-analysis varies depending on the user's needs and preferences.

The authors declare that they have no conflict of interest.