Keywords

Repeated binary measurements, Bivariate bernoulli distribution, Correlation statistic, Gastric emptying

Introduction

A diagnostic test is an approach used to gather clinical information for making a clinical diagnosis. Assessment of diagnostic tests, especially the likelihood ratios, is very important in clinical trials and diagnostic medicine [3-6]. There exist two versions of likelihood ratios known as positive likelihood ratio and negative likelihood ratio. A positive likelihood ratio is the ratio of the probability that an individual with the disease tests positive to the probability that an individual without the disease tests positive; a negative likelihood ratio is the ratio of the probability that an individual with the disease tests negative to the probability that an individual without the disease tests negative. An asymptotic hypothesis test is constructed to compare the positive and negative likelihood ratios [7]. The likelihood ratios have been paid much attention to measures diagnostic tests for repeated binary outcomes [1,8-10].

Although the likelihood ratios have their many advantages in clinical findings, laboratory tests, imaging studies, etc, they have some limitations [11-16]. Furthermore, identification of an appropriate compromise between the likelihood ratio positive and the likelihood ratio negative may depend on the investigators' knowledge and experience in clinical trials. Youden's index is widely used as a single statistic to evaluate different test algorithms [17], but it may not perform well when a subject has a low prevalence.

Considering the limitations, we propose a correlation statistic based on a bivariate Bernoulli distribution to assess the diagnostic tests for repeated binary outcomes. A bivariate Bernoulli distribution for binary variables was studied [18], but the authors did not give an explicit expression of bivariate distribution. In contrast with the likelihood ratio positive and the likelihood ratio negative, our correlation statistic strikes a sort of balance between sensitivity and specificity. It is based on the difference between the chance of two independent subjects with both true positive and true negative and chance of two independent subjects with both false positive and false negative.

This paper is organized as follows. In Section 2, we present a new form of bivariate Bernoulli distribution with an explicit expression and give the maximum likelihood estimators of the parameters. The correlation statistics proposed as a measure for the evaluation of the test performance in Section 3.

Through the simulated studies in Section 4, we demonstrate that the estimators perform well and that our proposed statistical measure is advantageous when the prevalence is low. The advantage of our statistical measure is further demonstrated by analyzing a data set on gastric emptying in Section 5. Some discussion is given in Section 6. Lemmas and theorems and their proofs are provided in the Appendix.

Bivariate Bernoulli Distribution and Parameter Estimation

First, we introduce a motivating example for our paper. Delayed gastric emptying results from a variety of chemical and functional etiologies. Gastric emptying studies are the standard of measuring gastric emptying, the protocol has been standardized [19]. Our aim is to establish the predictors to assess the possibility of decreasing the study time. This study is the retrospective chart review study design from Feb 2009 to May 2011. There are 600 subjects in our study. Their gastric emptying is the percentage of remaining radioactivity in the stomach, measured at 0, 1, 2, 3 and 4 hours after ingestion.

According to the standard protocol, a gastric emptying result for a subject is abnormal (or normal) at 2 and 4 hours if the percentage of remaining radioactivity is greater than 60% and 10% (or less), respectively. Here the gastric emptying results measured at 4 hours are used as subjects' actual results while the gastric emptying results measured at 2 hours are used as predicted results. The gastric emptying results at 2 hours and at 4 hours on the same subject are repeated measurements over time and then are often correlated to each other.

This motivates us to develop a unique and feasible statistical measure that accounts for this correlation in the assessment of test performance. Let Y₁ and Y₂ be the actual result and predicted result for a subject, respectively, where each of the actual and predicted results is either positive or negative. For simplicity, let 1 and 0 denote a positive result and a negative result, respectively. As we've known that if one of $p_1\text{ = P}\left(Y_1\text{ = 1}\right)$ and $p_2\text{ = P}\left(Y_2\text{ = 1}\right)$ is 0 or 1, then Y₁ and Y₂ are independent. Here to avoid the degenerate cases, we assume from now on that $0{\text{ < p}}_1\text{ < 1}$ and ${\text{0 < p}}_2\text{ <1}$. The correlation coefficient of Y₁ and Y₂ is given by

$$\rho \text{ = }\frac{p_{11}-p_1{\text{p}}_2}{\sqrt{p_1\left(1-p_1\right)p_2\left(1-p_2\right)}}(1)$$

Or equivalently,

$$\rho \text{ = }\frac{p_{11}{p}_{00}-p_{10}{\text{p}}_{01}}{\sqrt{p_1\left(1-p_1\right)p_2\left(1-p_2\right)}}(2)$$

Where $p_{y_1{y}_2}=\text{ P}\left({\text{Y}}_1=y_1,{\text{Y}}_2=y_2\right)$, y₁, y₂ = 0,1 and p₁₁ – p₁p₂ is the probability that two independent subjects have both true positive and true negative and probability that two independent subjects have both false positive and false negative. The correlation coefficient may then be viewed as a standardized difference on a certain range. The following lemma shows the range for the correlation coefficient.

LEMMA 1.

$$\begin{array}{l}\left(i\right)-\min \left\{\sqrt{\frac{p_1{p}_2}{\left(1-p_1\right)\left(1-p_2\right)}},\sqrt{\frac{\left(1-p_1\right)\left(1-p_2\right)}{p_1{p}_2}}\right\}\le \rho \le \\ \text{ min }\left\{\sqrt{\frac{p_1\left(1-p_2\right)}{\left(1-p_1\right)p_2}},\sqrt{\frac{\left(1-p_1\right)p_2}{p_1\left(1-p_2\right)}}\right\}\end{array}$$

$\left(ii\right)\rho =\min \left\{\sqrt{\frac{p_1\left(1-p_2\right)}{\left(1-p_1\right)p_2}},\sqrt{\frac{\left(1-p_1\right)p_2}{p_1\left(1-p_2\right)}}\right\}$ if and only if p₁₁ = min {p₁, p₂}.

In particular, $\rho$ = 1 if and only if p₁₁ = p₁ = p₂.

$\left(iii\right)\rho =-\min \left\{\sqrt{\frac{p_1{p}_2}{\left(1-p_1\right)\left(1-p_2\right)}},\sqrt{\frac{\left(1-p_1\right)\left(1-p_2\right)}{p_1{p}_2}}\right\}$ if and only if p₁₁ = max {0, p₁ + p₂ -1}.

In particular, $\rho$ = -1 if and only if p₁₀ = p₁ = 1 - p₂.

(iv) $\rho$ = 0 if and only if Y₁ and Y₂ are independent.

Proof. See the Appendix. The results from Lemma 1 inspire us to consider the following bivariate function

$$\begin{array}{l}p\left(y_1,y_2,p_1,p_2,\rho \right)={\text{ p}}_1^{y_1}{\left(1-p_1\right)}^{1-y_1}{p}_2^{y_2}{\left(1-p_2\right)}^{1-y_2}\text{+ }\\ {\left(-1\right)}^{y_1+y_2}\rho \sqrt{p_1\left(1-p_1\right)p_2\left(1-p_2\right),}(3)\end{array}$$

Where y₁, y₂ = 0, 1, p₁, p₂ and $\rho$ are defined as above. To make this bivariate function to become a probability distribution and use it statistical inference, we need to show the following theorem holds true.

Theorem 1

This bivariate function given in equation (3) is a well-defined and identifiable probability distribution.

The proof is given in the Appendix. Any two correlated categorical variables may be assumed to have the bivariate probability distribution with the parameters p₁, p₂ and $\rho$, which are the marginal probabilities of the variables and their correlation coefficient. Such distribution may be viewed as a generalization of the well-known univariate Bernoulli distribution to 2-dimension and we call it as a bivariate Bernoulli distribution denoted by (Y₁,Y₂)~BB (p₁, p₂, $\rho$).

Now we present the maximum likelihood estimators of the parameters p1, p2 and $\rho$. Let (y_i1, y_i2), i = 1, ..., n, be the sample from the bivariate Bernoulli distribution BB (p1, p2, $\rho$). The likelihood function is given by

$$\begin{array}{l}L={\displaystyle \prod }_{i=1}^{n}p\left(y_{1i},y_{2i}{p}_1,p_2,\rho \right)\text{ =}{\left(p\left(1,1,p_1,p_2,\rho \right)\right)}^{N_{11}}{\left(p\left(1,0,p_1,p_2,\rho \right)\right)}^{N_{10}}\\ {\left(p\left(0,1,p_1,p_2,\rho \right)\right)}^{N_{01}}{\left(p\left(0,0,p_1,p_2,\rho \right)\right)}^{N_{00}},(4)\end{array}$$

Where N₁₁, N₁₀, N₀₁ and N₀₀ are the numbers of subjects with true positive, false negative, false positive and true negative, respectively. The standard maximum likelihood methods often involve optimization of the log-likelihood function. The optimization is reached by taking the derivative of the log-likelihood function with respect to parameters. The Newton-Raphson iteration is frequently used to find the maximum likelihood estimators. However, this iteration finds an approximation to the maximum likelihood estimators. More importantly, it is difficult to show this iteration converges and lead to time-consuming computation. All these motivate us to find an explicit and closed-form expression for the maximum likelihood estimators. Now we use the following lemma to derive the maximum likelihood estimators.

LEMMA 2

Suppose that k is a positive integer and for t = 1, .., k, j_t is a nonnegative integer such that ${\displaystyle {\sum }_{t=1}^k{j}_t}<\text{ n}$. Then

$$\begin{array}{l}\stackrel{\max }{\left({\theta }_1,\mathrm{...},{\theta }_k\right)}\in {\Theta }_k\left({\displaystyle \prod }_{t=1}^k{\theta }_t^{j_t}\right){\left(1-{\displaystyle \sum }_{t=1}^k{\theta }_t\right)}^{n-{\displaystyle \sum }_{t=1}^k{j}_t}=\\ \left({\displaystyle \prod }_{t=1}^k{\left(\frac{j_t}{n}\right)}^{j_t}\right){\left(1-\frac{{\displaystyle \sum }_{t=1}^k{j}_t}{n}\right)}^{n-{\displaystyle \sum }_{t=1}^k{j}_t},(5)\end{array}$$

Where ${\Theta }_k=\left\{\left({\theta }_1,\mathrm{...},{\theta }_k\right):{\displaystyle {\sum }_{t=1}^k{\theta }_t<\text{ 1,0 < }{\theta }_t\text{ < 1,t =1,}\mathrm{..}\text{,k}}\right\}.$

The proof is given in the Appendix. From this lemma, we can derive the following theorem.

Theorem 2

The maximum likelihood estimators of the parameters are ${\stackrel{\wedge }{P}}_1=\frac{N_{11}+N_{10}}{n},{\stackrel{\wedge }{P}}_2=\frac{N_{11}+N_{01}}{n}$ and $\stackrel{\wedge }{\rho }=\frac{N_{11}{N}_{00}-N_{10}{N}_{01}}{\sqrt{\left(N_{11}+N_{10}\right)\left(N_{01}+N_{00}\right)\left(N_{11}+N_{01}\right)\left(N_{10}+N_{00}\right)}}$, where N₁₁, N₁₀, N₀₁ and N₀₀ are the numbers of subjects with true positive, false negative, false positive and true negative, respectively.

The proof is given in the Appendix.

Correlation Statistic

The repeated binary outcomes are often correlated and failure to consider their correlation in statistical measures may lead to poor assessments for association analysis. Now we see that correlation coefficient determines the strength and direction of association between two categorical variables.

Let Y₁ and Y₂ be the actual and predicted results for a subject, respectively. Assume that (Y₁, Y₂) has bivariate Bernoulli distribution $BB(p_1,p_2,\rho ),$ where p₁ = P (Y₁-1), p₂ = P (Y₂ = 1) and $\rho$ is the correlation coefficient. We discuss correlation coefficient in the following three cases. (i). A zero-correlation coefficient implies that P (Y₂ = 1|Y₁ = 1) = P (Y₂ = 1|Y₁ = 0) = P (Y₂ = 1), which suggests any actual result has no effect on a positive predicted result. Also, a zero-correlation coefficient implies that P (Y₂ = 0|Y₁ = 1) = P (Y₂ = 0|Y₁ = 0) = P (Y₂ = 0), which suggests any actual result has no effect on a negative predicted result. Thus, a zero-correlation coefficient implies that the two results are independent. (ii). A positive correlation coefficient implies that P (Y₂ = 1|Y₁ = 1) = P (Y₂ = 1|Y₁ = 1) > P (Y₂ = 1) and P (Y₂ = 0|Y₁ = 0) > P (Y₂ = 0), which suggests the actual result has an increased effect on the predicted result when the two results is the same. Also, a positive correlation coefficient implies that P (Y₂ = 1|Y₁ = 0) < P (Y₂ = 1) and P (Y₂ = 0|Y₁ = 1) < P (Y₂ = 0), which suggests the actual result has an decreased effect on the predicted result when the two results is different. Thus, a positive correlation coefficient implies that the two results are positively associated. (iii). A negative correlation coefficient implies that P (Y₂ = 1|Y₁ = 1) < P (Y₂ = 1) and P (Y₂ = 0|Y₁ = 0) < P (Y₂ = 0), which suggests the actual result has an decreased effect on the predicted result when the two results is the same. Also, a negative correlation coefficient implies that P (Y₂ = 1|Y₁ = 0) > P (Y₂ = 1) and P (Y₂ = 0|Y₁ = 1) > P (Y₂ = 0), which suggests the actual result has an increased effect on the predicted result when the two results is different. Thus, a negative correlation coefficient implies that the two results are negatively associated. Therefore, the correlation coefficient provides the direction of association between the actual and predicted results.

However, for a nonzero correlation coefficient, whether the correlation coefficient is positive or negative, its absolute value measures the strength of the association between the actual and predicted results. The greater the absolute value of the correlation coefficient is, the stronger the association between the actual and predicted results becomes. Thus, the correlation coefficient provides the strength and direction of association between categorical variables.

Since the correlation coefficient is unknown, we use its maximum likelihood estimator

$$CS=\frac{N_{11}{N}_{00}-N_{10}{N}_{01}}{\sqrt{\left(N_{11}+N_{10}\right)\left(N_{01}+N_{00}\right)\left(N_{11}+N_{01}\right)\left(N_{10}+N_{00}\right)}6}(6)$$

as a statistical measure for the performance of the diagnostic test and call it as correlation statistic. Such measure may have a potential to evaluate the association between correlated binary variables.

Compared with other measures such as likelihood ratio positive, likelihood ratio negative, average absolute likelihood ratio and Youden's index, our CS shows advantages for assessing the test performance. As we know, the CS is a statistical measure for estimating correlation between the actual result and predicted result from a general bivariate Bernoulli model. Therefore, the CS has potential to provide a direct evaluation of association between true and predicted results. However, other measures depend on sensitivity and specificity, which is an indirect evaluation of association between true and predicted results. These may lead to a conclusion that the CS would achieve better prediction of true result than other measures, which is shown from simulation studies in Section 4.

Simulation Studies

To evaluate the performance of the proposed ${\stackrel{\wedge }{P}}_1$, ${\stackrel{\wedge }{P}}_2$ and $\stackrel{\wedge }{P}$, we conduct a moderate scale simulation experiment. We generate n = 60, 120, 150 observations from the bivariate Bernoulli distribution:

$$\begin{array}{l}{P}_{y_1{y}_2}={\text{ P}}_1^{y_1}{\left(1-p_1\right)}^{1-y_1}{\text{P}}_2^{y_2}{\left(1-p_2\right)}^{1-y_2}+\\ {\left(-1\right)}^{y_1+y_2}\rho \sqrt{{\text{P}}_1\left(1-p_1\right)P_2\left(1-p_2\right)},{\text{ y}}_1,y_2=\text{ 0,1}\text{.}\end{array}$$

We consider the following four scenarios: (a) p₁ = 0.2, p₂ = 0.4, $\rho$ = 0.3. (b) p₁ = 0.6, p₂ = 0.8, $\rho$ = 0.5. (c) p₁ = 0.2, p₂ = 0.4, $\rho$ = -0.3. (d) p₁ = 0.6, p₂ = 0.8, $\rho$ = -0.2. We generate 1000 data sets in each of the configurations. The simulation results on the simulated means, standard errors and mean squared errors are summarized in Table 1.

To compare the performance between our proposed statistical measure and the likelihood ratio, we conduct a moderate scale simulation experiment. We generate n = 300 in (a) and 600 in (b) observations from the bivariate normal distribution:

$\text{f}\left(y_1,y_2\right)=\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-r^2}}\exp \left\{-\frac{1}{2\left(1-r^2\right)}\left(\frac{{\left(y_1-{\mu }_1\right)}^2}{{\sigma }_1^2}-\frac{2r\left(y_1-{\mu }_1\right)\left(y_2-{\mu }_2\right)}{{\sigma }_1{\sigma }_2}+\frac{{\left(y_2-{\mu }_2\right)}^2}{{\sigma }_2^2}\right)\right\}$

We consider the following two scenarios: (a) µ₁ = 8, µ₂ = 8, σ₁ = 3, σ₁ = 3, r = 0.8. (b) µ₁ = 30, µ₂ = 30, σ₁ = 6, σ₁ = 6, r = 09. For each of (a) and (b), we classify Y1 and Y2 into abnormal or normal results based on the threshold, that is, Y1 (or Y2) is abnormal if it is less than or equal to the threshold and otherwise, it is normal. Also, we treat Y1 as the predictor and treat Y2 as actual status. We generate 1000 data sets in each of the configurations. For each simulation run, the thresholds for Y1are the quantiles which correspond to the probabilities 10%, 20%, ..., 90%, respectively while the thresholds for Y2 are the quantiles which correspond to the probabilities 30% and 60%, respectively. We use our proposed method to identify the appropriate threshold which leads a good predictor. The simulation results on the sensitivity and specificity (SENS and SPEC), likelihood ratio positive and likelihood ratio negative (LRP and LRN), average absolute likelihood ratio (AALR), Youden's index (YOUDEN), correlation statistic (CS) and assigned threshold values (THRES 1 and THRES 2) are summarized in Table 2. In the table, an entry is the average over 1000 simulations for the corresponding measure. For the CS, YOUDEN and LRP, the greater the measure becomes, the better the test is. But for the measures LRN and AALR, the smaller the measure becomes, the better the test is. From Table 2, we can see that in all the simulated cases, the CS is able to identify the correct threshold. Also, the CS may draw an acceptable balance between sensitivity and specificity. The greatest CS approximately corresponds to the greatest sensitivity (or specificity) given that specificity (or sensitivity) is greater than or equal to around 0.8. The well-known YOUDEN which is defined as SENS + SPEC-1, is often used to identify the threshold. However, it does not perform well to make prediction especially for the cases (Scenario (a), n (300), THRES 1 (8.76)) and (Scenario (a), n (600), THRES 1 (8.76)). It should be noted that the greatest YOUDEN does not achieve a good balance between sensitivity and specificity. For example, for the case (Scenario (a), n (300), THRES 1 (8.76)), the greatest YOUDEN approximately corresponds to a relatively low specificity (0.663). If we use likelihood ratio positive or likelihood ratio negative to identify the threshold individually, it misses the correct value in each case. Also, the greatest LRP or smallest LRN does not achieve an acceptable balance between sensitivity and specificity in many cases. It still remains challenge to compromise between the likelihood ratio positive and the likelihood ratio negative for balance between sensitivity and specificity. One disadvantage of the likelihood ratio positive is that its value may be infinity in some extreme cases. Although the smallest AALR corresponds to the greatest CS, surprisingly, the average absolute likelihood ratio, which is conceptually an average of likelihood ratios over a population/sample, is also able to identify the correct threshold, if we are willing to use it in a counterintuitive way: The smaller the measure, the better the test.

Application to Gastric Emptying Data

Gastric emptying study is the standard of measuring gastric emptying and the aim of this study is to establish the predictors and to assess the possibility of shortening the study time [2,19,20,].

In our data analysis, we consider the gastric emptying at 2 hours as predictor and the gastric emptying at 4 hours as actual status. Although the gastric emptying results at 2 hours and at 4 hours are count variables, we classify them based on the thresholds. The gastric emptying results at 4 hours is defined as abnormal if the percentage of remaining radioactivity is greater than 10% and otherwise it is defined as normal. For the gastric emptying results at 2 hours, the possible thresholds are 30%, 40%, ..., 80%. The gastric emptying results at 2 hours are defined as abnormal if it is greater than a threshold and otherwise it is defined as normal. We use our proposed method to identify the threshold of 50% to classify the gastric emptying results at 2 hours. The results about sensitivity and specificity shows the threshold of 50% is feasible. Some simulation results are summarized in Table 3 where THRES is the threshold value. The area under the ROC curve (AUC) is the average sensitivity of the biomarker over the range of specificities, which is often used as an evaluation of the overall performance of the biomarker. Our data show the estimated AUC is 0.816, which provides evidence that our correlation statistic is useful for correctly classifying subjects with predicted positive and subjects with predicted negative. From the table and resulting AUC value above, we conclude the threshold of 50% is best choice. As we have known, the sensitivity and specificity are true positive rate and are true negative rate, respectively. The positive likelihood ratio considers only predicted positive and ignore predicted negative while the negative likelihood ratio considers only predicted negative and ignore predicted positive. However, our correlation statistic provides comprehensive and powerful measures and has a potential to correctly and reliably assess a diagnostic test. It can be used as alternative to average absolute likelihood ratio and Youden's index when they are not applicable. Also, our correlation statistic could provide the direction of association between two categorical variables.

Discussion

Due to the importance of assessment of test in medicine and clinical trials, many researchers have paid attention to statistical measures. The likelihood ratio positive and the likelihood ratio negative are the most commonly used measures of diagnostic test. However, the likelihood ratio positive and the likelihood ratio negative often present the reverse tendencies and to determine the compromise between the likelihood ratio positive and the likelihood ratio negative relies on the investigators 'sound knowledge and experience in medicine and clinical trials. In this article, we proposed a statistical measure. Compared with the likelihood ratio positive and the likelihood ratio negative, our measure could have a unique solution and produce an acceptable balance between sensitivity and specificity.

We should also point out that the bivariate Bernoulli distribution for correlated binary variables focuses on construction of statistical measure for test performance, but it is important to explore the applicability of our proposed bivariate Bernoulli distribution in bivariate interobsrver agreement since the correlation coefficient can be used to indicate the direction and strength of the difference between the chance of two uncorrelated individuals with both true positive and negative rates and the chance of the two individuals with false positive and negative rates. It can be used as alternative to average absolute likelihood ratio (AALR) and Youden's index when they are not applicable. In addition, we can explore the applicability of our proposed bivariate Bernoulli distribution in the other literature such as logistic regression for two binary outcomes, repeated measurements, etc., and extend the bivariate distribution to higher-dimension.

Acknowledgements

Dr. Rai is grateful to generous support from Dr. DM Miller, Director James Graham

Brown Cancer Center and Wendell Cherry Chair in Clinical Trial Research.

Conflict of Interest

The authors have declared no conflict of interest.

References

1. Bonta D, Lee H, Ziessman H (2011) Shortening the 4-hour gastric-emptying protocol. Clinical Nuclear Medicine 36: 283-285.
2. Bhatt G, Li XF, Wu XY, Rai SN, Civelek AC (2014) Factors affecting 4-Hour Solid Gastric Emptying Study - Evaluating the potential to reduce the study time. Report.
3. Banoo S, Bell D, Bossuyt P, Herring A, Mabey D, et al. (2006) Evaluation of diagnostic tests for infectious diseases: General principles. Nature Reviews Microbiology 4: S21-S31.
4. Rudwaleit M, Sieper J (2007) A case of axial undifferentiated spondyloarthritis diagnosis and management. Nature Clinical Practice Rheumatology 3: 298-303.
5. deFranchis R (2009) Cirrhosis: Screening for esophageal varices. Nature Reviews Gastroenterology & Hepatology 6: 449-450.
6. Peeling RW, Artsob H, Pelegrino JL, Buchy P, Cardosa MJ, et al. (2010) Evaluation of diagnostic tests: dengue. Nature Reviews Microbiology 8: S30-S38.
7. Luts J, Nofuentes JAR, Luna JDL, Van S (2011) Asymptotic hypothesis test to compare likelihood ratios of multiple diagnostic tests in unpaired designs. Journal of Statistical.
8. Fink HA, Lederle FA, Roth CS, Bowles CA, Nelson DB, et al. (2000) The accuracy of physical examination to detect abdominal aortic aneurysm. Archives of Internal Medicine 160: 833-836.
9. Hahne AJ, Keating JL, Wilson SC (2004) Do within-session changes in pain intensity and range of motion predict between-session changes in patients with low back pain? Australian Journal of Physiotherapy 50: 17-23.
10. Green DM, Cox CL, Zhu L, Krull KR, Srivastava DK, et al. (2012) Risk factors for obesity in adult survivors of childhood cancer: A report from the Childhood Cancer Survivor Study. Journal of Clinical Oncology 30: 246-255.
11. Langlotz CP (2003) Fundamental measures of diagnostic examination performance: usefulness for clinical decision making and research. Radiology 228: 3-9.
12. Grimes DA, Schulz KF (2005) Refining clinical diagnosis with likelihood ratios. Lancet 365: 1500-1505.
13. Liang H (2006) Checking linearity of non-parametric component in partially linear models with an application in systemic inflammatory response syndrome study. Statistical Methods in Medical Research 15: 273-284.
14. Koul HL, Song W (2010) Model checking in partial linear regression models with Berkson measurement errors. Statistica Sinica 20: 1551-1579.
15. McGee SR (2002) Simplifying likelihood ratios. Journal of General Internal Medicine 17: 647-650.
16. Marill KA (2012) Diagnostic testing and the average absolute likelihood ratio: application to diagnosing wide QRS complex tachycardia and other ED diseases. American Journal of Emergency Medicine 30: 1895-1906.
17. Ruopp MD, Perkins NJ, Whitcomb BW, SchistermanEF (2008) Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection. Biometrical Journal 50: 419-430.
18. Marshall AW, Olkin I (1985) A family of bivariate distributions generated by Bernoulli distribution. Journal of the American Statistical Association 80: 332-338.
19. Tougas G, Eaker EY, Abell TL, Abrahamsson H, Boivin M, et al. (2000) Assessment of gastric emptying using a low fat meal: Establishment of international control values. The American Journal of Gastroenterology 95: 1456-1462.
20. Abell TL, Camilleri M, Donohoe K, Hasler WL, Lin HC, et al. (2008) Consensus recommendations for gastric emptying scintigraphy: A joint report of the American Neurogastroenterology and Motility Society and the Society of Nuclear Medicine. Journal of Nuclear Medicine Technology 103: 753-763.

---