An expert panel was assembled to support a review of a series of recent publications using a modified Delphi format. These publications were scored based on a consideration of confidence in their methods, results, conclusions, and applicability to risk-based decision making. Mean confidence scores for the papers reviewed ranged from 53 to 74 (maximum score = 100), and key strengths and concerns were identified. This review highlights the need for transparency in meta-analyses. Different conclusions were reached in available meta-analyses because of varying criteria used to select studies, selection of different risk estimates within the same study, and study availability. Confidence in potential a causal relationship between glyphosate exposure and NHL was considered low.
NHL: Non-Hodgkin's Lymphoma; CV: Coefficient of Variation; CI: Confidence Interval; RR: Risk Ratio; AHS: Agricultural Health Study; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; mRR: meta-Risk Ratio; mSMR: meta-Standard Mortality Ratio; mHR: meta-Hazard Ratio
Meta-analysis provides a quantitative, formal approach to systematically assess available research to support weight of evidence conclusions about that body of research [1]. As such, meta-analysis serves as a useful tool for supporting decision-making. Conclusions from meta-analyses can be used by decision makers to support medical, public health, regulatory, and legal decisions related to the topic area. However, when multiple meta-analyses are available that report differing conclusions for a given topic area, decision makers can be faced with uncertainty which complicates decision making, particularly when the underlying reasons for the different results are unclear [2]. Potential reasons for different conclusions across meta-analyses include the use of different inclusion/exclusion criteria, differing assessments of study quality and potential biases, selection of different measures of association between studies, and potential vested interests in the findings.
Studies to be included in a meta-analysis should be based on a consideration of the methodological quality of each study, as assessed in terms of its potential for selection bias, information bias/exposure misclassification, confounding, reporting bias, and other issues affecting validity. In addition, other potential sources of bias should be evaluated based on subject identification strategy, participation rates, investigator blinding, assessment methods for exposures, outcomes, and potential confounders, statistical approach, reporting of results, and other considerations.
Glyphosate is an organophosphorus herbicide that has been used worldwide [3]. Controversy surrounding glyphosate precipitates from the decision in 2015 by the World Health Organization's International Agency for Research on Cancer (IARC) to classify glyphosate as "probably carcinogenic in humans" [4]. This classification decision stands in contrast to those decided by the U.S. Environmental Protection Agency [5], who concluded that "glyphosate is not likely to be carcinogenic to humans", as well as those made by the European Food Safety Authority [6], the Canadian Pest Management Regulatory Agency [7], and the European Chemical Agency [8].
A series of recent meta- and pooled analyses papers on glyphosate and non-Hodgkin's lymphoma (NHL) were identified in the published literature, and were subject to this review. A summary of these studies and their underlying case-control data sources is provided in Table 1. All of the meta-analyses subject to this review incorporated risk estimates from both case-control and cohort studies as part of their input [9-14], while the pooled analysis of Leon, et al. [15] relied solely on cohort data. From this comparison table, it is clear that some publications have reported statistically significant increases in NHL, while others have not. For decision makers, the statistical significance and the strength of the association can be important factors to consider as part of their deliberations. Regardless of statistical testing, three of the meta-risk estimates were near null. The goal of this review is to provide understanding in the reasons for these differing results, which may include differences in methodology, data set inclusion/exclusion, and/or other decisions made in the process of conducting the analyses. An independent expert panel was assembled to provide some insight into these important questions so that regulators and policy makers can make decisions with a clearer understanding of available meta- and pooled analyses for glyphosate. Although it is recognized that all of these papers have undergone peer review as part of the process for publishing, the primary focus of that review process serves to address the basic question from the journal, "Is this study worthy of publication?". In contrast, the focus of this review process addresses a different but important question, "Should this study be used to support risk assessment-based decision making?". The following text seeks to: (1) Summarize the methods for assembling and engaging an expert panel, and (2) Summarize the confidence scores and opinions submitted by the panel. The focus of this peer review was on the association between glyphosate exposures and NHL. Some of the studies reviewed considered exposure to other pesticides, other cancer endpoints, as well as subtypes of NHL.
Table 1: Summary of meta- and pooled analysis publications and underlying case-control and cohort studies for Glyphosate and NHL. View Table 1
Scientists with expertise in epidemiology and/or conducting meta-analyses were identified from a variety of sources including: (1) SciPinion's internal database of users; (2) Searches for authors of recent publications on the topic of interest in online databases (e.g., Pubmed; Google Scholar); (3) Searches of profiles on social media databases (e.g., LinkedIn); (4) General internet searches; and (5) Referrals from other scientists. Candidates were invited via email to apply to this expert panel opportunity via a web app (https://app.scipinion.com). Six experts were selected from the available applicants based upon a consideration of objective expertise metrics (e.g., number of publications, years of experience, key word counts in CV). The six experts selected for this panel originate from five different countries (Australia, Belgium, Canada, Italy, and United States). The combined expertise of the panel includes 8 advanced degrees (5 PhD, 1 MD, 1 MBA, 1 DrPH), approximately 148 years of post-degree experience, and more than 1,500 publications. In addition, all panelists have served in an editorial board capacity for one or more scientific journals.
To minimize potential participation and selection bias, as well as for potential groupthink, the panelists were blinded to the review sponsor, and to each other during the course of the review. All participation in this review was performed online via a web app (https://app.scipinion.com). The review was structured using a modified Delphi format [16] that consisted of three rounds of participation:
• Round 1 - During Round 1 the panelists worked independently. The panel was tasked with reviewing the key papers (Table 1) and answer charge questions. The charge questions included those used to score each study (Appendix A). To minimize potential scope bias, the panelists were also asked to submit a charge question of their own for their fellow panel members to consider. Round 1 was held from 6/11/21 through 7/9/21. All six panel members participated in this round as scheduled.
• Round 2 - During Round 2 the panelists were permitted to interact anonymously (e.g., as "Expert 1", "Expert 2", ..."Expert 6"; with numbers assigned randomly to each panelist). The panel was tasked with reviewing each other's answers to Round 1 charge questions (provided as a downloadable pdf report and also via online access). They were given the opportunity to interact with one another by submitting comments on each other's answers, and rating (thumbs up or down) each other's comments during the round. Round 2 was held from 7/12/21 through 7/21/21. All six panelists participated in the debate as scheduled. A total of 45 comments and 19 comment ratings were submitted during this round (Appendix A).
• Round 3 - During Round 3 the panel was tasked with revising their answers to Round 1 charge questions as needed, in case Round 2 participation resulted in change of their opinion. Round 3 was held from approximately 7/22/21 through approximately 7/30/21. Several panelists identified an update to one of the meta-analyses reviewed (Boffetta, et al. [13]; an update of the Donato, et al. [12]), which was published during the Round 1 participation window). This updated analysis was also reviewed and scored as part of Round 3. All 6 panelists participated in this round as scheduled.
The expert panel was tasked with providing confidence ratings for each of the key studies. Confidence was rated on a scale of 1 (lowest confidence) to 10 (highest confidence) in each of four areas: (1) Study methods; (2) Study results; (3) Study conclusions/discussion; (4) Application to risk assessment decision making. An overall study score was calculated from each panelist as the mean of the four ratings multiplied by a factor of 10 (maximum score = 100). For each publication, the mean score across all six panelists was calculated. For the publication of Donato, et al. [12] and its recent update (Boffetta, et al. [13]), the scores for the two publications were averaged together (separate scores for each are also provided in Appendix A). All score calculations and statistics [means, standard deviations (SD), coefficients of variation (CV) were performed in Microsoft Excel (version 16.36).
A summary of the study scores returned by the panel is summarized in Figure 1. Mean study scores ranged from 53 to 74 (maximum score = 100). Within each publication, a moderate degree of variation is noted in the scores between panelists. Mean confidence scores fell within the high-confidence range (upper tertile or > 67) for two studies [10,15], while mean scores for all others fell within the medium-confidence range (middle tertile or 33-67). Variation in confidence scores was lowest for the same two studies [10,15] (CV = 18-19%), when compared to the other studies (CV = 25-35%). None of the mean study scores fell within the low confidence range (lowest tertile or < 33). The reader is referred to Appendix A for the detailed insights and rationales from the panelists that underly these study confidence scores. A summary of key points and insights noted for each publication is provided below (chronologically in order of publication date).
Figure 1: Expert panel ratings and confidence scores in glyphosate meta- analyses.
View Figure 1
Schinasi and Leon [9] provided a systematic review and a series of meta-analyses of available epidemiologic research on the relationship between NHL and occupational exposure to 21 pesticide chemical groups and 80 active ingredients. For glyphosate exposure (n = 6 studies), the authors reported a statistically significant meta risk ratio of 1.5 (1.1-2.0).The analysis found a low level of heterogeneity (I2 = 32.7%).
The panelists noted that the search strategy was clearly articulated, relying upon multiple databases. Although some panelists would prefer the number of databases and languages considered to be expanded, the studies identified are consistent with other meta-analyses conducted. The panelists appreciated that a variety of exposure measurements were considered, although the final decision was made to use ever-exposed since it provided the greatest number of cases. Also appreciated were the extensive sensitivity analyses conducted, the absence of apparent conflicts of interest by the study authors. The most significant concern noted for the study relates to the decisions of the study authors to rely on the higher risk estimates from univariate analyses of the underlying studies rather than the lower risk estimates from the multivariate analyses. The authors did not follow the PRISMA Guidelines [17], did not undertake a quality assessment of the studies, or assess publication bias. Due to the authors decision to use dichotomous exposure reported in the studies, several aspects of exposure, such as exposure lags and duration of exposure were ignored.
Chang and Delzell [10] conducted a systematic review and meta-analysis for glyphosate exposure and risk of NHL and other lymphohematopoietic cancers. A meta-relative risk for NHL was reported to be 1.3 (95% CI: 1.0-1.6) based on seven studies. The authors noted that bias and confounding may account for observed associations. There was no heterogeneity found in the pooled analysis (I2 = 0.00%). The authors concluded that a causal relationship has not been established between glyphosate exposure and risk of any type of lymphohemotopoietic cancers including NHL.
The panelists appreciated that multiple electronic databases were searched, and that two authors independently reviewed and selected the studies for inclusion. Although the authors relied upon essentially the same studies as Schinasi and Leon [9], their consistent use of the adjusted risk estimates from multivariate analysis represents an important difference, and was commended. A quality assessment of underlying studies, including methodological qualities, was conducted using Austin Bradford Hill's viewpoints [18]. The panel also appreciated the authors inclusion of both fixed- and random-effects models, conducting multiple sensitivity analyses, as well as considerations for potential selection bias, information bias, exposure misclassification, confounding reporting bias, as well as other issues that could impact the validity of their results. The panel felt that the authors appropriately acknowledged many of the uncertainties and limitations of the underlying data. The panel also noted a few limitations for this study, including that the authors did not follow the PRISMA Guidelines [17]. A self- acknowledged limitation was recognized in that the authors could not combine multiple studies using three or more exposure categories as these differed across studies, which limits the ability to look at exposure response relationships. There was no assessment of publication bias. Lastly, it was noted that the study was funded by Monsanto (a manufacturer of glyphosate) and therefore raises some potential for conflict of interest.
Leon, et al. [15] conducted a pooled analyses of three large agricultural worker cohorts that covered multiple pesticides and active ingredients. The panelists noted that this study differed from the others in the sense that it produced new estimates by pooling the raw data, and did not combine the original risk estimates from the individual studies. Cox regression models were used to estimate cohort-specific hazard ratios. The authors reported that associations of pesticides with NHL appear to be subtype- and chemical-specific. For NHL and glyphosate exposure, the authors reported a hazard ratio from the pooled data of 0.95 (95% CI: 0.77-1.18). There was moderate heterogeneity found in the pooled analysis (I2 = 57.0%).
The panel considered the cohort design of this study to be a strength and is better capable of overcoming recall bias, and participation bias that can impact measures of associations derived from case-control studies. The pooled analysis of cohorts can have an advantage over meta-analysis since it can account for both within- and between-cohort confounding. The panel appreciated that the analytical methods were state-of-the-art and harmonization of exposures through crop-exposure-matrix is likely the best approach to the problem. Missing data were handled using multiple imputations, an entirely appropriate method if certain assumptions are met (uncertain). Because the of size of the European cohorts, the AHS cohort did not dominate the pooled sample, which potentially makes this analysis a useful addition over earlier meta-analyses with risk estimated weighted by the AHS. The authors report no evidence of association of having ever used glyphosate with risk of NHL and the effect estimate supporting this is far more precise than that seen from meta-analyses that predated the pooled cohort or excluded it. Several limitations of this study were noted by the panel. The estimation of exposure was based on crop exposure matrices, which could introduce exposure misclassification. No additional analyses were considered to approximate more intense, longer duration, or higher exposures. The authors did not describe the extent of missing data. The panel noted substantial heterogeneity in the measures of association between the three cohorts, and that the three cohorts differed in variables, such as age and gender. The panel would have appreciated the inclusion of cohort specific risks estimates, and/or modeling random cohort effects and checking cohort-exposure interactions. Additional analyses, such as hierarchical regression, assessment of latent confounding, estimation of absolute risks and attributable fractions, would have been appreciated.
Zhang, et al. [11] investigated whether there was an association between high cumulative exposures to glyphosate and increased risk of NHL in humans. Using the highest exposure groups when available in each study, the authors report statistically significant meta-relative risk in glyphosate-exposed individuals of 1.41 (1.13-1.75). The authors concluded that these results suggest a compelling link between exposures to glyphosate-based herbicides and increased risk for NHL. There was moderate heterogeneity found in the pooled analysis (I2 = 52.8%).
The panel noted that this study followed the PRISMA guidelines [17], had no language restrictions, and reviewed the quality of the studies using an accepted ratings scale (Newcastle-Ottawa) [19]. The panel appreciated that both fixed-effects and random-effects models were used in conducting the author's analysis, and that potential publication bias was assessed. The panel was complimentary of the summary tables provided for methods and results. Inclusion of the updated findings from AHS cohort is a strength for this paper. The sensitivity analyses conducted were considered to be extensive. The most significant concern noted by the panel pertained to the authors' use of very different measures of exposure across the studies to derive summary risk measures, which may make the results uninterpretable. The methodology used in this paper stands in contrast to Schinasi and Leon [9] who state that "it was not possible to combine estimates based on multiple categories of exposure in formal meta-analyses". Additional concerns were noted for the use of a single database for the literature search (but did include additional secondary references). The panel also indicated that the authors may have overstated their conclusions in claiming the link between glyphosate and NHL was compelling, particularly since this finding is in contrast to some of the underlying studies used in the analysis (e.g., AHS). The authors focus on a fixed-effects model rather than a random-effects drew some criticism from the panel. The panel would have appreciated the inclusion of additional discussion on issues such as that more distal exposures (historical measures) tend to be measured with greater error, and consideration of participation bias, as well as consideration of the impact of the scale score on summary risk measures.
Donato, et al. [12] and Boffetta, et al. [13] carried out a systematic review and meta-analysis of epidemiologic studies on the association between occupational exposure to glyphosate and risk of NHL. In the initial study, the authors performed random-effects meta-analyses for ever-exposure to glyphosate, dose-response, and risk of specific NHL subtypes. The authors meta-relative risk for NHL or 1.03 (0.86-1.21, and a meta-RR for highest category of exposure was 1.49 (0.37-2.61). There was no heterogeneity found in the pooled analysis (I2 = 0.00%). The authors concluded that their meta-analysis provided no overall evidence of an increased risk for NHL and occupational exposure to glyphosate. In the updated analysis by Boffetta, et al. [13], which included the updated pooled data of Pahwa, et al. [20] in place of the separate data sets of De Roos, et al. [21] and McDuffie, et al. [22], the meta-RR for ever-exposure to glyphosate was 1.05 (0.90-1.24), and that for the highest category of exposure was 1.15 (0.72-1.83). The authors concluded that the updated analysis reinforced their previous conclusion of a lack of an association between exposure to glyphosate and risk of NHL.
The panel appreciated that this study followed PRISMA guidelines14, utilized multiple publication databases and languages, and relied on two authors independently assessing which studies be included. Use of a random effects model was considered a reasonable default for analysis, however inclusion of a fixed model for comparison purposes would be helpful in demonstrating that the decision was not consequential to the results. Methods for assessing bias and sensitivity analysis were considered reasonable and appropriate. Limitations of this study identified by the panel include the following. The inclusion of the pooled analysis of Leon, et al. [15] dominated the results of this meta-analysis, and unfortunately study-specific estimates from the underlying cohorts were not used to examine heterogeneity. Assessment of study quality was considered inadequate. The authors report evidence of publication bias, but did not include quantitative bias and uncertainty analysis. The analysis by levels of exposure suffers from lack of harmonization of exposure categories. The study mixes case-control and cohort studies, with the positive findings from some case-control studies discounted (potential bias) while accepting potential misclassification of exposure through the use of crop exposure matrices in the Norwegian and French cohorts. The panel noted that the authors conclusion "provide evidence that the risk of NHL is not increased in workers exposed to glyphosate" may be overstated given the many uncertainties. The refinements incorporated in Boffetta, et al. [13] in response to criticism [23] were relatively minor, and some panelists questioned whether it warranted a new publication. Panelists also expressed concern over using the pooled data set of Pahwa, et al. [20], in addition to the pooled data set of Leon, et al. [15], which ignores the differing characteristics of the individual studies [21,24], thereby limiting the ability to assess publication bias. The authors of this study may have overstated their conclusions given the limitations of the underlying data and their analysis. It was noted that one of authors of this meta-analysis had previously acted as a consultant to a glyphosate producer.
Kabat, et al. [14] conducted sensitivity analyses to determine how the definition of exposure and the choice of latency period affect the summary risk estimates of Zhang, et al. [11] based on a subset of the studies (five of seven). The authors concluded that summary estimates of risk varied considerably depending on the exposure level and latency assumed. For example, using the highest reported exposure levels, summary risk estimates for glyphosate and NHL were highest using a 20-year lag [RR = 1.41 (95% CI 1.13-1.76)], while on the other hand, for ever-exposure with no lag period, the summary relative risk with updated estimates was considerably lower (1.05, 95% CI 0.87-1.28). There was no heterogeneity found in the pooled analysis (I2 = 0.00%).
The panel appreciated that authors set out to illustrate how study author choices (in Zhang, et al. [11] for the selection of effect estimates and exposure metrics can influence meta-analysis results. The methods and discussion of results were generally considered to be appropriate and reasonable. Limitations in this study noted by panel members include a consideration of the following. It was unfortunate that the updated cohort data of Leon, et al. [15] were not considered. Although the authors cautioned against pooling the cohort study with case-control studies, an alternative approach was not provided for how to combine this evidence in a quantitative manner. It was noted that this study ignored two of seven original studies included in (Zhang, et al. [11], since studies that used crop-exposure matrices were excluded. This exclusion decision by Kabat, et al. [14] was criticized by some panelists. Panel members noted that the inclusion of pooled analyses (rather than the individual studies) compromises the ability to assess heterogeneity, particularly when there are relatively few studies to analyze.
After reviewing the available meta- and pooled analyses for glyphosate, panelists were asked their opinions on the degree of confidence in a causal relationship between glyphosate exposure and NHL (on a 1-10 scale). With respect to conclusions on a causal relationship between glyphosate exposure and NHL, the panel was generally consistent in concluding low confidence (mean = 3, range = 1-5; Figure 2). However, this low confidence does not preclude a different conclusion on specific NHL subtypes, which at present has only been assessed in a very limited subset of the available studies [10,15]. The reader is referred to Appendix A for the insights and rationales that underly these scores. The relatively low confidence score from the panel is generally consistent with the conclusions of regulatory agencies [5-8] on the carcinogenicity of glyphosate.
Figure 2: Expert confidence in causal relationship between glyphosate exposure and non-Hodgkin's lymphoma.
View Figure 2
The results of this critical review highlight the importance of transparency when conducting a meta-analysis. It is expected that as new studies become available, and as studies are updated, the results of meta-analyses can change overtime. In some cases, it is less clear what impact author decisions (e.g., the selection of adjusted vs. unadjusted risk estimates; selection of alternative exposure metrics) can have on the meta-analyses results and their conclusions.
It is well recognized that case-control studies and cohort studies have specific limitations. Case-control can suffer from recall bias (which can result in exposure misclassification) and selection bias, as has been reported for glyphosate studies [25]. Cohort studies, on the other hand, can suffer from exposure misclassification and other participation and follow-up issues. However, both types of studies have their utility, and can provide evidence to support decision making. This expert panel review also highlights the need for improved methods for quantifying evidence from case-control studies and evidence from cohort studies in a single quantitative framework. The panel also considers the available data for NHL and glyphosate to be limited to a handful of studies, and the confidence in any meta-analysis cannot exceed the confidence in the underlying studies on which they are based.
Pooled analyses of cohort studies [15] and case-control studies [20] can be useful, and can increase the statistical power of the analyses. However, if results for individual data sets are also provided heterogeneity can be assessed. Moreover, each of the studies may have their own strengths, limitations, and biases, and these can get ignored in a pooled analysis.
The database of available case-control and cohort studies available for glyphosate exposure and NHL remains relatively limited, but is expected to improve as cohort studies are updated to include longer follow-up, and additional case-control studies are con-ducted. Despite the limited database to work with, multiple meta-analyses have been published on this topic, in some cases with differing results and conclusions. Evidence of weak potential increase in some case-control studies do not appear to be confirmed by other case-control studies. Despite differences in statistical significance, the panelists felt that collectively the results from the case-control studies were not appreciably different, in that they rule out the possibility of a strong association. Available cohort studies, in particular the AHS, show a lack of an association. Although some refinements could be made to these analyses, there appears to be general consensus within the expert panel that additional meta-analyses are not needed at this time for glyphosate. Instead, efforts would be better placed on supplementing the existing databases by conducting new high-quality cohort and case control studies to provide more definitive answers around competing hypotheses.
Resulting from an independent peer review process of proposals submitted, SciPinion LLC was awarded a grant from Truth in Science (grant number 0421-01) for conducting an in-dependent review of publications on glyphosate and cancer, which was specifically applied to conduct an expert panel review of recent meta- and pooled analyses publications on glyphosate and NHL. The authors would like to thank Drs. Carol Burns, Igor Burstyn, and Maurice Zeegers for participating in the expert panel, along with co-authors CP, GDE, and PJV. The authors, CRK and SMH, are owners of SciPinion and thus have a financial interest in the content of this manuscript. All other authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. During the review, the funder was blinded to the identity of the expert panel members during the review, and the expert panel members were blinded to the source of funding and to each other.