Int Arch Nurs Health Care ISSN: 2469-5823

• Abstract
• Keywords
• Abbreviations
• Introduction
• Methods
• Results
• Discussion
• Strengths and Limitations
• Conclusion
• Ethics
• Conflict of Interest
• Funding
• Acknowledgement
• Figure 1
• Table 1
• Table 2
• Supplemental File
• References

Download PDF
    

International Archives of Nursing and Health Care

¹Department of general pediatrics, Erasmus MC-Sophia Children's Hospital, Rotterdam, The Netherlands
²Department of Public Health, Erasmus Medical Centre, University Medical Centre Rotterdam, The Netherlands

Abstract

Background: For successful practical implementation of clinical decision rules, measuring their acceptability is essential. The Ottawa Acceptability of Decision Rules Instrument (OADRI) was developed to serve this need.

Objective: To make OADRI available in the Netherlands by translating the instrument into Dutch, to use this instrument to identify nurses' acceptability of two clinical decision rules in pediatric emergency care and to perform applicable validation analyses.

Design, setting and participants: Prospective questionnaire evaluation among pediatric emergency department (ED) nurses in the period September 2010-October 2012.

Methods: The OADRI was translated into Dutch according to international guidelines. Decision rule acceptability was measured during three different moments of two impact trials. We described OADRI mean item scores and diversity of answer possibilities, where > 50% "No opinion/Don't know" where defined as non-informative OADRI items.

Results: Implementation activities of the clinical decision rules were positively associated with acceptability scores, especially for the OADRI items of the subscales "aspects of innovation" and "environment". Validation activities showed ceiling effects for each OADRI item, indicating limited content validity of the instrument. The decision rules appeared to be clear, with only 2 out of 12 items being considered less informative.

Conclusions and relevance: The Dutch translated OADRI showed well dissemination in a non-English country to measure acceptability of clinical decision rules. Responses to individual OADRI items guided possible improvement of decision rule implementation. However, limitations in our validity analyses necessitate more broad-spectrum validation before OADRI implementation.

Keywords

Children, Fever, Gastroenteritis, Clinical Decision Rule, Implementation, Emergency Department, Serious Bacterial Infections, Nursing

Abbreviations

ED: Emergency Department; SBI: Serious bacterial infection; CRP: C-reactive protein; AUC: area under the receiver operating characteristic curve; CI: Confidence Interval

Introduction

Application of clinical decision rules may improve efficiency and quality of care by recommendations on, for example, performing additional diagnostic tests or initiating treatment [1,2]. They have been developed to aid physicians and nursing staff to make a diagnosis or predict an outcome [1,3,4]. After external validation, the next methodological step is to perform an impact analysis to test whether or not the decision rule actually improves clinical decisions and will benefit patient care or reduce costs [1,5,6]. During this last implementation phase compliance to rule advices is mostly one of the main outcome measures [7]. However, measuring its acceptability is essential for successful implementation.

The Acceptability of Decision Rules Instrument (OADRI), aimed to measure the acceptability of clinical decision rules, was developed by a research group who was informed by the Ottawa Model of Research Use [8-10]. They grouped various barriers and facilitators related to whether a research innovation was adopted into practice into three large categories: 1) aspects of the innovation; 2) decision making, and 3) environment. After pilot testing in a single tertiary care hospital, the 12-item instrument OADRI was validated via postal surveys amongst emergency physicians from Australia, Canada, United Kingdom and United States in the context of the Canadian C-Spine Rule and the Canadian CT head rule [11,12].

In previous research we performed two impact trials of clinical decision rules [13,14]. The first rule was based on a diagnostic decision model for children with fever at risk for serious bacterial infections (SBI) [13]. The second rule was a therapeutic decision model for children with vomiting and diarrhea, who were at risk for dehydration [14]. In the present study we used the OADRI instrument to measure the acceptability of these two clinical decision rules among nurses during their impact trials at the pediatric ED. We aimed to disseminate the OADRI to a non-English speaking country by translating the instrument into Dutch and we undertook applicable validation activities of the instrument.

Methods

Study design and setting

We performed a prospective questionnaire evaluation among our 19 pediatric emergency department (ED) nurses at the Erasmus MC-Sophia Children's Hospital, Rotterdam, the Netherlands. A fixed sample of pediatric ED nurses anonymously completed the instrument at three evaluation moments during two impact trials as performed at our department in the period September 2010-October 2012 [13,14]. We continuously updated the implementation process by reminders by mail, posters and group instruction. The first evaluation moment took place at the beginning of the impact trials (2010), the second moment was halfway the impact trials (2011) and the last moment was at the end of the impact trials (2012).

Study intervention and participants

The original OADRI was translated from English into Dutch according to international guidelines with written permission of Jamie C. Brehaut, one of its developers [10]. We translated OADRI according to international guidelines [15] including two forward and two backward translations by independent native speakers without knowledge of the questionnaire. The Dutch version of the OADRI was used to test the acceptability and measure potential facilitators and barriers during the impact trials of the two clinical decision rules. Data were assured to be analysed confidentially. Since no patients were involved in this acceptability study and all questionnaire data were anonymised, informed consent was IRB exempt.

The clinical decision rules used in the impact trials provided high- or low-risk estimates for SBI (trial I) [13] and dehydration (trial II) [14], respectively. These impact trials were running parallel to our OADRI evaluation study. In the intervention group nurses were guided, after initial triage, to initiate additional tests (trial I) or rehydration intervention (trial II) for high-risk children [13,14]. Both control groups were evaluated according to the usual emergency care.

Data collection

Respondents were asked to indicate their level of agreement with each of the 12 statements on a 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree), or indicating "no opinion/don't know" (Supplemental File). The first 7 items were phrased such that a higher number indicated greater acceptability and for the last 5 the opposite was true. This strategy was chosen by the developers to avoid yes-saying bias [16].

The final total score consisted of the mean of all 12 items (recoded where necessary) and thus ranged from 0 to 6. Non-completed items were excluded from the final total scores. The mean of the remaining items served as the instrument score. Respondents who completed less than 8 of the 12 items were considered as not having completed the instrument and were excluded from the final analyses. Items for which "No opinion/Don't know" was selected were coded as the middle of the scale in line with the original validation paper [10].

OADRI validation activities and decision rule acceptability

Taking our small study size into account we only performed applicable validation activities. First, we evaluated the rates of missing items, and the mean and range of item scores. Next, we evaluated ceiling and floor effects (i.e. > 15% of responders reporting the highest or lowest response category) [17].

We interpreted incomplete questionnaires, added free text parts (explanation of answers) and majority answers of "No opinion/Don't know" as proxy statements for rule unclarity. We evaluated the diversity of the different answer possibilities, items for which > 50% "No opinion/Don't know" was selected were defined as non-informative. We measured mean item scores, indicating greater acceptability when higher numbers were selected [10].

To test differences between the item scores of the OADRI during the three moments of the impact trials we used linear regression analyses. Moreover, we tested changes in mean item scores within the suggested original categories of OADRI items [10]: 1) Aspects of innovation: items 1-4; 2) Decision maker: items 6-9, 11 and 3) Environment: items 5, 10 and 12.

Finally, we visualized implementation strategies during the trial period (e.g. feedback and periodic teaching sessions) [13,14], which might influence rule acceptability. Outcome measures were compared between the two evaluated clinical decision rules. All analyses were performed using SPSS 20.0 for Windows.

Results

Based on two forward translations of the OADRI one final Dutch OADRI questionnaire was defined. Two-fold backward translations, by independent native speakers without knowledge of the questionnaire, resulted in similar questionnaires to the original OADRI questionnaire. Of the 19 eligible pediatric ED nurses, 74-89% (14-17 nurses) completed the survey at the three different assessments during trial I (fever). For trial II (gastroenteritis) the response rate varied between 56-84% (11-16 nurses) during the different moments of the trial. Demographics of the participating nurses are presented in Table 1. The median age was 54 years (IQR 53-59), 89% were women (n = 17), and the median years of working experience was 35 years (IQR 34-42).

OADRI validation activities and decision rule acceptability

Table 2 describes the mean item scores for each moment of the impact trials. A ceiling effect was observed for all 12 OADRI questions in both trials, where 67% (8/12) was caused by reporting the highest category. Floor effects were not observed in OADRI responses during both trials.

There were no incomplete submissions (i.e. less than 8 of the 12 items completed). For OADRI responses during both trials we only had one missing answer in the total of all questionnaires. None of the participants added free text to their questionnaire. Overall percentage of items coded as "No opinion/Don't know" was 16% for trial I and 18% for trial II, and mostly due to OADRI item 9 ("evidence supporting the rule is flawed") and 11 ("does not account for important clinical cue").

OADRI item 9 was consequently defined as non-informative (> 50% "No opinion/Don't know") for all moments in both trials in contrast to OADRI item 11 which was only partly non-informative (first two moments trial I, second moment trial II).

Mean item scores tended to be above 3, indicating overall acceptability of the rule (i.e. higher numbers indicate greater acceptability) (Table 2). Overall, the mean item scores increased over time, especially from moment 1 to moment 2. For trial I the mean item score improvement was significant (p = 0.026), in contrast to trial II (p = 0.260). When we categorized OADRI items into the original item classification we observed significant increases in mean item score in category I (aspects of the innovation) (p = 0.020) and category III (environment) (p = 0.048) for trial I (Table 2).

We observed improved acceptability in relation to implementation activities per study moment, measured by increased mean item scores (Figure 1).

Discussion

Main findings

We completed the Dutch translation of the OADRI according to international guidelines and the disseminated use of OADRI to measure acceptability of clinical decision rules seems usable in this non-English country. Validation activities showed ceiling effects for each OADRI item indicating limited content validity of the instrument. However, we observed good acceptability of the introduction of two clinical decision rules among ED nurses as the rules appeared to be clear, with only 2 items being less informative ("evidence supporting the rule is flawed" (item 9) and "does not account for an important clinical cue" (item 11)). Responses to individual OADRI items guided possible improvement of decision rule implementation.

Comparison with other studies

In this study we evaluated the OADRI among ED nurses as in our impact trials the clinical decision models were integrated in the nurses' regular work-flow to evaluate the possibility of standardization and/or reallocation of their diagnostic tasks. To our knowledge this is the first study measuring nurses' acceptability of two clinical decision rules in different moments of their implementation at the pediatric ED. Different strategies for successful implementation of clinical decision support systems in practice are reported in literature [1,18]. In line with their recommendations our decision rules were computer based and they were part of the clinical workflow containing recommendations, rather than just assessments [18]. Moreover, as recommended by Reilly et al., our ED nurses were checked for accurate use of the rules to provide feedback, but also to assess whether clinical judgment improves on decision rule based judgment [1]. Other studies have been written about the translation of clinical research into practice and the role of behavioural changes among health care professionals on this topic [1,7,19-21]. According to a systematic review effective interventions should include reminders; multifaceted interventions including e.g. auditing, feedback and interactive educational meetings [21]. Overall, all these different implementation strategies are supposed to increase rule acceptability in clinical practice. However, measuring acceptability of newly introduced clinical decision rules is not routinely done. A systematic review describing clinical prediction rules for children included 137 studies and reported in only 8 studies broad validation of the results. None of the included studies had undergone impact analysis and subsequently measured rule acceptability [22].

Clinical and research implications

From literature we learned that before we can improve clinical practice using clinical decision rules we should interact with all the barriers and incentives that are already known. But in clinical practice it is sometimes difficult to predict which barriers need more attention to support optimal implementation. At our ED, we used the OADRI for interactive periodic teaching sessions to increase the acceptability of using both decision rules. We discussed for example OADRI questions related to the importance of the rules, including the underlying evidence of rule development. Moreover, we discussed the difficulties of using the rules in clinical practice where the answers on OADRI questions related to environment were essential sources of information. Overall, we experienced the OADRI covered enough tools to support these teaching sessions. We observed that especially OADRI mean item scores considering aspects of the innovation (e.g. the rule is easy to use; the rule is useful in my practice) and aspects of environment (e.g. colleagues support, working environment) increased over time. Hence the OADRI can support this important study moment by interact on specific items scored on this questionnaire and this Dutch translated version is just the first step of OADRI dissemination through non-English speaking countries.

In our study we observed a better acceptability of trial I (fever) compared with trial II (gastroenteritis) as measured by the OADRI. One of the possible reasons for this result includes the limited response rate to OADRI by the nurses in trial II. Results may be underpowered and therefore no effect could be identified. Second, the rehydration advices of trial II might be less innovatory than guidance on initiating additional diagnostic tests regarding fever in trial I, taking the large study numbers into account evaluating rehydration therapy in children with gastroenteritis [23,24]. Moreover, responses to OADRI during trial II started with higher mean item scores than trial I, which may indicate higher acceptability from the beginning, but also might have resulted in fewer differences in mean OADRI item scores between the different study moments.

As previously mentioned, we focused on implementation of the clinical decision model on nurses' level, i.e. nurses were guided to initiate additional tests/interventions for high-risk children. However, OADRI is not specifically developed for nurses and future studies could focus on acceptability of physicians or other medical staff either.

Strengths and Limitations

The main strength of this study is our translation of the OADRI into Dutch to create more possibilities using this instrument more internationally including other populations and settings. As we measured the acceptability of rule implementation in time we could describe the process of implementation as a dynamic process and identify with the OADRI specific barriers and facilitators along this implementation period which gave the opportunity to intervene directly.

This study has also some limitations. First, as our study was covering one tertiary pediatric ED we could include only a limited number of ED nurses. Moreover, as our ED is a high research setting, nurses are very used to participate in different studies which might have resulted in a relative overestimation of our validation results of OADRI [25,26]. Second, our study size did not justify a full validation study. Reproducibility of OADRI, which is the degree to which repeated measurements in the test-retest period provide similar answers, was not measured in this study due to the anonymous questionnaire reply. Moreover, future research should focus on extended validation measures as internal consistency and construct validity.

Third, the original categorization of OADRI items as suggested by the developers was based on their post-hoc thoughts, rather than based on reliable methods of item selection and reduction analyses [10,17]. Additional broader evaluation of studies with greater sample sizes could provide more complete validation information, including evaluation of possible reduction or changes in categorization of OADRI items. Moreover, the small sample size of our study prevents good generalizability to other ED nurses. Last, it would have been preferred to analyse individual changes in time rather than mean item scores differences in time. This would have corrected for variability between subjects [27]. However, due to the anonymously character of the study design we could not relate the questionnaires responses within the different study moments to persons.

Conclusion

Disseminated use of the OADRI seems usable in the acceptability measurement of newly implemented clinical decision rules in practice, as shown by an application of its Dutch version. Although broad-spectrum validation should be incorporated in future application of OADRI, we observed good acceptability of the introduction of two clinical decision rules among ED nurses at our emergency department. Improvement of clinical decision rules acceptability could be achieved by responses to individual guided interaction based on OADRI items.

Abbreviations

ED: Emergency Department; SBI: Serious bacterial infection; CRP: C-reactive protein; AUC: area under the receiver operating characteristic curve; CI: Confidence Interval

Ethics

Since no patients were involved in this study, hospital ethic approval was not necessary.

Conflict of Interest

All authors declare to have no conflict of interest.

Funding

Authors EK and DG are supported by ZonMW, the Dutch organisation for Health Research and Development, and Erasmus University Medical Centre Rotterdam. The study sponsor had no role in study design, in the collection, analysis, and interpretation of data; in the writing of the report; nor in the decision to submit the paper for publication.

Acknowledgement

We want to thank Jamie C. Brehaut for his approval using the OADRI, and moreover for reviewing the translated instrument before starting the study. We gratefully acknowledge the contributions and dedication of all emergency nurses who participated in this study. We gratefully acknowledge Matthijs Goedvolk (MG), Yvette van Ierland (YvI), Adam Weir (AW) and Marjon Cnossen (MC) for their help in the translation process of the OADRI.

References

1. Reilly BM, Evans AT (2006) Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144: 201-209.
   


2. Stiell IG, Bennett C (2007) Implementation of clinical decision rules in the emergency department.AcadEmerg Med 14: 955-959.
   


3. Wasson JH, Sox HC, Neff RK, Goldman L (1985) Clinical prediction rules. Applications and methodological standards.N Engl J Med 313: 793-799.
   


4. Laupacis A, Sekar N, Stiell IG (1997) Clinical prediction rules. A review and suggested modifications of methodological standards.JAMA 277: 488-494.
   


5. Nijman R, Vergouwe Y, Thompson M, Veen van M, Meurs van A, Lei van der J, et al. (2013) Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study. Bmj 346: f1706.
   


6. Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, et al. (2013) Prognosis Research Strategy (PROGRESS) 3: prognostic model research.PLoS Med 10: e1001381.
   


7. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, et al. (1999) Why don't physicians follow clinical practice guidelines? A framework for improvement.JAMA 282: 1458-1465.
   


8. Graham ID, Logan J (2004) Innovations in knowledge transfer and continuity of care.Can J Nurs Res 36: 89-103.
   


9. Logan J, Graham ID (1998) Toward a comprehensive interdisciplinary model of health care research use. SciCommun 20: 227-246.
   


10. Brehaut JC, Graham ID, Wood TJ, Taljaard M, Eagles D, et al. (2010) Measuring acceptability of clinical decision rules: validation of the Ottawa acceptability of decision rules instrument (OADRI) in four countries. Med Decis Making 30: 398-408.
    


11. Stiell IG, Wells GA, Vandemheen KL, Clement CM, Lesiuk H, et al. (2001) The Canadian C-spine rule for radiography in alert and stable trauma patients.JAMA 286: 1841-1848.
    


12. Stiell IG, Wells GA, Vandemheen K, Clement C, Lesiuk H, et al. (2001) The Canadian CT Head Rule for patients with minor head injury.Lancet 357: 1391-1396.
    


13. de Vos-Kerkhof E, Nijman RG, Vergouwe Y, Polinder S, Steyerberg EW, et al. (2015) Impact of a clinical decision model for febrile children at risk for serious bacterial infections at the emergency department: a randomized controlled trial. PLoS One 10: e0127620.
    


14. Geurts G, de Vos-Kerkhof E, Polinder S, E. S, Van der Lei J, Moll HA, et al. Impact of clinical decision support based treatment in young children with gastroenteritis at the emergency department: a randomised controlled trial. Submitted. Dutch trial register: NTR2304. 2014.
    


15. Beaton DE, Bombardier C, Guillemin F, Ferraz MB (2000) Guidelines for the process of cross-cultural adaptation of self-report measures.Spine (Phila Pa 1976) 25: 3186-3191.
    


16. Butt ML, Pinelli J, Boyle MH, Thomas H, Hunsberger M, et al. (2009) Development and evaluation of an instrument to measure parental satisfaction with quality of care in neonatal follow-up.J Dev BehavPediatr 30: 57-65.
    


17. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, et al. (2007) Quality criteria were proposed for measurement properties of health status questionnaires.J ClinEpidemiol 60: 34-42.
    


18. Kawamoto K, Houlihan CA, Balas EA, Lobach DF (2005) Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. Bmj 330: 765.
    


19. Grol R (1992) Implementing guidelines in general practice care.Qual Health Care 1: 184-191.
    


20. Grol R,Wensing M (2004) What drives change? Barriers to and incentives for achieving evidence-based practice.Med J Aust 180: S57-60.
    


21. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, et al. (1998) Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ 317:465-468.
    


22. Maguire JL,Kulik DM, Laupacis A, Kuppermann N, Uleryk EM, et al. (2011) Clinical prediction rules for children: a systematic review.Pediatrics 128: e666-677.
    


23. Freedman SB, Ali S, Oleszczuk M, Gouin S, Hartling L (2013) Treatment of acute gastroenteritis in children: an overview of systematic reviews of interventions commonly used in developed countries. Evid Based Child Health 8:1123-1137.
    


24. Freedman SB, Parkin PC, Willan AR, Schuh S (2011) Rapid versus standard intravenous rehydration in paediatric gastroenteritis: pragmatic blinded randomised clinical trial. BMJ 343:d6976.
    


25. van Veen M, Steyerberg EW, Ruige M, van Meurs AH, Roukema J, et al. (2008) Manchester triage system in paediatric emergency care: prospective observational study.BMJ 337: a1501.
    


26. Nijman RG, Vergouwe Y, Thompson M, van Veen M, van Meurs AH, van der Lei J, et al.(2013) Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study. BMJ 346:f1706.
    


27. Noone M, Semkovska M, Carton M, Dunne R, Horgan JP, O'Kane B, et al.(2014) Construction and updating of a public events questionnaire for repeated measures longitudinal studies. Front Psychol5:230.