Skip to main content
Download PDF
Download ePub

Expanding methods to address RE-AIM metrics in hybrid effectiveness-implementation studies

Implementation Science Communications volume 5, Article number: 123 (2024) Cite this article

Abstract

Background

Dissemination and implementation science is an evolving field that focuses on the strategies and mechanisms by which scientific evidence is adopted, used, and sustained in clinical and community practice.

Main body

Implementation scientists are confronted by the challenge to balance rigor and generalizability in their work while also attempting to speed the translation of evidence into clinical and community practice. Hybrid Effectiveness-Implementation studies and the RE-AIM framework were conceptualized to address these challenges. Hybrid Effectiveness-Implementation (HEI) studies provide methods of examining the effectiveness of health promoting interventions while concurrently assessing the utility of dissemination and implementation strategies designed to enhance the application of evidence-based principles in practice. RE-AIM provides a set of planning and evaluation dimensions that can be assessed with a goal to balance internal and external validity. The purpose of this commentary is to provide clarity on definitions of each approach and how to effectively use them together to answer research questions that will advance dissemination and implementation science for health promotion.

Conclusions

We provide examples of concerted use of RE-AIM within HEI studies from the literature and focus on language to provide a clarity and consistency across research questions, designs, and settings. We share how to operationalize RE-AIM dimensions in HEI studies for both dissemination and implementation strategies. Future directions include refining, defining, and evaluating each RE-AIM dimension within hybrid studies.

Peer Review reports

Introduction

Dissemination and implementation science is defined as the study of the strategies and mechanisms by which scientific evidence is disseminated and implemented in community or clinical settings to improve outcomes for a specified population [1, 2]. This can include developing and testing innovative approaches to identify, understand, and overcome barriers to the adoption, adaptation, integration, scale-up and sustainability of evidence-based interventions, tools, policies, and guidelines. Over the past 25 years fundamental challenges in dissemination and implementation science have centered on the need to balance internal and external validity, assess a broad range of outcomes, and speed the translation of evidence into clinical and community practice settings [3]. There remains a substantial translational lag time in evidence-based interventions that promote health such as those for improving physical activity, tobacco cessation, healthful eating, and weight control [4,5,6,7,8,9,10,11,12,13]. Therefore, dissemination and implementation researchers have worked to develop program, practice, and policy interventions that address (a) multi-leveled outcomes, (b) clinical and community infrastructure, (c) implementation staff characteristics and perceptions, and (d) the characteristics and perceptions of the population intended to achieve a health benefit—all while attempting to rule out alternative explanations for outcomes within complicated and complex clinical and community systems [14].

Two approaches were independently developed to plan, intervene upon, and evaluate dissemination and implementation strategies concurrently with effectiveness outcomes. These approaches are the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, and Maintenance) and Hybrid Effectiveness-Implementation (HEI) studies [3, 15]. Both RE-AIM and HEI approaches have been widely used together, given their complementary methodological strengths. A PubMed search using the words “RE-AIM” and “Hybrid Effectiveness Implementation” yielded 322 manuscripts in the 10 years since the initial paper on HEI studies was published (2012–2022). Similarly, a search in the National Institutes of Health Reporter for 2019–2022 yielded 95 active projects that use RE-AIM to test health interventions within HEI studies across a wide range of behavioral interventions.

While these two approaches have previously been used in combination [14], there remains inconsistencies and lack of clarity on how to operationalize RE-AIM outcomes across HEI study types [16]. Therefore, the purpose of this conceptual paper is to provide clarity on definitions of each approach and how to effectively use them together to answer significant research questions that will advance dissemination and implementation science and health promotion. Based on our areas of expertise, we highlight nutrition, physical activity, and obesity studies throughout this manuscript.

The RE-AIM framework

RE-AIM provides a set of planning and evaluation outcomes that can be assessed with a goal to balance internal and external validity while concurrently providing data that can be used to predict or determine the public health impact of interventions when delivered in real-world settings [17]. RE-AIM was developed to address the multi-leveled nature of dissemination and implementation outcomes with a dual focus on assessing health benefits as well as intervention delivery by staff and organizations [3]. Specifically, reach is defined as the absolute number, proportion, and representativeness of individuals who are willing to participate in the intervention relative to the broader eligible population (i.e., beneficiaries). Effectiveness is the impact of the intervention on the intended health outcomes for a given sample or population. This includes potential negative effects, effects on quality of life, and effects across subgroups. Adoption is the absolute number, proportion, and representativeness of settings and intervention delivery agents who are willing to initiate a program, relative to the broader eligible population of staff and settings. Implementation refers to the degree to which the intervention is delivered as intended: is the fidelity to the various elements of an intervention’s key functions or components, including consistency of delivery as intended and the time and cost of the intervention. Maintenance at the individual level is the sustained change in health/behavioral outcomes targeted by the intervention, while Maintenance at the organizational level is the sustained implementation of the evidence-based interventions as intended. Researchers using the RE-AIM framework were (and are) encouraged to test scientific questions in contexts that reflect what would occur in typical clinical or community practice [3].

Conceptualizing HEI studies

HEI studies address research questions about the effectiveness of an intervention and its implementation within a single study [18]. HEI approaches distinguish between two types of interventions—one that focuses on the beneficiary’s health outcomes and one that focuses on dissemination and implementation outcomes. To distinguish these two types of interventions, with the context of HEI studies, we refer to those that target beneficiary health outcomes as the health promoting intervention while those that target dissemination and implementation outcomes as to strategies [19]. For example, considering physical activity promotion, HEI studies can focus on assessing the effects of an intervention on physical activity behavior while concurrently assessing the implementation context or the utility of implementation strategies designed to enhance the delivery of the intervention in a specific settings [15].

The main goal of HEI studies is to help determine whether a study should focus primarily on intervention effectiveness while exploring implementation factors, or primarily on implementation quality while measuring health outcomes, or both [18]. Indeed the three most common types of HEI studies focus on the following: (a) testing the effects of an intervention on health outcomes (primary outcome), while gathering information on intervention implementation (Type 1); (b) testing the effects of strategies on adoption, implementation, or sustainability of an evidence-based intervention (primary outcome), while measuring health outcomes (Type 3); and testing the effects on an intervention on health outcomes and the effects of a strategy on dissemination and implementation outcomes (co-primary outcomes; Type 2). While there are more variations of HEI studies [18], and these may also be expanded to include context as an additional independent variable [20], this discussion will focus on the three most common types described in the seminal HEI paper.

Distinguishing evidence-based interventions and dissemination and implementation strategies

A fundamental need in the application of HEI studies is to clearly specify and distinguish an intervention intended to result in health improvements among beneficiaries versus dissemination and implementation strategies applied at the level of setting, organization, and/or staff with a goal to improve adoption, implementation, and maintenance [21]. A functional starting point is to consider the intervention as an independent variable in the study that include comparison conditions with hypotheses related to improvements in participant health outcomes. This includes, among other approaches, theory-based interventions that target social cognitions [22] or group dynamics principles [23] to promote health behaviors in the beneficiaries that are exposed to a given intervention. Dissemination and implementation science also uses the term evidence-based intervention to highlight that in addition to being hypothesized to improve health outcomes, that the intervention has demonstrated efficacy in a single large trial or across a series of trials summarized in a review of literature [1].

Dissemination and implementation strategies can also be described as the independent variable or an observation in an HEI study that are hypothesized to directly produce or describe improvements in intervention adoption, implementation, and sustainability. Dissemination and implementation strategies include activities that address physical activity promotion indirectly by focusing on improving implementation quality and improving the fit between the delivery setting and the behavioral intervention. In general, strategies can be further described as those that focus on planning efforts (e.g., gathering data, building support, and developing partnerships in the delivery setting), educational approaches (e.g., sharing information on the evidence-based intervention), system adaptations (e.g., changes to implementer workflow), quality improvement (e.g., training staff), the use of economic incentives (e.g., incentives based on performance standards), and addressing policy (e.g., guideline development and application) [24]. Expert recommendations exist to support researchers in selecting and specifying dissemination and implementation strategies [25, 26].

The intersection of RE-AIM framework and HEI studies

Considering the intended outcome of an evidence-based intervention or dissemination and implementation strategy, based on the RE-AIM dimensions, can also be helpful. Specifically, any activities that are intended to improve reach or adoption are dissemination strategies. In each case, the strategies are focused on the spread of an intervention to the population intended to see a health benefit (i.e., reach) or to the population of staff and settings intended to deliver the intervention (i.e., adoption). Activities that are intended to improve implementation and sustainability are considered implementation strategies. In these cases, strategies are focused on the quality of delivery from initial application by specific staff in a given setting (i.e., implementation) or on sustained quality of delivery (i.e., maintenance). Finally, activities intended to improve intervention health outcomes or the degree to which beneficiaries sustain those improvements are evidence-based, or health promoting, interventions.

We acknowledge that there can be some nuances in distinguishing between interventions and strategies to change specific RE-AIM outcomes. Prior to the introduction of HEI designs and terminology, behavioral research was (and is) rich with examples of multi-level interventions that have historically focused on behavior change through delivery agent training [27], engagement of organizational leadership [28], systems integration [29], and partnership with community-based organizations [30]. These multilevel interventions typically include components that intend to directly improve beneficiary’s behavioral outcomes (i.e., the evidence-based intervention within HEI approaches) and indirectly improve behavioral outcomes through changes to the environment or improvements in implementation fidelity (i.e., a dissemination and implementation strategy within HEI approaches).

To improve the ease of description of HEI study types that use RE-AIM, mapping processes can help to provide clarity during preparation, execution, and interpretation of a study [31]. This can help avoid issues related to underreporting or misreporting in of reach, adoption, cost, and setting level maintenance, or misclassification of reach and adoption indicators [16, 32]. These issues are often exacerbated in HEI studies due to the need to identify appropriate denominators for reach and adoption, distinguish between individual- and setting-level outcomes, and distinguish between health promoting interventions and dissemination and implementation strategies. Further, most dissemination strategies (i.e., intended to improve reach or adoption) inaccurately labeled as implementation strategies.

Table 1 RE-AIM operationalization across HEI study types
Full size table

Specifically, regardless of HEI type, reach, effectiveness, and maintenance of health effects are assessed at the level of the beneficiary –with effectiveness and maintenance shifting from a primary, to a co-primary, to a secondary outcome, in a HEI type 1, 2 and 3, respectively. Similarly, adoption, implementation, and maintenance are assessed at the level of the staff, setting, and/or system that is delivering the evidence-based intervention within their organization. These outcomes also shift from a secondary aim, a co-primary aim, and a primary aim in the HEI 1, 2 and 3, respectively. Of note, while assessed at different levels, reach and adoption are dissemination outcomes, while staff, setting, and system implementation and organizational maintenance are implementation outcomes.

There are several examples of published studies that have employed RE-AIM in HEI studies. Examples of the use of RE-AIM within HEI type 1 studies include examining sugar-sweetened beverage reduction interventions in US adults and children curran [33, 34], and testing physical activity interventions among cancer survivors in Canada [35]. RE-AIM has also been used in type 2 HEI studies testing interventions to improve clinician counseling and patient physical activity in Mexico [36], to improve physical activity among older adults in Canada [37], and to improve young children’s eating and health behaviors in the US [38]. Finally, RE-AIM has been used in type 3 HEI studies to assess the effects of diabetes prevention programs in the Veteran Affairs health system [39, 40], and the effects of interventions on young children’s eating and physical activity behaviors [41, 42]. To better illustrate the joint application of these two approaches, we include detailed examples of studies using RE-AIM and HEI studies in nutrition, physical activity, and obesity [42,43,44,45] in Table 2. Across studies, RE-AIM outcomes are operationalized inconsistently within each HEI study type, and there is lack of clarity regarding the intervention/strategy being evaluated and the primary outcome(s) it targets.

Table 2 Example of current RE-AIM operationalizations within HEI study types
Full size table

The way forward

This manuscript aims to provide clear definitions and guidance on how to operationalize RE-AIM outcomes in HEI studies. We also offer new types of HEI studies anchored on specific RE-AIM outcomes and try to guide this anchoring process using a subway map analogy expanded from Lane-Fall et al., [46] (Fig. 1). First, consider that reach and adoption reflect dissemination outcomes, while implementation and maintenance at the staff/organization/system level reflect implementation outcomes. Hence, a study can focus on examining adoption of an evidence-based intervention (i.e., dissemination outcome) or the fidelity of an evidence-based intervention (implementation outcome). Depending on the dissemination and implementation strategy employed and the target outcomes, type 2 and 3 hybrids studies might include Effectiveness and one or more of the other four dimensions of RE-AIM: reach, adoption, implementation, maintenance (Fig. 1).

Fig. 1
figure 1

"Subway" schematic for design and outcome considerations based on hybrid effectiveness-implementation studies and the RE-AIM framework

Full size image

For instance, if the focus of a study is to determine the effects of a dissemination strategy on intervention reach, a hybrid effectiveness-reach study type may be used. This study would test whether such strategy (e.g., mass media campaign) improves the participation rate and representativeness of the target beneficiary population. If the focus of the study is to increase the participation and representativeness of the staff delivering the evidence-based intervention, that may be a hybrid effectiveness-adoption study. Alternatively, if the goal is to investigate the influence of specific dissemination and implementation strategies have on the evidence-based intervention longevity within a system, that would be considered a hybrid effectiveness-maintenance study.

To better illustrate this, we offer a subway map to help visualize the process of selecting the most appropriate RE-AIM-HEI study type for the research question of interest. The subway line starts at the HEI stop of the Lane-Fall et al. subway map [46]. The primary outcome should guide the path taken down the subway map. If the primary outcome is intervention effectiveness (e.g., impact on health/behavioral outcomes) and the secondary outcome is a dissemination or implementation outcome, the HEI 1 lane should be followed and RE-AIM operationalized as per usual (see Table 1 for guidance on this). If the primary outcomes include a health/behavioral outcome and a dissemination or implementation outcome, the HEI 2 lane is followed. If the primary outcome is a dissemination or implementation outcome, and the secondary outcome a health/behavioral outcome, the HEI 3 lane is followed. HEI 2 and 3 lines further splinter to reflect the different RE-AIM hybrid studies we previously described depending on the focus of the primary dissemination and implementation outcome. Researchers and practitioners can use this subway map to determine which RE-AIM hybrid study is most appropriate to address their dissemination or implementation research question.

Notably, Curran and colleagues stated that the HEI “definition and typology offered [in their manuscript] must be considered constructs still in evolution.” [15] This has been evident over the last ten years of applying HEI [18]. We anticipate that the complexity and application of the RE-AIM Framework and HEI study types will continue to evolve, too. For example, while less prominent in the literature to date, investigators may be interested in a hybrid effectiveness-reach study. Further still, depending on the study, a team may be using reach or effectiveness as the primary, co-primary or secondary outcome. We provided a subway illustration for these evolving study types for your consideration as well; where HeR indicates reach is a primary outcome and effectiveness a secondary outcome; HER indicates effectiveness and reach as co primary; and HEr indicates effectiveness as primary and reach as secondary. See Supplemental Figure.

In sum, RE-AIM can be operationalized in all three types of HEI studies. The outcome specification approach we propose here would promote consistency in implementation research by facilitating our understanding of the impact, generalizability, and sustainability not only of evidence-based interventions, but also of dissemination and implementation strategies. As RE-AIM continues to evolve, it is well positioned to address the realist evaluation question of what interventions are effective, with which dissemination and implementation strategies, for whom, in what settings, how and why, and for how long [3, 16].

Conclusion

This manuscript provides recommendations for using RE-AIM within HEI study types, and offers a new typology of hybrid studies anchored on RE-AIM outcomes. A general concern for dissemination and implementation researchers is that there is confusion about dissemination and implementation as an action versus dissemination and implementation as a science. The “science” designation implies that there is a research question, research design, and research methodologies used to understand a specific phenomenon. Frameworks such as RE-AIM and study types such as HEI are just one way to contribute to the science of implementing health interventions. Empirical and practical gaps still remain in the field and novel ways to approach them, such as the approach offered here, should continue to be developed, challenged and refined.

Data availability

Not applicable.

References

  1. Estabrooks PA, Brownson RC, Pronk NP. Dissemination and implementation science for public health professionals: an overview and call to action. Prev Chronic Dis. 2018;15:E162.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Neta G, Glasgow RE, Carpenter CR, Grimshaw JM, Rabin BA, Fernandez ME, et al. A framework for enhancing the value of research for dissemination and implementation. Am J Public Health. 2015;105(1):49–57.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Galaviz KI, Harden SM, Smith E, Blackman KC, Berrey LM, Mama SK, et al. Physical activity promotion in Latin American populations: a systematic review on issues of internal and external validity. Int J Behav Nutr Phys Act. 2014;11:77.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Sweet SN, Ginis KA, Estabrooks PA, Latimer-Cheung AE. Operationalizing the RE-AIM framework to evaluate the impact of multi-sector partnerships. Implement Sci. 2014;9:74.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Blackman KC, Zoellner J, Berrey LM, Alexander R, Fanning J, Hill JL, et al. Assessing the internal and external validity of mobile health physical activity promotion interventions: a systematic literature review using the RE-AIM framework. J Med Internet Res. 2013;15(10):e224.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Van Acker R, De Bourdeaudhuij I, De Cocker K, Klesges LM, Cardon G. The impact of disseminating the whole-community project ‘10,000 steps’: a RE-AIM analysis. BMC Public Health. 2011;11:3.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Antikainen I, Ellis R. A RE-AIM evaluation of theory-based physical activity interventions. J Sport Exerc Psychol. 2011;33(2):198–214.

    Article  PubMed  Google Scholar 

  9. Collard DC, Chinapaw MJ, Verhagen EA, van Mechelen W. Process evaluation of a school based physical activity related injury prevention programme using the RE-AIM framework. BMC Pediatr. 2010;10: 86.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Dunton GF, Lagloire R, Robertson T. Using the RE-AIM framework to evaluate the statewide dissemination of a school-based physical activity and nutrition curriculum: exercise your options. Am J Health Promotion. 2009;23(4):229–32.

    Article  Google Scholar 

  11. Bopp M, Wilcox S, Laken M, Hooker SP, Saunders R, Parra-Medina D, et al. Using the RE-AIM framework to evaluate a physical activity intervention in churches. Prev Chronic Dis. 2007;4(4):A87.

    PubMed  PubMed Central  Google Scholar 

  12. Dzewaltowski DA, Estabrooks PA, Klesges LM, Bull S, Glasgow RE. Behavior change intervention research in community settings: how generalizable are the results? Health Promot Int. 2004;19(2):235–45.

    Article  PubMed  Google Scholar 

  13. Estabrooks PA, Gyurcsik NC. Evaluating the impact of behavioral interventions that target physical activity: issues of generalizability and public health. Psychol Sport Exerc. 2003;4(1):41–55.

    Article  Google Scholar 

  14. Estabrooks PA. An overview of dissemination and implementation science in physical activity and health promotion. Kinesiol Rev. 2023;12:14.

    Article  Google Scholar 

  15. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci. 2021;5(1):e126.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Curran GM, Landes SJ, McBain SA, Pyne JM, Smith JD, Fernandez ME, et al. Reflections on 10 years of effectiveness-implementation hybrid studies. Front Health Serv. 2022;2:2.

    Article  Google Scholar 

  19. Curran GM, Landes SJ, McBain SA, Pyne JM, Smith JD, Fernandez ME, et al. Reflections on 10 years of effectiveness-implementation hybrid studies. Front Health Serv. 2022;2:10.

    Article  Google Scholar 

  20. Kemp CG, Wagenaar BH, Haroz EE. Expanding hybrid studies for implementation research: intervention, implementation strategy, and context. Front Public Health. 2019;7: 325.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Curran GM. Implementation science made too simple: a teaching tool. Implement Sci Commun. 2020;1:27.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Burgess E, Hassmen P, Welvaert M, Pumpa KL. Behavioural treatment strategies improve adherence to lifestyle intervention programmes in adults with obesity: a systematic review and meta-analysis. Clin Obes. 2017;7(2):105–14.

    Article  CAS  PubMed  Google Scholar 

  23. Estabrooks PA, Harden SM, Burke SM. Group dynamics in physical activity promotion: what works? Social Personality Psychol Compass. 2012;6:22.

    Article  Google Scholar 

  24. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57.

    Article  PubMed  Google Scholar 

  25. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Balis LE, Houghtaling B, Harden SM. Using implementation strategies in community settings: an introduction to the Expert Recommendations for Implementing Change (ERIC) compilation and future directions. Transl Behav Med. 2022;12(10):965–78.

    Article  PubMed  Google Scholar 

  27. Kennedy SG, Smith JJ, Morgan PJ, Peralta LR, Hilland TA, Eather N, et al. Implementing resistance training in secondary schools: a cluster randomized controlled trial. Med Sci Sports Exerc. 2018;50(1):62–72.

    Article  PubMed  Google Scholar 

  28. McEachan RR, Lawton RJ, Jackson C, Conner M, Lunt J. Evidence, theory and context: using intervention mapping to develop a worksite physical activity intervention. BMC Public Health. 2008;8: 326.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Estabrooks PA, Harden SM, Almeida FA, Hill JL, Johnson SB, Porter GC, et al. Using integrated research-practice partnerships to move evidence-based principles into practice. Exerc Sport Sci Rev. 2019;47(3):176–87.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Benjamin SE, Ammerman A, Sommers J, Dodds J, Neelon B, Ward DS. Nutrition and physical activity self-assessment for child care (NAP SACC): results from a pilot intervention. J Nutr Educ Behav. 2007;39(3):142–9.

    Article  PubMed  Google Scholar 

  31. Smith JD, Li DH, Rafferty MR. The implementation research logic model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci. 2020;15(1):84.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Harden SM, Gaglio B, Shoup JA, Kinney KA, Johnson SB, Brito F, et al. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst Rev. 2015;4:155.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Zoellner JM, Hedrick VE, You W, Chen Y, Davy BM, Porter KJ, et al. Effects of a behavioral and health literacy intervention to reduce sugar-sweetened beverages: a randomized-controlled trial. Int J Behav Nutr Phys Act. 2016;13:38.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Zoellner JM, Porter KJ, You W, Chow PI, Ritterband LM, Yuhas M, et al. Kids SIPsmartER, a cluster randomized controlled trial and multi-level intervention to improve sugar-sweetened beverages behaviors among Appalachian middle-school students: rationale, design & methods. Contemp Clin Trials. 2019;83:64–80.

    Article  PubMed  PubMed Central  Google Scholar 

  35. McNeely ML, Sellar C, Williamson T, Shea-Budgell M, Joy AA, Lau HY, et al. Community-based exercise for health promotion and secondary cancer prevention in Canada: protocol for a hybrid effectiveness-implementation study. BMJ Open. 2019;9(9): e029975.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Galaviz KI, Estabrooks PA, Ulloa EJ, Lee RE, Janssen I, Lopez YTJ, et al. Evaluating the effectiveness of physician counseling to promote physical activity in Mexico: an effectiveness-implementation hybrid study. Transl Behav Med. 2017;7(4):731–40.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Mangin D, Lamarche L, Oliver D, Bomze S, Borhan S, Browne T, et al. Health TAPESTRY Ontario: protocol for a randomized controlled trial to test reproducibility and implementation. Trials. 2020;21(1):714.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Berkel C, Smith JD, Bruening MM, Jordan N, Fu E, Mauricio AM, et al. The family check-up 4 health: study protocol of a randomized type II hybrid effectiveness-implementation trial in integrated primary care (the healthy communities 4 healthy students study). Contemp Clin Trials. 2020;96: 106088.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Damschroder LJ, Reardon CM, AuYoung M, Moin T, Datta SK, Sparks JB, et al. Implementation findings from a hybrid III implementation-effectiveness trial of the Diabetes Prevention Program (DPP) in the Veterans Health Administration (VHA). Implement Sci. 2017;12(1):94.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Damschroder LJ, Moin T, Datta SK, Reardon CM, Steinle N, Weinreb J, et al. Implementation and evaluation of the VA DPP clinical demonstration: protocol for a multi-site non-randomized hybrid effectiveness-implementation type III trial. Implement Sci. 2015;10:68.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Swindle T, Johnson SL, Whiteside-Mansell L, Curran GM. A mixed methods protocol for developing and testing implementation strategies for evidence-based obesity prevention in childcare: a cluster randomized hybrid type III trial. Implement Sci. 2017;12(1):90.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Vaughn AE, Studts CR, Powell BJ, Ammerman AS, Trogdon JG, Curran GM, et al. The impact of basic vs. enhanced Go NAPSACC on child care centers’ healthy eating and physical activity practices: protocol for a type 3 hybrid effectiveness-implementation cluster-randomized trial. Implement Sci. 2019;14(1):101.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Ma J, Yank V, Lv N, Goldhaber-Fiebert JD, Lewis MA, Kramer MK, et al. Research aimed at improving both mood and weight (RAINBOW) in primary care: a type 1 hybrid design randomized controlled trial. Contemp Clin Trials. 2015;43:260–78.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Frazier MC, Balis LE, Armbruster SD, Estabrooks PA, Harden SM. Adaptations to a statewide walking program: use of iterative feedback cycles between research and delivery systems improves fit for over 10 years. Transl Behav Med. 2024;14(1):45–53.

    Article  PubMed  Google Scholar 

  45. Harden SM, Johnson SB, Almeida FA, Estabrooks PA. Improving physical activity program adoption using integrated research-practice partnerships: an effectiveness-implementation trial. Transl Behav Med. 2017;7(1):28–38.

    Article  PubMed  Google Scholar 

  46. Lane-Fall M, Curran G, Beidas R. Scoping implementation science for the beginner: locating yourself on the "subway line" of translational research. BMC Med Res Methodol. 2019;19. https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12874-019-0783-z.

Download references

Acknowledgements

Not applicable.

Funding

KIG is supported by the National Heart, Lung, and Blood Institute of NIH (K01HL149479).

PAE is supported by the University of Utah Clinical and Translational Science Institute Dissemination and Implementation Science Core (UM1TR004409).

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Authors and Affiliations

  1. Physical Activity Research and Community Implementation Laboratory, Human Nutrition, Foods, and Exercise, Virginia Tech, 1981 Kraft Drive, Blacksburg, VA, 24060, USA

    Samantha M. Harden

  2. Department of Applied Health Science, Indiana University School of Public Health-Bloomington, #1025 E 7th, Room 116D, Bloomington, IN, 47405, USA

    Karla I. Galaviz

  3. Department of Health & Kinesiology, College of Health, University of Utah, 250 South 1850 East, Salt Lake City, UT, 84112, USA

    Paul A. Estabrooks

Authors
  1. Samantha M. Harden
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Karla I. Galaviz
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Paul A. Estabrooks
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

All authors met regularly to discuss content of the commentary and how to present the recommendations. Although all materials represent shared contributions, we note that: PAE created Table 1, SMH created Table 2, and KIG created the Figures. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Samantha M. Harden or Karla I. Galaviz.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Harden, S.M., Galaviz, K.I. & Estabrooks, P.A. Expanding methods to address RE-AIM metrics in hybrid effectiveness-implementation studies. Implement Sci Commun 5, 123 (2024). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s43058-024-00646-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s43058-024-00646-0

Keywords

Implementation Science Communications

ISSN: 2662-2211