Skip to main content
Download PDF
Download ePub

Collaboration strategies affecting implementation of a cross-systems intervention for child welfare and substance use treatment: a mixed methods analysis

Implementation Science Communications volume 5, Article number: 127 (2024) Cite this article

Abstract

Background

Collaboration strategies refer to policies and practices used to align operations and services across organizations or systems. These strategies can influence implementation of cross-system interventions focused on improving integration of care, but remain under-specified and under-examined. This study identifies collaboration strategies and the conditions under which they affected implementation of Sobriety Treatment and Recovery Teams (START), an evidence-based intervention focused on integrating child welfare and behavioral health services for families involved with both systems.

Methods

Our study sample included 17 county child welfare agencies that implemented START. Data on collaboration strategies and organizational context were obtained from key informant interviews, frontline worker surveys, and contracts. Contextual data were drawn from secondary data, and fidelity data were drawn from an administrative database. Qualitative and quantitative data were integrated using coincidence analysis, and used to identify combinations of conditions that uniquely differentiated agencies with higher and lower fidelity to START.

Results

Fidelity was lower for intervention components requiring cross-system collaboration. Although key informants acknowledged the importance of collaboration for START implementation, few agencies used formal collaboration strategies other than staff co-location or reported high communication quality between frontline staff in child welfare and behavioral health. In coincidence analysis, four conditions differentiated agencies with higher and lower fidelity with 100% consistency and 88% coverage. We found that either strong leadership support or, in high need communities, third-party resource support from local behavioral health boards were sufficient for high fidelity. Similarly, in high need communities, absence of third-party resource support was sufficient for low fidelity, while in low need communities, absence of communication quality was sufficient for low fidelity.

Conclusion

Administrators, frontline workers, and interested third parties (i.e., other stakeholders not directly involved in implementation) can use collaboration strategies to facilitate implementation. However, the effectiveness of collaboration strategies depends on local context. In agencies where internal leadership support for implementation is low but need for intervention is high, third-party resource support may still be sufficient for high fidelity. Further research is needed to test effectiveness of collaboration strategies in different conditions and on a broader range of process and implementation outcomes.

Trials registration

ClinicalTrials.gov, NCT03931005, Registered 04/29/2019, https://classic.clinicaltrials.gov/ct2/show/NCT03931005.

Peer Review reports

Background

Cross-system interventions that integrate services or activities across different organizations or sectors of care can improve client outcomes and enhance community impact [1,2,3]. However, these interventions can be difficult to implement and sustain due to the need to coordinate activities across stakeholders that may have differing missions, professional roles, capacity, or modes for distributing resources [4, 5]. Collaboration strategies refer to policies and practices used to align operations and services across organizations or systems in the implementation of evidence-based, cross-system interventions [6]. These strategies can take different forms (e.g., data sharing agreements, joint service or staffing arrangements), and may be deployed by different stakeholders, from frontline staff to organizational leaders or third-party brokers.

Despite the importance of strong collaboration for successful implementation and sustainment of evidence-based, cross-system interventions [7, 8], little is known about the use and impact of collaboration strategies in implementation or whether certain strategies may be more effective than others. The current study seeks to address this gap by examining collaboration strategies and the conditions under which they impacted the implementation of an evidence-based intervention, Sobriety Treatment and Recovery Teams (START). START is a cross-system intervention focused on integrating child welfare and behavioral health systems for families involved with child welfare due to parental substance use [9, 10].

Importance of cross-system collaboration for families involved with child welfare due to parental substance use

In 2022, U.S. child welfare agencies received more than 4.2 million referrals to investigate alleged maltreatment of more than 7.5 million children [11]. Child maltreatment is a serious public health concern. Children who have experienced maltreatment are at increased risk for a number of adverse developmental, health, and mental health outcomes, including learning problems, problems relating to peers, depression, and higher risk of engaging in high-risk or negative health behaviors in adolescence and adulthood [12,13,14].

Alcohol and/or drug misuse are among the most prevalent caregiver risk factors for maltreatment [15], particularly for children three years or younger [16]. Caregiver substance use disorder (SUD) has also been associated with increased complexity and severity of child maltreatment [17, 18]. SUD treatment can support improved outcomes for families, particularly if services are initiated soon after initial involvement with the child welfare system and coordinated with child welfare services [19,20,21]. However, many caregivers struggle to engage in treatment within child welfare permanency timelines mandated by the 1997 United States Adoption and Safe Families Act [22].

Limited child welfare caseworker knowledge of substance use disorders or treatment resources can delay referrals to treatment [23, 24]. Following referral, limited availability of treatment programs that address ancillary service needs of caregivers involved with the child welfare system can delay access to treatment [25,26,27]. These challenges are often further exacerbated by differing priorities, perspectives, and information-sharing processes of the child welfare, court, and substance use treatment systems with which families are involved [28,29,30].

Effective cross-systems collaboration between child welfare and substance use treatment systems can improve caregiver access and engagement in treatment and increase likelihood of reunification [31,32,33], but is challenging to achieve, particularly in the absence of institutionalized, multilevel supports for collaboration [5, 28]. Identifying effective collaboration strategies is essential for successful implementation of cross-system interventions. However, cross-system collaboration strategies remain under-specified, and limited information exists regarding their effectiveness under different conditions.

Sobriety Treatment and Recovery Teams (START)

START is an evidence-based, cross-systems intervention for families with co-occurring child maltreatment and substance use. The intervention includes several sequenced practice components delivered across child welfare agency and substance use treatment systems, including screening caregivers for substance use disorders, shared decision-making meetings to plan services, wraparound support from a family peer mentor with lived experience in recovery and child welfare, and referral to a behavioral health treatment provider for assessment and at least four treatment sessions within 38 days of entering the child welfare system. A brief overview of key components of the model is provided in Fig. 1. When implemented as intended, START can expedite caregiver access and engagement in substance use treatment, increase likelihood of family reunification, and reduce subsequent maltreatment risk [9, 10, 34, 35]. However, successful implementation is contingent on a variety of contextual and organizational factors, including quality of collaboration between the child welfare and behavioral health systems [36].

Fig. 1
figure 1

Sobriety Treatment and Recovery Teams (START) timeline. UNCOPE is a validated, brief screening instrument for substance abuse treatment (Proctor & Hoffmann, 2016). SUD/MH = Substance use and mental health. SUD screening is recommended but not required by the national START model, and was voluntarily adopted by Ohio START

Full size image

Conceptual model

Our study of collaboration strategies influencing START implementation is anchored by a conceptual model informed by the cross-sector alignment framework [37] and the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework (see Fig. 2 and Appendix 1 for more information). In our model, collaboration strategies are defined as implementation policies and practices used to align operations and services across organizations [38]. These strategies can take multiple forms, may be implemented by different stakeholders (e.g., administrators, frontline staff, third parties), and their effectiveness may depend on the local system and organizational context in which START is being implemented.

Fig. 2
figure 2

Conceptual framework of factors influencing START implementation outcomes. * In Ohio, local behavioral health boards are responsible for coordinating publicly funded behavioral health prevention, treatment, and recovery supports and could thus also help facilitate child welfare and substance abuse treatment system alignment efforts

Full size image

Given the growing number of evidence-based interventions with cross-systems service integration components, there is a need to better understand which collaboration strategies are most effective at improving systems alignment in different contexts. Informed by this conceptual model, we draw on data from the Ohio START initiative to identify collaboration strategies and other contextual and organizational factors associated with intervention fidelity. Fidelity refers to the extent to which an intervention is implemented as intended by the intervention developer [39], and strongly influences whether evidence-based interventions yield expected outcomes when implemented in new settings [40, 41].

Methods

Study design

We used a convergent mixed methods approach, in which qualitative and quantitative data from the Ohio START evaluation were independently collected and then integrated during analyses [42]. First, we collected and analyzed data from qualitative key informant interviews and child welfare agency contracts with substance use treatment providers to identify collaboration strategies used to facilitate START implementation. We also descriptively analyzed administrative data, frontline worker surveys and other secondary data to obtain information on fidelity and on other contextual and organizational factors hypothesized to influence fidelity. We then integrated these data in analyses using a configurational comparative method known as coincidence analysis [43]. This study is reported using the Good Reporting of a Mixed Methods Study (GRAMMS) checklist [42].

Study context and sample

In 2017, Ohio began implementing START under the leadership of the Public Children Services Association of Ohio (PCSAO), a nonprofit organization that represents Ohio’s 85 county-based public child welfare agencies. In Ohio, child welfare services are state supervised but county administered, meaning individual counties have considerable discretion in how services are operated and delivered; thus, our unit of analysis was the county child welfare agency. Because START implementation was staggered, the current study draws on data from 17 of 30 child welfare agencies in the first two cohorts that implemented START (Cohort 1 n = 9; Cohort 2 n = 8). These 17 agencies were purposively selected to maximize variation in local system context (e.g., geographic diversity, agency funding, and governance structure), and agreed to participate in a mixed-methods evaluation of START implementation. Most agencies (n = 11; 64.7%) were in rural counties, 2 were suburban, and 4 were urban; 6 were in the Appalachian region.

Data sources

Data used in this study include key informant interviews, child welfare agency contracts with substance use treatment providers, frontline worker surveys, secondary data on local system context, and administrative data on START fidelity.

Key informant interviews. Between January 2020 and April 2021, we conducted 48 small group interviews with 104 unique participants involved in START implementation with the 17 child welfare agencies. This included 17 interviews with 52 child welfare staff (leaders, supervisors, and frontline workers), 25 interviews with 44 behavioral health treatment providers, and 6 interviews with behavioral health board representatives responsible for planning, evaluating, funding, and coordinating local mental health and substance use treatment services. Additional information about interview procedures is available in Appendix 2 and elsewhere [38].

With respondents’ permission, all interviews were recorded and professionally transcribed; team members also completed debrief summaries after each interview to highlight key themes and guide initial codebook development. All transcripts were coded by two investigators in the qualitative software NVivo 12.0 using template analysis, in which an initial codebook reflecting key content domains of interest was refined to incorporate emergent themes [44]. Any discrepancies were discussed until consensus was reached. Coded data were reviewed with a 16-member expert panel to specify and validate collaboration strategies; additional information about this process is available elsewhere [6]. Data were subsequently re-analyzed to determine the presence or absence of collaboration strategies within each agency and determine perceived quality of collaboration between child welfare and behavioral health.

Child welfare agency contracts with substance use treatment providers. We collected all child welfare agency contracts executed with behavioral health providers and pertaining to START between 2018 and 2020. When properly specified and enforced, contracts can support inter-agency collaboration by providing clarity and direction for resource allocation, accountability, shared decision-making, and coordination [45, 46]. In the current study, START-related contracts in each agency were reviewed and coded to obtain information on presence and type of formal agreement between child welfare agencies and behavioral health providers, provider roles and responsibilities, performance expectations, data sharing expectations, and specificity of contract terms.

Frontline worker surveys. We surveyed all child welfare caseworkers serving START families in fall 2020, approximately 2–3 years after initial START adoption. Surveys were conducted as part of the larger, statewide evaluation of START and assessed behavioral health referral patterns, quality of collaboration, implementation climate, and implementation leadership. Questions about behavioral health referral patterns were used to identify behavioral health partners and the average % of cases referred for services within county as opposed to outside of county. Collaboration quality was assessed using the Wilder Collaboration Factors Inventory, which is comprised of 47 items across six categories of factors shown to affect the success of collaboration endeavors. Implementation climate was assessed using the 6-item Implementation Climate Measure [47], and implementation leadership was assessed using the 12-item Implementation Leadership Scale [48]. Survey questions were reviewed by community partners and internally pilot tested prior to administration.

Surveys were distributed to a total of 174 workers in the 17 participating agencies. Survey participation was voluntary, and informed consent was obtained prior to administering surveys. A total of 140 workers completed the survey (between 3 and 34 per agency; overall response rate 80.5%). Survey measures were aggregated to the child welfare agency level to calculate a mean score for each participating agency.

Secondary data on local system context. Data reflecting local system context were drawn from multiple sources, including 2018 data on county child welfare agency structure and statistics from the Ohio Department of Job and Family Services, behavioral health board structure from the Ohio Association of County Behavioral Health Authorities, opioid-related overdose deaths from the Ohio Department of Health, naloxone administration rates from the Ohio Department of Public Safety, suspected drug overdoses among emergency department (ED) visits of Ohio residents aged 11 years and older from the Ohio Department of Health Violence and Injury Prevention, rates of neonatal abstinence syndrome (NAS) among resident live births from the Ohio Hospital Association and the Ohio Public Health Information Warehouse, numbers of children in child welfare custody (per 1000), screened in child maltreatment reports, and cases opened from the PCSAO Factbook, and county sociodemographic characteristics from the U.S. Census Bureau.

Fidelity data. Fidelity data were collected using the Needs Portal, a web-based management information system used by child welfare caseworkers and participating providers to track and share client-level information in real time, e.g., about assessments, service referrals, treatment modalities, treatment service dates, or other contacts with clients [49, 50]. As part of standard case reporting procedures, START team members prospectively entered information about whether and when each component of START was delivered. In the current study, data from March 2019 to October 2022 were cleaned and aggregated to the child welfare agency level to differentiate agencies with higher vs. lower START fidelity.

Outcome measure

Our dependent variable was fidelity, or the degree to which START was implemented as intended. Consistent with Carroll et al. (2007)’s fidelity framework [39], our measure of fidelity focused on adherence to core elements of the START model. In Ohio, START is comprised of multiple, sequenced components expected to occur within specified time periods (see Fig. 1), including (a) completion of a brief, substance use screen (UNCOPE) within 30 days of the child welfare case opening [51]; (b) first shared decision-making meeting (SDMM), in which child welfare, behavioral health, other service professionals, and families come together to plan services, within 4 days of referral to START; (c) first contact with a family peer mentor within 7 days of the SDMM; (d) comprehensive behavioral health assessment within four days of the SDMM; (e) first substance use treatment session within four days of assessment; and (f) completion of four substance use treatment sessions within 12 days of assessment. These components were identified as critical for fidelity in prior studies of the START model [9] and were also prioritized by our community partners and by START developers in assessing adherence to START.

In measuring fidelity, we focused on coverage based on number of model components delivered and duration based on whether the components were delivered in a timely way. We first used data from the Needs Portal to examine the percentage of cases that received each intervention component, the timeliness of service receipt, and variation across all county child welfare agencies in the state (see Table 1). Agencies were considered high fidelity if they were above statewide averages in the percentage of cases receiving each intervention component and in timeliness with which services were received. Based on feedback from our state partner (PCSAO), we revised our fidelity measure to omit component (f) (completion of four substance use treatment sessions within 12 days of assessment); this fidelity indicator was perceived as less within child welfare or START partnership control and as needing to be modeled separately.

Table 1 Ohio START Fidelity
Full size table

Coincidence analysis

In the analytic phase, a configurational comparative method (CCM) known as coincidence analysis was used to integrate our qualitative and quantitative data to identify collaboration strategies and other factors that differentiated agencies with high vs. low START fidelity. Additional information about CCMs is provided in Appendix 4.

In applying CNA, we first separately analyzed data from key informant interviews, child welfare agency contracts, frontline worker surveys, and secondary data to develop an initial list of 61 potential explanatory factors (see Appendix 3); of these, 28 were measures of collaboration strategies identified in Fig. 2 and the remaining measures reflected organizational and contextual factors hypothesized in our conceptual model as influencing fidelity. All explanatory factors were calibrated for use in CNA, meaning they were recoded as binary (yes/no) or categorical variables (max value range 1–5). Data were then entered into an Excel spreadsheet, in which rows represented each agency and columns reflected specific variables of interest.

The next step was to reduce the number of conditions in our data. We did this by applying the “minimally sufficient conditions” (“msc”) function within the R coincidence analysis (“cna”) package [43] to identify configurations of conditions with particularly strong connections to high or low fidelity. We considered all one-, two-, and three-condition configurations that met pre-designated thresholds for consistency and coverage. Consistency indicates how often a solution results in the outcome when present, and coverage indicates empirical prevalence of the solution within the sample (i.e., number of cases with configuration and outcome present divided by the total number of cases with the outcome). We started with a consistency threshold of 75%, and assessed all configurations that satisfied coverage thresholds of 75%, 80%, 85%, 90%, and 95% to compare output at different thresholds.

We generated a condition table listing all configurations of conditions that met these thresholds. Review of this table resulted in identification of smaller subsets of conditions to model. During this phase, we also used mathematical output from the -msc- function and subject matter knowledge to further reduced the dimensionality of selected factors, e.g., dichotomizing certain variables rather than preserving their original, categorical format. Models were developed iteratively using appropriate functions within the R “cna” package, based on our assessment of overall consistency and coverage of identified solutions and potential model ambiguity (i.e., when competing models explain the outcome equally well based on their consistency and coverage scores). We prioritized configurations with high consistency, lower complexity (i.e., fewer conditions), higher coverage scores (i.e., % cases for which the outcome is explained by the solution), and for which different values of the same conditions explained both high and low fidelity. We also assessed models for potential model ambiguity (when competing models explain the outcome equally well, based on consistency and coverage scores) [52].

To help mitigate risk of overfitting the data (i.e., including causally irrelevant factors), final model selection was based on fit-robustness (FR) scores, followed by highest consistency, highest coverage, and lowest complexity. Table 2 provides a description of conditions included in our final coincidence analysis model. All analyses were conducted using the R “cna” package, R Studio, and Excel.

Table 2 Explanatory conditions included in final coincidence analysis model
Full size table

Results

Fidelity

We rated each county child welfare agency in our sample using our measure of fidelity. Of the 17 agencies in our study sample, 6 agencies were classified as high fidelity, i.e., above statewide average in the percentage of cases receiving each intervention component and the timeliness with which services were received; the remaining 11 agencies were classified as low fidelity.

Collaboration strategies

Participating agencies varied in type and usage of different collaboration strategies for aligning child welfare and behavioral health providers for START implementation (Table 3). At the administrative level, the most commonly used collaboration strategy was to co-locate a family peer mentor employed by a behavioral health provider with child welfare caseworkers (71%; 12 of 17) and the least commonly used strategy was to jointly fund a program or position (0%). Most agencies (59%) chose to jointly supervise family peer mentors with behavioral health provider partners, and most (59%) also had START-specific contracts in place; however, few agencies (11%) had high-quality contracts in place (i.e., contracts with clear goals and performance metrics). In qualitative interviews, participants varied in their perceptions of contracts, with some describing the contracts as valuable for setting clear expectations and providing role clarity and others perceiving them as unnecessary “as long as they talk to us, as long as there is communication.”

Table 3 Collaboration strategies used in START implementation (n = 17)a
Full size table

At the frontline level, use of formal strategies for promoting system alignment was relatively rare, with most child welfare caseworkers and behavioral health providers relying primarily on informal relationships and information exchange. For example, while child welfare caseworkers and behavioral health providers in most agencies (65%; 11 of 17) reported consistent exchange of needed information about clients and services, only 35% did so via formally established channels or procedures, and only 35% reported regular behavioral health provider participation in SDMM. In qualitative interviews, participants indicated that lack of time and/or resources to pay for therapists’ time were barriers to behavioral health therapist participation in SDMM. As one child welfare agency leader noted, “Even when we tried conference call or scheduling it [SDMM] at clients’ typically scheduled session time, it wasn’t well received [by therapists]… because it’s not a Medicaid billable hour… or at least not billable at a clinical or case management rate.”

Analyses of aggregated frontline worker survey data revealed that the quality of communication between child welfare and behavioral health was rated highly in less than a quarter (24%; 4 of 17) of agencies. While many participants reported improvements in communication following START implementation, particularly at the administrative level, there were only four agencies in which child welfare caseworkers, family peer mentors, and behavioral health providers all reported high quality, bidirectional communication at the frontlines of care. As a behavioral health director working with one of these agencies noted, “We have a great relationship with Children’s Services… We are there to serve their clients, they’re there to help us do that, communication is excellent, whatever I need from them they’re usually very responsive and same back to them… We have weekly updates on every client that we serve.” Participants from most agencies were more likely to identify gaps in communication and ways in which they wished communication could be improved. As another behavioral health supervisor working with a different agency noted, “I wish there was a better working relationship… versus working opposed to each other… A lot of our clinicians and therapists hear one side of things from clients and feel like their clients aren’t being treated fairly and when they have reached out to advocate, they’re not heard because phone calls don’t get returned… We have tried to provide brief reports to Children’s Services to help them [but] struggled with them doing the same thing, letting us know what’s going on.”

When asked about other stakeholder engagement systems alignment efforts, most agencies (71%; 12 of 17) reported at least some engagement from behavioral health boards in START, though less than a quarter (24%; 4 of 17) described regular, START-specific engagement. Instead, most agencies described behavioral health boards as either not involved or engaged only sporadically or “as needed.” Key informants in over half of agencies (53%; 9 of 17) also described presence of other formal mechanisms for facilitating collaboration between child welfare and behavioral health providers in their communities, such as community coalitions or a family dependency treatment court.

When asked about specific strategies used to facilitate system alignment, key informants most frequently described inclusion of behavioral health board representatives on START steering committees (53%; 9 of 17), assistance from boards in identifying high-quality behavioral health providers in the community to contract with for services (41%; 7 of 17), or direct provision of resources to support START program implementation or fund treatment for clients (24%; 4 of 17). In qualitative interviews, participants noted that behavioral health board engagement in START varied depending on factors such as the number of counties a board oversaw and board members’ prior relationship with child welfare. Board engagement was higher in communities where child welfare agency leaders had previously collaborated with board members through joint participation in other community coalitions, advisory councils, or initiatives. As one highly engaged behavioral health board director noted, “We have a lot of history working with child welfare [in this county] on a wraparound program through the Family & Children First Council [local collaborative for issues affecting children and families]… where our board, our local development disabilities board, and our child welfare board all contribute money to pooled funds to help families with multisystem high acuity needs, and that relationship goes back almost 30 years.” This level of engagement was a sharp contrast to the much more limited board engagement reported by other child welfare agencies, some of whom were unaware of their local behavioral health board members were or how to leverage boards to benefit START implementation in their communities: “We’ve had zero contact [with behavioral health board]. We haven’t reached out and neither have they… I don’t even know who the board members are, at least not in the last several years…”.

Coincidence analysis results

Using coincidence analysis, we identified several conditions whose presence or absence differentiated agencies with high vs. low fidelity (Table 4). Specifically, agencies with high START fidelity either had [1] high leadership support for START or [2] high community need for child welfare services and third party resource support from the behavioral health board. These conditions explained high fidelity in 5 of 6 agencies (83% coverage) with 100% consistency. Only one agency met criteria for both solution pathways (i.e., high leadership support and high community need for child welfare resources and resource support from the behavioral health board).

Table 4 Coincidence analysis results: factors that differentiate agencies with higher and lower START implementation
Full size table

By contrast, agencies with low START fidelity either had (1) low communication quality and low community need or (2) high community need but low leadership support and no resource support from the behavioral health board. These conditions explained low fidelity in 10 of 11 agencies (91% coverage) with 100% consistency. There was no overlap between agencies in these two solution pathways.

Discussion

Strong collaboration is critical for implementing interventions that integrate services across different organizations or systems [7, 8, 53]. However, there is limited research on collaboration strategies and the conditions under which these strategies are most effective at improving implementation. Consistent with prior research [1, 54], our study of Ohio START implementation confirmed the difficulty of maintaining fidelity when cross-sector collaboration is involved. In Ohio START, fidelity was higher for intervention components directly under child welfare agency control (e.g., brief SUD screens and SDMMs) and lower for components requiring collaboration with behavioral health providers (e.g., comprehensive behavioral health assessment and substance use treatment). These findings reinforce the importance of collaboration strategies, i.e., implementation policies and practices focused on aligning services or operations across disparate organizations or systems, for helping to strengthen linkages between inner and outer contexts when implementing cross-systems interventions [55].

In prior research, we identified seven different collaboration strategies used by stakeholders in implementing Ohio START [6]. In the current study, we found that while key informants in all participating agencies acknowledged the importance of cross-system collaboration for effective implementation, actual use of collaboration strategies for aligning child welfare and behavioral health was low, i.e., used by fewer than half of agencies. The most frequently used administrative strategy, co-location of peer mentors with child welfare caseworkers, was also the only one explicitly required as an implementation strategy by the national purveyor of the START model. At the frontlines, use of structured collaboration strategies such as a regular inclusion of behavioral health providers in SDMMs was relatively uncommon, with most frontline workers reporting reliance on informal relationships for sharing information. In qualitative interviews, we learned that behavioral health providers often did not participate in SDMMs because time spent in these meetings was not reimbursed, suggesting the need to carefully consider partner resource needs when developing budgets for implementation of cross-system interventions. Engagement of behavioral health boards as a third-party facilitator of collaboration (e.g., in brokering relationships or providing resources needed to support coordination efforts) was also not widespread, often due to limited county child welfare leadership awareness of board members or how they could be engaged to improve START implementation.

Perhaps because of the lack of formal supports for cross-sector collaboration, relationship quality and communication quality between child welfare and behavioral health was also relatively low in our sample. Evidence from other studies demonstrates how combining collaboration strategies at multiple levels (e.g., system, organizational, frontlines) of the system is important for effective service delivery [56,57,58]; the limited use of collaboration strategies might explain the fidelity gaps we observed in this analysis and suggests a need for supporting the development of strong multi-level collaborations for implementing Ohio START and other cross-system interventions.

Collaboration strategies affecting fidelity

Coincidence analysis results provided broader insight into specific collaboration strategies and factors in the local community and organizational context that differentiated agencies with high vs. low fidelity. Specifically, the presence or absence of four conditions collectively explained fidelity in the majority of agencies in our sample: resource support from a third-party broker (county or multi-county behavioral health board), communication quality, leadership support, and severity of community need for child welfare services.

Consistent with prior literature on the importance of implementation leadership [59], we found that high child welfare leadership support for START was sufficient for high fidelity, regardless of whether an agency used any formal, collaboration strategies for bridging child welfare and substance use treatment systems. In communities where need for child welfare services was high, we found that third-party resource support from local behavioral health boards was also sufficient for high fidelity even when child welfare leadership support for START was low. In these communities, resource support from local behavioral health boards (key stakeholders in the “outer context”) could have served as an important substitute for “inner context” leadership in driving implementation. This finding is consistent with prior research demonstrating that leadership in both outer and inner contexts can influence evidence-based practice implementation and sustainment [59, 60]. Further research is needed to better understand how community dynamics and collaboration strategies used to bridge disparate contexts may interact with outer and inner context leadership support to influence implementation.

We also found that while presence of high quality communication was not by itself sufficient for high fidelity, the absence of high quality communication explained low fidelity in communities where need for child welfare services was lower. These findings reinforce prior research suggesting that high quality communication between frontline staff may be necessary but not sufficient for fidelity in the absence of other determinants of implementation [61].

Limitations

Several limitations must be taken into consideration in interpreting study results. First, while our study included rich qualitative and quantitative data on a broad array of collaboration strategies and factors that could have influenced START fidelity, we did not include any data from clients. Client perspectives on extent to which services were client-centered, integrated, or responsive to their needs could provide important insights and should be addressed in future research. We also did not assess quality of intervention delivery, a moderator of fidelity that should be addressed in future research.

In addition, while coverage of the final solution model was high, two agencies were not included in any of the identified solution pathways, indicating a possible role for additional factors beyond those included in the model. Review of available data on these two agencies (1 high fidelity agency and 1 low fidelity agency) reveal that these agencies were similar in three of the four conditions included in the final solution model (no resource support from the board, low leadership support, high perceived quality of frontline communication), but differed in community context. Specifically, the high fidelity agency was located in a large, urban community in which need for child welfare services was high (51-74th percentile) but not in the highest quartile in the state, while the low fidelity agency was located in a smaller, Appalachian county where need for child welfare services was in the lowest quartile (< 25th percentile) in the state. Future research involving more nuanced measures of community context could explore in more depth the potential role of context on implementation outcomes.

Finally, our analyses examined whether collaboration strategies differentiated agencies with high vs. low fidelity, but did not assess their impact on perceived leadership support, relationship quality, communication quality, or other determinants of implementation. For example, it is possible that the impacts of some collaboration strategies on fidelity (e.g., data sharing agreements, staff co-location) were fully mediated by their impact on perceived leadership support or communication quality, two conditions that did emerge as meaningful in our analyses. Future research could more directly assess impact of different collaboration strategies on different determinants of implementation, rather than on more distal implementation outcomes.

Implications for research, policy, and practice

Despite the limitations noted above, our study is the first to present and empirically test a conceptual model of different collaboration strategies hypothesized to affect implementation of evidence-based, cross-system interventions. Key study strengths include the inclusion of multilevel collaboration strategies, the collection and use of mixed methods data, and the application of an innovative method (coincidence analysis) to integrate these data in analyses. Study findings confirm that collaboration strategies are important for intervention fidelity and highlight the importance of considering equifinality in implementation research. To further refine and specify our conceptual model, additional research on collaboration strategies and implementation outcomes in different contexts is needed.

Our study also provides preliminary insights into how policymakers and practitioners can improve uptake of identified collaboration strategies. One particularly notable finding in our study was that local leaders often did not use identified collaboration strategies – particularly administrative and third-party facilitator strategies – simply because they were not aware of these strategies or how they might benefit START implementation. To address this gap, our study team used findings to develop a practitioner-oriented collaboration strategies toolkit, which we validated and distributed to community partners and Ohio START technical assistance providers. Alternative, evidence-informed approaches for raising awareness of collaboration strategies could include creating learning collaboratives or other opportunities for implementers from different agencies and community contexts to share strategies for addressing shared community needs and learn from one another. Future research could also empirically test effectiveness of these approaches at improving uptake of different collaboration strategies.

Conclusions

Cross-system interventions, particularly those that integrate health, behavioral health, and social services, show promise for improving client outcomes and enhancing community impact if barriers to inter-agency collaboration can be overcome. This study identified four conditions that differentiated agencies with high vs. low START fidelity. Our findings emphasize the importance of collaboration strategies and of local community and organizational context to implementation of complex, cross-system interventions; in particular, findings suggest that while leadership support matters for implementation, resource support from interested third parties not directly involved in implementation can improve intervention fidelity in communities where need for intervention is high but internal leadership support is low.

Data availability

De-identified data will be made available from the first author upon reasonable request.

Abbreviations

SDMM:

Structured decision-making meetings

START:

Sobriety Treatment and Recovery Teams

SUD:

Substance use disorder

UNCOPE:

Used, Neglected, Cut Down, Objected Preoccupied, Emotional Discomfort

References

  1. Bunger A, Yoon S, Maguire-Jack K, Phillips R, West K, Clark-Hammond G, et al. Implementation and mental health outcomes of a service cascade linking child welfare and children’s mental health systems: a case study of the Gateway CALL demonstration. Adm Policy Mental Health Mental Health Serv Res. 2023;50:327–41.

    Article  Google Scholar 

  2. Mays G, Mamaril C, Timsina L. Preventable death rates fell where communities expanded population health activities through multisector networks. Health Aff. 2016;35(11):2005–13.

    Article  Google Scholar 

  3. Baxter S, Johnson M, Chambers D, Sutton A, Goyder E, Booth A. The effects of integrated care: a systematic review of UK and international evidence. BMC Health Serv Res. 2018;18:350.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Cooper M, Evans Y, Pybis J. Interagency collaboration in children and young people’s mental health: a systematic review of outcomes, facilitating factors, and inhibiting factors. Child: Care Health Dev. 2016;42(3):325–42.

    Article  PubMed  CAS  Google Scholar 

  5. Drabble L. Pathways to collaboration: exploring values and collaborative practice between child welfare and substance abuse treatment fields. Child Maltreat. 2007;12(1):31–42.

    Article  PubMed  Google Scholar 

  6. Bunger A, Chuang E, Girth A, Lancaster K, Smith R, Phillips R, et al. Specifying cross-system collaboration strategies for implementation: a multi-site qualitative study with child welfare and behavioral health organizations. Implement Sci. 2024;19:13.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Mattessich P, Rausch E. Cross-sector collaboration to improve community health: a view of the current landscape. Health Aff. 2014;33:1968–74.

    Article  Google Scholar 

  8. Towe V, Leviton L, Chandra A, Sloan J, Tait M, Orleans T. Cross-sector collaborations and partnerships: essential ingredients to shape health and well-being. Health Aff. 2016;35:1964–9.

    Article  Google Scholar 

  9. Hall M, Huebner R, Sears J, Posze L, Willauer T, Oliver J. Sobriety treatment and recovery teams in rural Appalachia: implementation and outcomes. Child Welfare. 2015;94(4):119–38.

    PubMed  Google Scholar 

  10. Hall M, Kelmel A, Huebner R, Walton M, Barbee A. Sobriety treatment and recovery teams for families with co-occurring substance use and child maltreatment: a randomized controlled trial. Child Abuse Negl. 2021;114:104963.

    Article  PubMed  Google Scholar 

  11. U.S. Department of Health & Human Services, Administration for Children an Families, Administration on Children, Youth, ad Families, Children's Bureau. Child Maltreatment 2022. 2024. Available from https://www.acf.hhs.gov/cb/data-research/child-maltreatent.

  12. Petersen A, Joseph J, Feit M. New directions in child abuse and neglect research: consequences of child abuse and neglect. Washington DC: National Academies; 2014.

    Google Scholar 

  13. Richmond-Crum M, Joyner C, Fogerty S, Ellis M, Saul J. Applying a public health approach: the role of state health departments in preventing maltreatment and fatalities of children. Child Welfare. 2013;92(2):99–117.

    PubMed  PubMed Central  Google Scholar 

  14. Smith V, Wilson CR, Ryan SA, Gonzalez P, Patrick SW, Quigley J, et al. Families affected by parental substance use. Pediatrics. 2016;138(2):e20161575.

    Article  PubMed  Google Scholar 

  15. U.S. Department of Health & Human Services, Administration for Children and Families, Administration on Children, Youth, and Families, Children's Bureau. Child Maltreatment 2020. 2022. Available from https://www.acf.hhs.gov/cb/data-research/child-maltreatment.

  16. Patrick S, Frank R, McNeer E, Stein B. Improving the child welfare system to respond to the needs of substance-exposed infants. Hosp Pediatr. 2019;9(8):651–4.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Ghertner R, Waters A, Radel L, Crouse G. The role of substance use in child welfare caseloads. Child Youth Serv Rev. 2018;90:83–93.

    Article  Google Scholar 

  18. Feder K, Letourneau E, Brook J. Children in the opioid epidemic: addressing the next generation’s public health crisis. Pediatrics. 2019;143(1):e20181656.

    Article  PubMed  Google Scholar 

  19. Green B, Rockhill A, Furrer C. Does substance abuse treatment make a difference for child welfare outcomes? A statewide longitudinal analysis. Child Youth Serv Rev. 2007;29(4):460–73.

    Article  Google Scholar 

  20. Grella C, Needell B, Shi Y, Hser Y. Do drug treatment services predict reunification outcomes of mothers and their children in child welfare? J Subst Abuse Treat. 2009;36(3):278–93.

    Article  PubMed  Google Scholar 

  21. Osterling K, Austin M. Substance abuse interventions for parents involved in the child welfare system: evidence and implications. J Evid Based Soc Work. 2008;5(1–2):157–89.

    Article  PubMed  Google Scholar 

  22. Choi S, Ryan J. Completing substance abuse treatment in child welfare: the role of co-occurring problems and primary drug of choice. Child Maltreat. 2006;11(4):301–89.

    Article  Google Scholar 

  23. Chuang E, Wells R, Bellettiere J, Cross T. Identifying substance abuse treatment needs of caregivers involved with child welfare. J Subst Abuse Treat. 2013;45(1):118–25.

    Article  PubMed  PubMed Central  Google Scholar 

  24. He A, Phillips J. Interagency collaboration: strengthening substance abuse resources in child welfare. Child Abuse Negl. 2017;64:101–8.

    Article  PubMed  Google Scholar 

  25. Patrick S, Schiff D, Ryan S, Quigley J, Gonzalez P, Walker L. A public health response to opioid use in pregnancy. Pediatrics. 2017;139(3):e20164070.

    Article  PubMed  Google Scholar 

  26. Radel L, Baldwin M, Crouse G, Ghertner R, Waters A. Substance use, the opioid epidemic, and the child welfare system: key findings from a mixed methods study. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation; 2018.

    Google Scholar 

  27. Radel L, Baldwin M, Crouse G, Ghertner R, Waters A. Medication-assisted treatment for opioid use disorder in the child welfare context: challenges and opportunities. Washington DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation; 2018.

    Google Scholar 

  28. Green B, Rockhill A, Burrus S. The role of interagency collaboration for substance-abusing families involved with child welfare. Child Welfare. 2008;87(1):29–62.

    PubMed  Google Scholar 

  29. Young N, Gardner S, Dennis K. Responding to alcohol and other drug problems in child welfare: weaving together practice and policy. Washington DC: Child Welfare League of America; 1998.

    Google Scholar 

  30. Drabble L. Advancing collaborative practice between substance abuse treatment and child welfare fields: what helps and hinders the process? Adm Social Work. 2011;35(1):88–106.

    Article  Google Scholar 

  31. He A. Interagency collaboration and receipt of substance abuse treatment services for child welfare-involved caregivers. J Subst Abuse Treat. 2017;79:20–8.

    Article  PubMed  Google Scholar 

  32. Ogonnaya I, Keeney A. A systematic review of the effectiveness of interagency and cross-systems collaborations in the United States to improve child welfare outcomes. Child Youth Serv Rev. 2018;94:225–45.

    Article  Google Scholar 

  33. Marsh J, Smith B. Integrated substance abuse and child welfare services for women: a progress review. Child Youth Serv Rev. 2011;33(3):466–72.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Hall M, Wilfong J, Huebner R, Posze L, Willauer T. Medication-assisted treatment improves child permanency outcomes for opioid-using families in the child welfare system. J Subst Abuse Treat. 2016;71:63–7.

    Article  PubMed  Google Scholar 

  35. Huebner R, Willauer T, Posze L. The impact of Sobriety Treatment and Recovery Teams (START) on family outcomes. Families Society: J Contemp Social Serv. 2012;93(3):196–203.

    Article  Google Scholar 

  36. Huebner R, Young N, Hall M, Posze L, Willauer T. Serving families with child maltreatment and substance use disorders: a decade of learning. J Family Social Work. 2017;20:288–305.

    Article  Google Scholar 

  37. Landers G, Minyard K, Lanford D, Heishman H. A theory of change for aligning health care, public health, and social services in the time of COVID-19. Am J Public Health. 2020;110(2):S178-80.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Bunger A, Chuang E, Girth A, Lancaster K, Gadel F, Himmeger M, et al. Establishing cross-systems collaborations for implementation: protocol for a longitudinal mixed methods study. Implement Sci. 2020;15(55). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s13012-020-01016-9.

  39. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2: 40.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Hill H, Erickson A. Using implementation fidelity to aid in interpreting program impacts: a brief review. Educational Researcher. 2019;48(9):590–8.

    Article  Google Scholar 

  41. Breitenstein S, Gross D, Garvey C, Hill C, Fogg L, Resnick B. Implementation fidelity in communitybased interventions. Res Nurs Health. 2010;33(2):164–73.

    Article  PubMed  PubMed Central  Google Scholar 

  42. O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13:92–8.

    Article  PubMed  Google Scholar 

  43. Baumgartner M, Ambuhl M, Epple R, Parkinnen V, Thiem A. CNA: Causal modeling with coincidence analysis 2023. https://cran.r-project.org/package=cna

  44. King N. Doing template analysis. In: Cassell G, editor. Qualitative organizational methods: core methods and current challenges. Thousand Oaks, CA: Sage; 2012:426–50.

  45. Cheng Y. Governing government non-profit partnerships: linking governance mechanisms to collaboration stages. Public Perform Manage Rev. 2019;42(1):190–212.

    Article  Google Scholar 

  46. Peng S, Liao Y, Lu J. Stay or exit: why do nonprofits maintain collaborations with government? Am Rev Public Adm. 2020;50(1):18–32.

    Article  Google Scholar 

  47. Jacobs S, Weiner BJ, Bunger A. Context matters: measuring implementation climate among individuals and groups. Implement Sci. 2014;9:46.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Aarons GA, Ehrhart M, Farahnak L. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9:45.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Dellor E, Lovato-Hermann K, Wolf J, Curry S, Freisthler B. Introducing technology in child welfare referrals: a case study. J Technol Hum Serv. 2015;33(4):330–44.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Dellor E, Allbright-Campos M, Lee J, Bunger A, Gadel F, Freisthler B, Ohio START. An adaptation of the national sobriety treatment and recovery teams model. Child Youth Serv Rev. 2024;160:107575.

    Article  Google Scholar 

  51. Proctor S, Hoffmann N, The UNCOPE. An effective brief screen for DSM-5 substance use disorders in correctional settings. Psychol Addict Behav. 2016;30(5):613–8.

    Article  PubMed  Google Scholar 

  52. Baumgartner M, Thiem A. Model ambiguities in configurational comparative research. Sociol Methods Res. 2017;46(4):954–87.

    Article  Google Scholar 

  53. Fichtenberg C, Delva J, Minyard K, Gottlieb L. Health and human services integration: generating sustained health and equity improvements. Health Aff. 2020;39:567–73.

    Article  Google Scholar 

  54. Kay E, Batey D, Mugavero M. The HIV treatment cascade and care continuum: updates, goals, and recommendatiosn for the future. AIDS Res Therapy. 2016;13:35.

    Article  Google Scholar 

  55. Lengnick-Hall R, Stadnick N, Dickson K, Moullin J, Aarons GA. Forms and functions of bridging factors: specifying the dynamic links between inner and outer contexts during implementation and sustainment. Implement Sci. 2021;16:34.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Provan KG, Milward H. A preliminary theory of interorganizational network effectiveness: a comparative study of four community mental health systems. Adm Sci Q. 1995;40(1):1–33.

    Article  Google Scholar 

  57. Selden S, Sowa J, Sandfort J. The impact of nonprofit collaboration in early child care and education on management and program outcomes. Public Adm Rev. 2006;66:412–25.

    Article  Google Scholar 

  58. Singer S. A comprehensive theory of integration. Med Care Res Rev. 2020;77:196–207.

    Article  PubMed  Google Scholar 

  59. Aarons GA, Ehrhart M, Farahnak L, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annu Rev Public Health. 2014;35:255–74.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Aarons GA, Green AE, Trott E, Williging CE, Torres E, Ehrhart M, et al. The roles of system and organizational leadership in system-wide evidence-based intervention sustainment: a mixed-methods study. Adm Policy Mental Health Mental Health Serv Res. 2016;43:991–1008.

    Article  Google Scholar 

  61. Albright K, Navarro E, Jarad I, Boyd M, Powell B, Lewis C. Communication strategies to facilitate the implementation of new clinical practices: a qualitative study of community mental health therapists. Translational Behav Med. 2022;12(2):324–34.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the dedicated child welfare and behavioral health leaders and staff who took time to talk with us. We are also grateful to our partners in the Ohio Association of County Behavioral Health Authorities for sharing their data and/or insights on START implementation with us.

Funding

Research reported in this publication was supported by the National Institute on Drug Abuse (R34DA046913) and the Robert Wood Johnson Foundation Systems for Action program (ID 76434). BJP was supported by grants from the NIH, including R25MH080916, R01CA262325, P50CA19006, and P50DA054072. The content is solely the responsibility of the authors and does not represent the official views of the National Institute on Drug Abuse or the Robert Wood Johnson Foundation.

Author information

Authors and Affiliations

  1. University of California, Berkeley, Berkeley, CA, USA

    Emmeline Chuang

  2. Ohio State University, Columbus, OH, USA

    Alicia Bunger, Rebecca Smith, Amanda Girth & Elinam Dellor

  3. Western Oregon University, Monmouth, OR, USA

    Rebecca Phillips

  4. Indiana University School of Medicine, Bloomington, IN, USA

    Edward Miech

  5. Wake Forest University, Winston-Salem, NC, USA

    Kathryn Lancaster

  6. University of California, San Francisco, CA, USA

    Jared Martin

  7. Public Children Services Association of Ohio, Columbus, OH, USA

    Fawn Gadel, Marla Himmeger, Jen McClellan & Jennifer Millisor

  8. Children and Family Futures, Irvine, CA, USA

    Tina Willauer

  9. Washington University in St. Louis, St. Louis, MO, USA

    Byron J. Powell

  10. University of California San Diego, San Diego, CA, USA

    Gregory A. Aarons

Authors
  1. Emmeline Chuang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Alicia Bunger
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Rebecca Smith
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Amanda Girth
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Rebecca Phillips
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Edward Miech
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Kathryn Lancaster
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Jared Martin
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Fawn Gadel
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Marla Himmeger
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Jen McClellan
    View author publications

    You can also search for this author inPubMed Google Scholar

  12. Jennifer Millisor
    View author publications

    You can also search for this author inPubMed Google Scholar

  13. Tina Willauer
    View author publications

    You can also search for this author inPubMed Google Scholar

  14. Byron J. Powell
    View author publications

    You can also search for this author inPubMed Google Scholar

  15. Elinam Dellor
    View author publications

    You can also search for this author inPubMed Google Scholar

  16. Gregory A. Aarons
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

EC: Conceptualization, Formal analysis, Investigation, Methodology, Supervision, Writing – Original draft, Writing – Review & editing; AB: Conceptualization, Formal Analysis, Investigation, Funding acquisition, Writing – Review & editing; RP: Formal analysis, Investigation, Writing – Review & editing; AG: Formal analysis, Investigation, Writing – Review & editing; RS: Formal analysis, Investigation, Writing – Review & editing; EM: Methodology, Writing – Review & editing; KL: Formal analysis; Investigation; JM: Formal analysis, Investigation; FG: Writing – Review & editing; MH: Writing – Review & editing; JMc: Writing – Review & editing; JM: Writing – Review & editing; TW: Writing – Review & editing; BP: Writing – Review & editing; ED: Investigation, Writing – Review & editing; GA: Writing – Review & editing.

Corresponding author

Correspondence to Emmeline Chuang.

Ethics declarations

Ethics approval and consent to participate

All procedures were approved by the Ohio State University Institutional Review Board.

Consent for publication

Not applicable.

Competing interests

Dr Aarons is a member of the Implementation Science Communications Editorial Board. All decisions on this manuscript were made by another editor.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chuang, E., Bunger, A., Smith, R. et al. Collaboration strategies affecting implementation of a cross-systems intervention for child welfare and substance use treatment: a mixed methods analysis. Implement Sci Commun 5, 127 (2024). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s43058-024-00666-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s43058-024-00666-w

Keywords

Implementation Science Communications

ISSN: 2662-2211