Audit and feedback is an effective implementation strategy to increase fidelity to a multi-component labor induction protocol designed to reduce obstetric inequities

Abstract

Background

Studies have demonstrated that standardizing labor induction (IOL), often with the use of protocols, may reduce racial inequities in obstetrics. IOL protocols are complex, multi-component interventions. To target identified implementation barriers, audit and feedback (A&F) was selected as an implementation strategy. Here, we aimed to understand the acceptability and effect of A&F on fidelity to this complex intervention through quantitative and qualitative approaches.

Methods

This secondary analysis of a type I hybrid effectiveness-implementation trial (10/2018–12/2022) compared 2 years before (PRE) to 2 years after (POST) implementation of an IOL protocol at two sites. Fidelity to each of 8 specific protocol components was collected via chart review. During the POST period, unit-aggregated A&F reports were distributed via email every 3 months to site clinicians. Reports tracked fidelity to protocol components over time. For this analysis, we compared component fidelity PRE to POST-implementation. Additionally, during the POST period, we compared fidelity by month after each A&F (Month#1 v. Month#2/3) to evaluate the effect of A&F over time. Acceptability of A&F reports was evaluated using qualitative interviews.

Results

8509 labor inductions were included (PRE = 4214, POST = 4295). A&F reports were successfully distributed every 3 months for the 2-year POST period. PRE to POST-implementation, fidelity to 4 of the 8 components increased significantly (cervical Foley utilization, latent labor examination frequency, amniotomy timing, and intrauterine pressure catheter utilization), without change in the other 4 components. For 2 of those 4 components where improvement was noted, there was no difference in fidelity by month after A&F report; rather, there was sustained improvement across the POST-implementation period. On the other hand, for the remaining 2 components, fidelity peaked in the first month after each A&F report, with some decline in the following 2 months prior to the next A&F report. Qualitative analysis (n = 24) supported A&F acceptability, with A&F described as “motivating” and “helpful.”

Conclusions

A&F was an effective implementation strategy to promote fidelity to certain components of this labor induction protocol. With some decline in effect after the first month POST-A&F report, increased A&F frequency should be considered in future work targeting obstetric outcomes, as well as health inequities.

Contributions to the literature:

• Clinician audit and feedback is an effective implementation strategy to promote fidelity to a multi-component inpatient obstetric intervention overall.

• Protocol components that require a clinician to perform a single action may be most conducive to and sustainable with audit and feedback approaches, while components that require multiple actions over time for adherence may require increased frequency of audit and feedback or supplementary strategies.

• More controversial protocol components that rely on multiple individuals on the healthcare team, as well as patients, to agree upon and engage in the action, may be less conducive to clinician-facing audit and feedback strategies.

Background

Audit and feedback (A&F) is an implementation strategy for improving the use of evidence-based practices in healthcare [1]. Individuals or groups are assessed and compared either to each other or other target standards, then feedback is given. Prior work outside of maternal health has shown that A&F can, at least marginally, impact the success of implementation, with a wide range of success across studies and contexts [2,3,4,5]. There is a paucity of data around the impact of A&F for interventions designed to improve maternal health, particularly on a labor unit [6]. Moreover, how to best deliver A&F, including the ideal dosage for optimal implementation and sustainment, both outside of and within the maternal health context is unknown [3, 6, 7].

In addition to the lack of data around the impact of A&F on maternal health interventions, there is also minimal data utilizing A&F for interventions designed to improve health equity, a critical issue among the obstetric population [7, 8]. There are significant disparities between Black and non-Black women in the United States in birth outcomes [9,10,11]. Black women in the United States are twice as likely to experience a fetal mortality and nearly 4 times more likely to die themselves in and around pregnancy [10, 11]. In addition, Black women have higher cesarean delivery rates than non-Black women, even when accounting for sociodemographic and clinical differences [12]. While multiple studies have demonstrated racial disparities in obstetric outcomes, a limited number of interventions have successfully addressed them [10]. The utilization of protocols to standardize care has been shown to decrease adverse outcomes and disparities in various medical fields, including obstetrics [13,14,15]. The American College of Obstetricians and Gynecologists (ACOG) has led a national effort to establish protocols and standardize labor and delivery management [16].

Single-site, retrospective analyses by our group demonstrated that the standardization of labor induction practices may be of critical importance in tackling these disparities. Our work compared women enrolled in a randomized trial that utilized a standardized labor induction protocol to a concurrent observational cohort managed at clinician discretion. When stratified by race, the standardized labor induction protocol led to a 70% reduction in neonatal morbidity and a 35% reduction in cesarean delivery rate for Black women, thereby reducing the observed racial disparity [17, 18]. Thus, we undertook a two-site type I hybrid effectiveness-implementation trial to evaluate the effectiveness of instituting a standardized labor induction protocol on overall obstetric outcomes and racial disparities in obstetric morbidity, while simultaneously collecting implementation data of importance to wider implementation. The results of the primary analysis of this trial are reported elsewhere [19].

However, like many interventions, protocols for labor, such as the standardized labor induction protocol used in our preliminary work, are complex and multi-faceted [20]. In informal discussions with the healthcare team and site leaders, as well as formal mixed-methods work, one of the critical barriers to clinician protocol use was understanding the current state and progress of utilizing the intervention, due to the high volume of patients undergoing labor induction and lack of ability to determine how others on the unit were contributing to protocol adherence in the team-based structure of labor and delivery. In addition, clinicians reported issues with having the confidence that they would be able to change practice in the context of a busy and high acuity labor unit [21]. Control theory proposes that behavior is regulated by a negative feedback loop, in which an individual compares the perception of the current state against a goal state, and will strive to reduce perceived discrepancies by modifying behavior [22]. As clinicians would not otherwise have an objective means to assess their utilization of evidence-based practices for labor induction, in theory, A&F would provide clinicians with concrete data on unit progress and proof of practice change or lack thereof, thereby driving behavior change. A&F was thus selected by a team of multidisciplinary constituents as the primary implementation strategy for this labor induction protocol prior to initiating the type I hybrid trial.

In addition to the lack of data around A&F for maternal health interventions and disparities-focused interventions, there is also a paucity of data around A&F for promoting utilization of multi-component interventions in any field. Here, we aimed to understand the effect of A&F on fidelity to this intervention and whether that effect is sustained over time in a quantitative approach, while simultaneously assessing acceptability of A&F in a qualitative approach. Our overarching goal was to add to the generalizable knowledge around optimal delivery of A&F, while specifically understanding A&F in the context of fidelity to implementation of a complex maternal health equity focused intervention on labor units.

Methods

This is a secondary analysis of a type I hybrid effectiveness-implementation trial (10/2018–12/2022) which compared 2 years before (PRE) to 2 years after (POST) implementation of a standardized labor induction protocol at two sites [19]. Implementation of a labor induction protocol occurred in a stepped approach at two separate hospitals within the University of Pennsylvania Hospital System. Both sites are urban hospitals with busy obstetrical services – delivery volume at Site #1 is 4100/year, while at Site #2 is 4800/year. The project was approved by the University of Pennsylvania Institutional Review Board as quality improvement with a waiver of informed consent. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines were followed in the writing of this report [23].

The evidence-based practice: a standardized labor induction protocol

Labor induction is defined as the stimulation of labor contractions during pregnancy before labor begins spontaneously, with the goal of achieving a vaginal delivery. As the cervix needs to soften, thin, and open to prepare for delivery, cervical ripening methods, such as medications or devices, are utilized. Labor induction, which makes up 30% of all deliveries in the U.S. and accounts for almost 1.2 million U.S. women annually [24, 25], is associated with a cesarean delivery rate of around 25% nationally. Thus, labor induction results in approximately 250,000 yearly U.S. cesareans [26].

Protocols and guidelines that are established to standardize care should incorporate evidenced-based measures that improve outcomes [27, 28]. In randomized and observational trials, active labor management practices have been shown to shorten labor [29,30,31,32], reduce infectious morbidity [30, 31], and decrease cesarean delivery rates [30, 33] without increasing neonatal morbidity. Traditionally, active labor management protocols have been used for patients who present in spontaneous labor. However, critical components of active labor protocols, such as the regular assessment of labor progress, amniotomy at the onset of labor, and effective utilization of oxytocin, could also be valuable for patients undergoing labor induction and lead to similarly substantial improvements. The specific standardized labor induction protocol implemented at both sites in this work is shown in Supplemental Fig. 1.

Fig. 1

Example of unit-level audit and feedback report. Changes in fidelity were reported qualitatively, with either a green “smiley face” if raw improvement had been made, yellow “neutral face” if no changes was made, or a red “sad face” for worsening adherence for each component

Implementation of the protocol: a focus on audit and feedback as an implementation strategy

Across medicine, implementation research has demonstrated that training and education alone is an insufficient strategy to successfully incorporate an intervention into routine care [34]. Our implementation process involved several evidence-based implementation strategies guided by the Powell et al. 2012 [35] ERIC taxonomy of planning, education, restructuring, and quality management. In the preparation phase, multidisciplinary buy-in for the project had been obtained at both sites, and formal training sessions with all obstetric clinicians and nursing staff were held in the 3 months prior to implementation at each site. A&F was selected as the primary strategy for the implementation phase. Every 3 months throughout the POST period, individualized unit-level audit and feedback reports were sent to each of the two units regarding the use of and adherence to the protocol for qualifying patients.

While there are numerous components of the protocol (Supplemental Fig. 1), for the purposes of these A&F reports, we selected 8 components of the protocol that could be discretely evaluated in chart review. Ten percent of individual chart review data was checked for quality assurance by the PI. Fidelity, defined as the adherence to each of these 8 components (Fig. 1), was shown for the month prior to report distribution and compared to adherence rates from the previous report 3 months before. To harness prior data around the components of A&F that enhance success [8], instead of showing a statistical comparison among adherence rates, a qualitative assessment was used for comparison, in order to make the information more understandable and approachable for clinicians. This qualitative assessment focused on improving adherence to all components, regardless of whether baseline adherence rates for a given component were already high. Either a green “smiley face” if improvement had been made, yellow “neutral face” if no changes were made, or a red “sad face” for worsening adherence for each component was used to report results; a sample is shown in Fig. 1. Reports were compiled by the principal investigator (PI: RH) using a combination of automated data reports from the electronic health record with individual chart review performed by the research team. These reports were sent via email to all site clinicians caring for patients on labor and delivery (including physicians, nurse-midwives, nurses, and trainees) and shown at staff/faculty meetings by the PI, with selected points of advice on how to target adherence to the components with sub-par adherence.

Quantitative analysis

Demographic and clinical characteristics of the patient population were evaluated using descriptive statistics. While protocol fidelity comparisons presented to clinicians were entirely qualitative, for the secondary analysis presented here, we compared component fidelity PRE to POST for each of the 8 protocol components utilizing statistical comparisons. Additionally, during the POST period, if statistical improvement was seen in a specific component from PRE to POST, we compared fidelity by month after each A&F (Month #1 v. Month #2/3) to evaluate the effect of A&F on that component over time. For example, if an A&F report was distributed April 1, 2021, it would compare protocol fidelity between the months of March 2021 and December 2020. April 2021 would be considered Month #1, the month immediately after report distribution. May and June 2021 would be considered Months #2/3, as the next report would be distributed in July 2021. Specifically, such a comparison could evaluate whether each A&F report had an immediate impact and whether that impact lasted over Months #2/3.

All comparisons performed were categorical, comparing fidelity to a given protocol component by exposure group (either PRE versus POST-implementation or Month #1 versus Month #2/3 after an A&F report in the POST period). Thus, χ2 tests were used for all comparisons. We elected not to perform any regression modeling for these analyses, as any differences in patient characteristics should not impact fidelity to the standardized protocol, which is recommended regardless. Statistical analyses were performed with Stata, version 15 (StataCorp LLC). All tests were 2 tailed, and p < 0.05 was considered statistically significant.

Qualitative analysis

Embedded into the broader type I hybrid effectiveness-implementation trial was an explanatory sequential mixed-methods (QUAN- > QUAL) study [21], which is reported here utilizing the Consolidated Criteria for Reporting Qualitative Research guidelines [36]. The validated, 4-question Acceptability of Intervention Measure (AIM [total 4–20]) was administered to labor and delivery clinicians 6 months post-implementation at the 2 sites (Site 1: 3/2021; Site 2: 6/2021) [37]. Respondents were grouped by total score into tertiles. The top (“High” Acceptability) and bottom (“Low” Acceptability) tertiles were invited to participate in a 30-min semi-structured qualitative interview from 6/2021 to 10/2021 until thematic saturation was reached in each acceptability group. Participants were purposively sampled by role and site.

The Consolidated Framework for Implementation Research (CFIR) guided the creation of the interview guide (Supplemental File) [38]. While interview questions elicited several concepts around induction protocol implementation, for the purposes of this manuscript, we will report the qualitative results as they relate to the acceptability of A&F as an implementation strategy from the clinician perspective. General perspectives on protocol acceptability are reported elsewhere [21]. Questions were designed to be open ended.

Individual interviews were conducted in-person, via video conferencing software, or over the phone and lasted an average of 30 min. The principal investigator (RFH), an obstetrician trained in qualitative interviewing, conducted all interviews. Permission was obtained, and all interviews were recorded. Audio from the interviews was transcribed by Datagain Transcription Services (Secaucus, NJ). The transcripts were then uploaded to NVivo 12 software for management and coding. A research coordinator trained in qualitative methods used an inductive process of iterative coding to ascertain recurrent relationships, themes, and categories and to develop the codebook, which was reviewed by the principal investigator (RH). Then, we used an integrated analysis approach [39], identifying a priori attributes (CFIR constructs), as well as a modified content analysis approach. The research coordinator who developed the codebook, in addition to a second trained research personnel applied the codebook to the transcripts and periodically refined the themes and definitions based on inter-rater reliability tests to facilitate analysis. Twenty percent of transcripts were double-coded (k = 0.83). The research coordinators then synthesized the outputs of the coding and identified the key themes described in this manuscript.

Results

Quantitative results

A total of 8509 patients met inclusion criteria across the study period; 4214 in the PRE and 4295 in the POST-implementation groups. Demographic and clinical characteristics overall are detailed in Table 1. Our population was 44.6% Black and delivered at a median gestational age of 39 weeks and BMI of 31. The majority were nulliparous (65.1%).

Table 1 Demographic and clinical characteristics of the patient population. This study sample includes all patients admitted for labor induction at either the Hospital of the University of Pennsylvania or Pennsylvania Hospital over the study period meeting inclusion and exclusion criteria for use of the standardized labor induction protocol

Unit-level A&F reports for the 2 labor units were successfully distributed every 3 months for the 2-year POST period at both sites, for a total of 7 distributed reports per site (14 total). PRE to POST-implementation, fidelity to 4 of the 8 components increased significantly (cervical Foley utilization, latent labor examination frequency, amniotomy timing, and intrauterine pressure catheter utilization), without change in the other 4 components (Table 2).

Table 2 Fidelity to 8 individual components of the labor induction protocol compared pre- and post- implementation

Those 4 components that demonstrated improved fidelity from PRE to POST were then further evaluated by comparing Month #1 to Months #2/3 after an A&F report to assess the effect of A&F on that component over time. For 2 of those 4 components (amniotomy timing and IUPC utilization), there was no difference in fidelity by month after A&F report; rather, there was sustained improvement in these components across the POST-implementation period (Table 3). On the other hand, for the remaining 2 components (cervical Foley utilization and latent labor examination frequency), fidelity peaked in the first month after each A&F report, with some fidelity decline in the following 2 months prior to the next A&F report. Trends across the POST period for each of these 2 protocol components are shown in Fig. 2.

Table 3 Fidelity to the 4 individual components of the labor induction protocol that demonstrated improvement from pre- and post- implementation, comparing Month #1 to Months #2/3 after an A&F report to evaluate the effect of A&F on that component over time

Fig. 2

Trends among the 2 protocol components which showed statistical differences when comparing Month#1 after an A&F report as compared to Month#2/3, over time in relationship to each audit and feedback report; A: Component #1- If cervical ripening balloon is utilized, if remains in place at 12 h after placement, remove it and initiate/continue oxytocin; B: Component #4—Latent labor exams should be performed: At least every 3 h if misoprostol and/or Foley being used; At least every 4 h if oxytocin is being used. Of note, scales differ across components to elucidate the points made. Red dots indicate Month #1 after an A&F report

Qualitative results

104 clinicians across both sites completed the AIM survey. In determining cut points for tertiles, those with scores ≥ 17 (n = 28) were placed in the “High Acceptability,” 14–16 (n = 33) in the “Middle Acceptability,” and ≤ 13 (n = 43) in the “Low Acceptability” groups [21]. A total of 24 interviews were performed: 12 in the High and 12 in the Low Acceptability groups. Interviewees included 15 physicians (13 obstetrician-gynecologists and 2 family medicine physicians), 2 certified nurse-midwives, and 7 registered nurses.

Regardless of High or Low Acceptability, participants almost universally reported that they appreciated that data around protocol fidelity was being collected and tracked and found the distributed audit and feedback reports to be “helpful” and “motivating.”

“I actually find those to be very helpful because, it's nice to know that we are like being -- not being monitored - but it's nice to know that it's something that's actively being looked at and like looking at ways that we can do better.” (OBGYN Resident Physician)

“I think that we're driven by trying to be better and I think by seeing the things that we are not good at, it's motivating to be better at them.” (OBGYN Resident Physician)

Participants consistently reported surprise when low protocol adherence was demonstrated on the A&F reports and noted that low fidelity served as a personal driver for behavior change.

“I like those because it's really helpful to see the actual evidence of ‘I think we follow this protocol much more than we actually do,’ and it's kind of horrifying to see that only 40% of patients or whatever are getting their water broken when they should or things like that.”(Maternal Fetal Medicine Fellow)

“I like it. Definitely, it puts things in perspective, because I feel like sometimes at work, you're so in the moment that you might think there's no way we're following this the right way or like, I can't believe we're not doing something like, you think you're always doing it, but whenever you see the numbers, you're like, wow, we're not doing it as much as I thought. So, it's definitely eye-opening.”(Labor Nurse)

Rarely, participants reported that the A&F reports did not change their behavior because they were already trying to do their best at every component.

“I read them, but I don't change my behavior based on them. 'Cause I feel like in general, I'm trying to like actively do the protocol, so I don't say like, ‘Oh gosh. We're really failing on this particular one, so let me pay more attention to that.’" (OBGYN Attending Physician)

Additionally, some clinicians found it reassuring to see that their personal experiences were reflected in the overall statistics of the unit, including seeing improvement in fidelity to specific components that they as individuals were actively working on or noticing that other clinicians were also having difficulty improving in specific areas.

“Yeah, I think it's always nice to get a sense of how we're doing and what sort of globally we are, as a group, struggling with or not always meeting our goals. And in some ways, it can be reassuring, like, okay, the things that I've been finding challenging about the protocol like other people have found challenging, and it can also be encouraging that like, okay, we're improving.”(Family Medicine Resident Physician)

Discussion

Audit and feedback served as the primary implementation strategy in this type I hybrid implementation-effectiveness trial. A&F was an effective implementation strategy to promote fidelity to 50% of the components of this labor induction protocol. Among the 4 components for which fidelity increased, 2 showed sustained improvement, while the other 2 showed some decline in effect after the first month POST-A&F. Qualitative data support the positive clinician perception of these A&F reports as helpful, motivating, and reassuring overall.

The body of literature supporting the impact of A&F remains overall limited [3, 4, 40]. There is data supporting the general concept that the success of A&F is dependent on context, including relationships between auditors and clinicians, the clinicians’ beliefs in the ability to enact change, and the way the feedback is presented [8]. Our work is consistent with other studies of A&F outside of the maternal health context, demonstrating modest improvements in intervention utilization and fidelity [3]. Prior work out of our group began to assess the issue of A&F dosage for interventions on labor and delivery [6]. In that prospective study, daily A&F on weekdays was utilized for fidelity to a single-component intervention, which showed success and sustained improvement over the weekends, likely indicating the ability for A&F to be performed less frequently than daily. However, A&F is time and resource-intensive, leading to a drive to perform A&F with the lowest effective dosage, particularly for an intervention as complex as a multi-component protocol. In this manuscript, while every 3-month A&F demonstrated impact, such an infrequent dosage may have been insufficient for the sustained effect for all components of the intervention. The ideal dosage of A&F for a labor intervention likely lies somewhere between daily and every 3 months.

This work lays the groundwork for future comparative studies evaluating A&F dosages and impact on the implementation of evidence-based practices, particularly in maternal health. In addition, this work utilized A&F for an intervention distinct in 2 aspects: (1) the protocol’s complexity and multiple components, and (2) the focus on improving health equity. Related to the protocol’s complexity, it is important to note that improvements were only seen in half of the intervention components. Improvement in these components ranged from one to twenty absolute percentage points, and we do not currently have data for what clinicians would determine to be meaningful improvement within that range. While 2 of 4 unimproved components had high fidelity at baseline, offering limited room for improvement, the other 2 (active management of latent labor and utilization of oxytocin for labor dystocia) simply showed less success. These data begin to show that A&F may not be a blanket strategy for implementing what seems like a single overarching intervention, but that unique and targeted implementation strategies need to be selected for individual components or steps in a complex process.

When one examines the components of the protocol in which adherence was impacted by A&F compared to those that are not, combined with qualitative data on barriers and facilitators to individual component use from the clinician perspective [data not yet published], new hypotheses about the mechanisms of A&F effectiveness begin to emerge. The two components for which improvement was seen with A&F and sustained involved a single action (rupture of membranes at a specific dilation and placement of an IUPC when indicated). The components that showed improvement but were not sustained as long with A&F required adherence over a period of time in the labor course; for example, frequency of latent labor exams occurring every 4 h, when latent labor can last as long as 36 h. Specifically, to adhere to this component, the clinician may need to take as many as 9 actions. The biggest barriers to compliance with these types of components are competing priorities and floor acuity, and such priorities may shift further from recommendations with increasing length of time from an A&F report. Components that require multiple actions to adhere to may either require a suite of implementation strategies (such as addition of coaching or facilitation) or more frequent A&F. Finally, among the 2 components where adherence rates did not begin high and yet improvement was not seen with A&F, both involved initiation of oxytocin for specific indications in either latent or active labor. The decision to start oxytocin is dependent not just on the clinician, but on the patient, the nurse, and the status of the patient’s contraction pattern and fetal heart rate monitoring. The lack of impact of clinician A&F on these components may indicate that alternative, possibly patient-facing targets or electronic health record-based solutions, are needed to overcome multilevel barriers to beginning oxytocin during labor induction.

With regard to health equity, adherence to the standardized labor induction protocol is designed to impact disparities in labor outcomes among Black and non-Black birthing people. In this study, A&F report emails started with that ‘why’ as a means of re-establishing shared priorities and framing the goals of behavior change. Future directions will analyze the clinical impact of the protocol on these inequities, in context to how protocol implementation differed among components by patient race.

This study is limited in its generalizability, as it evaluates the impact of A&F on the standardization of labor induction at 2 labor units in the same health system. Due to challenges in attributing utilization (or lack of utilization) of labor induction components on labor and delivery to individual clinicians given the shift- and team-based nature of the work, A&F was provided here at the unit level. Furthermore, another driving factor of adherence to a labor induction protocol is patient desires and perspectives on the components, which are not targeted using the clinician-focused A&F approach. Our analysis was not designed to answer important related questions, such as comparing models of A&F (such as unit versus individual level), comparing frequencies of A&F, sustainability of A&F impact over time, or assessing the specific mechanisms by which A&F may or may not have had an impact on fidelity to individual components. The 3-month A&F cadence evaluated in this work was selected based on perceived feasibility by the research team, and increased frequencies of A&F would need to be evaluated for comparative feasibility. On the other hand, this work has significant strengths. As it was performed in the context of a large type I hybrid trial, determination of fidelity to individual protocol components was performed in over 8000 charts via individual chart review. The study included a structured and standardized delivery of A&F throughout the POST-implementation period across both sites. Finally, the qualitative aspect of this work adds to the literature by assessing the clinician perspective on A&F during the course of this study.

Conclusions

In conclusion, A&F was an overall successful implementation strategy to promote fidelity to this labor induction protocol. With some decline in effect after the first month POST-A&F report for specific components, increased A&F frequency greater than every 3 months should be considered in future work for complex protocols targeting obstetric inequities.