Evaluation of a notes-based rapid qualitative analysis method to facilitate implementation

Abstract

Background

Qualitative methodologies offer a nuanced approach to understanding stakeholder perspectives, preferences, and context in implementation research. However, traditional qualitative data analysis can be time consuming and create barriers to responsive implementation of interventions. Rapid qualitative methods that yield timely, actionable results have emerged to expedite the evidence-to-practice gap, but often require all analysts to have implementation science expertise and resources for interview transcription. This study describes a novel rapid qualitative method to identify participant-driven social care recommendations in real time.

Methods

Caregivers of pediatric patients were enrolled onsite at two primary care clinics and one emergency department affiliated with a large urban pediatric healthcare system. A semi-structured interview guide was developed using the Health Equity Implementation Framework and Integrated Behavioral Model in partnership with multidisciplinary implementation stakeholders. Telephone interviews explored 60 caregivers’ experiences with and perceptions of receiving social resources from healthcare. For traditional analysis, NVivo12 was used to code the first 10 verbatim transcripts to generate themes in an integrated inductive/deductive approach. In the rapid approach, a summary notes template designed to capture implementation-related data was completed immediately following the same 10 interviews. A secondary analyst used the templates to create participant-level summaries and identify implementation-related themes. Themes found in each method were quantified and mapped onto each other using an analytic matrix to compare the number and consistency of themes.

Results

Themes generated in both methods mapped consistently onto each other; 92.8% of themes found in traditional analysis were accounted for within our rapid method. The quantity of themes was similar between the two methods: the traditional approach generated 69 themes and 22 subthemes, while our rapid approach generated 72 themes and 21 subthemes.

Conclusions

Our interview notes-based rapid qualitative method was successful in producing themes consistent with the traditional approach in both content and quantity. This approach is also pragmatic, as it does not require analysts to have deep implementation science expertise and saves transcription costs. By balancing rigor with time to actionable results, this rapid method provides a tool for implementation researchers to generate qualitative findings on an accelerated timeline to inform policy and practice.

Clinical trial registration

This study was registered at ClinicalTrials.gov, #NCT05251311, https://www.clinicaltrials.gov/study/NCT05251311, on September 30, 2021.

Contributions to the literature

• Our comparison of a novel rapid qualitative analysis method to a traditional qualitative analytic approach expands the limited evidence base on the rigor of rapid methods to accelerate use of qualitative research evidence and inform implementation efforts in real time

• The rapid qualitative technique described here enhances the pragmatism of existing methods as it does not necessitate interview transcription, saving time and financial resources

• This approach is the first to collect data using an interview notes template that does not require each researcher to have expertise in implementation frameworks and theories

Background

Qualitative methodologies offer a nuanced approach to understanding stakeholder perspectives, preferences, and context in implementation science research [1, 2]. These methods are particularly valuable in healthcare-based studies, as they can help clarify the complex ways that patients, caregivers, and clinical stakeholders view and experience health system interventions [1, 3]. Qualitative research leads to the generation of rich and robust data; however, traditional qualitative methods, which typically entail transcription, cleaning, and codebook-based analysis of data, require significant time to complete and may inadvertently recreate the evidence-to-practice gap [4, 5]. This lengthy data collection and analysis period is often at odds with timeline pressures for implementation, resulting in findings that are “less relevant and potentially even obsolete” by the time they are published and limiting utility for implementation purposes [2, 6, 7].

Rapid methods have gained popularity in implementation research due to their ability to yield timely, actionable, and patient-centered results [4]. Studies have identified several advantages of rapid methods in general that hold particular promise for the field of implementation science, including time, cost and resource savings as well as improved efficiency [4, 8]. Despite the potential benefits of rapid methods, tension exists between the accelerated generation of results and scientific rigor [6, 9]. For example, some have raised concerns about maintaining trustworthiness of results generated by rapid methods [9]. These critiques highlight a need for comparative study to assess the rigor of rapid qualitative methods relative to traditional methods.

Various rapid techniques aimed at reducing the time or resources required to perform qualitative methods have been described, including scribing [10], direct analysis of audio recordings [11, 12], mind maps [13], Consolidated Framework for Implementation Research structured templates [6], and interview notes with audio verification [7]. While several studies used transcript-based interview notes to condense qualitative data in their rapid technique [7, 14, 15], no previous study to our knowledge examines the rigor of a rapid analysis method that does not require interview transcription. In addition, these techniques have not been applied to research in social care, the field concerned with the implementation of social care interventions in the medical setting.

Context and aim of the study

Social care— clinically based programs that assess and respond to social risks such as food or housing insecurity, unsafe environments, and financial strain— is rapidly expanding across hospital systems in the United States [16, 17]. However, many questions remain regarding the most family-centered and equitable practices to support engagement with social care, and there is limited evidence to guide implementation [18, 19]. Concurrently, mandates and incentives from the Centers for Medicare and Medicaid and Joint Commission as well as modifications to the US News and World Report hospital rankings have intensified pressure on healthcare systems to implement protocols to screen for and address patients’ social risks [20]. Socially Equitable Care by Understanding Resource Engagement (SECURE) is a mixed-method hybrid type 1 effectiveness and implementation study grounded in the Health Equity Implementation Framework (HEIF) with the goal of understanding how best to facilitate family-level engagement with social resources from the pediatric clinical setting [21]. The qualitative component of this study includes interviews with caregivers about key implementation factors, including perceptions and preferences regarding social risk screening and resource information provision in the healthcare setting, as well as documentation, and follow-up. Qualitative data collection and analysis for this study were conducted over several years, between September 2022 and December 2024. However, given the accelerated pace of advancements in social care, patient-centered evidence was needed to inform implementation sooner than was feasible using traditional qualitative methods alone.

This study describes the development of a novel rapid qualitative method and its application to social care. This rapid qualitative data analysis method was designed for use in conjunction with traditional methods to identify caregiver-driven implementation recommendations and guide local social care efforts in real-time. To evaluate the rigor of this approach and assess its potential to enhance social care, we compared the consistency and quantity of themes generated by rapid analysis of an initial subset of interviews to those generated concurrently by traditional in-depth analysis.

Methods

Participant recruitment

SECURE enrolled 3949 adult caregivers of pediatric patients ages 0–25 years old at two primary care clinics and one emergency department affiliated with a large urban pediatric healthcare system between April 2022 and August 2023. Study sites were intentionally selected for their demographic diversity. Caregivers who spoke one of six study languages (English, Spanish, Arabic, Mandarin, Vietnamese, and Portuguese) were eligible for enrollment; however, the qualitative component of the study was conducted in English and Spanish only. Caregivers were enrolled on-site during waiting periods in individual patient rooms. All participants provided verbal informed consent to a Research Assistant certified in research ethics prior to enrollment in the study. After demographic data was obtained, caregivers were randomized to complete one of three assessments of social risk or need. All study procedures were conducted on the caregiver’s personal smart phone. We compared caregivers’ acceptance of and engagement with social resources when information through a searchable electronic resource map was presented (1) alone, (2) following a resource menu in which participants self-select social domains for assistance, or (3) following social risk screening. Participants were contacted by a resource navigator five days after initial enrollment for supplemental tele-resource navigation, unless they opted out. Additionally, all participants could opt-in to same-day resource navigation to address emergent social needs, if desired.

Interview guide development and data collection

The goal of the qualitative component of the SECURE study was to broadly explore caregivers’ feelings about social care. In service of this aim and to address policy and health system needs in the short term, a portion of the interview guide was focused more narrowly on understanding caregiver factors related to social care implementation. The semi-structured interview guide was developed based on an adapted version of the HEIF, and the Integrated Behavioral Model (IBM) [21, 22]. The IBM has six constructs that aim to examine attitudes, perceived norms, and personal agency around a behavior; the behavior examined in our study was engagement with social resources provided from healthcare [22]. The HEIF considers health equity determinants in intervention development and implementation and provides an organizing structure for how these determinants may affect intervention uptake (Fig. 1). The IBM and HEIF were selected for their complementary abilities to clarify both the individual-level factors and structural determinants that underlie a complex behavior such as engagement with social care. We expected that the IBM’s focus on behavioral intention as well as contextual factors such as environmental constraints would elucidate barriers and facilitators to social care engagement. Furthermore, given the roots of social need in systemic racism and other systems of oppression, the HEIF was fitting for this research.

Fig. 1

Health Equity Implementation Framework for the Socially Equitable Care by Understanding Resource Engagement study

We employed a multistep process to develop the implementation-focused sections of the interview guide to ensure that data collection was maximally grounded in implementation theory (IBM) as well as an implementation framework (HEIF). We first mapped social care concepts salient to our research questions to IBM constructs to identify the socio-behavioral factors that may underlie engagement with social care (Fig. 2). We then convened multidisciplinary stakeholders, including social workers, primary care and emergency medicine physician-researchers, clinic managers, and qualitative research experts to brainstorm key lines of inquiry corresponding to each study-specific IBM construct. For example, for the “experiential attitudes” IBM construct, which corresponded to “likes and dislikes about the idea of engaging with resources provided by healthcare” in the context of our study, the team suggested that we explore caregivers’ feelings about discussing social needs with their healthcare provider during a medical visit. The group also returned to the HEIF to confirm that all caregiver-level factors were accounted for, filling in gaps as needed to ensure that relevant implementation determinants would be thoroughly explored in interviews. After we had generated a comprehensive list of potential lines of inquiry aligning with each IBM and patient-level HEIF construct, we created a matrix matching theoretical constructs and determinants to their relevant study concept and potential interview question(s). We refined the draft interview questions, then piloted the guide with five caregivers of pediatric patients who met study eligibility criteria to confirm clarity and acceptability. The final interview guide explored caregiver experiences with the study social care intervention, facilitators and barriers to using community-based social resources, and experiences and preferences related to receiving social care in the pediatric clinical setting (see Supplemental Material).

Fig. 2

Integrated Behavioral Model framework for the Socially Squitable Care by Understanding Resource Engagement study

At the time of enrollment, caregivers could consent to be contacted for a semi-structured telephone interview; 70.4% of enrolled caregivers consented. Interview participants were selected using a hybrid random and purposive sampling strategy to ensure representation across demographic groups (race, ethnicity, preferred language, gender, insurance status, education, previous use of social resources) and intervention conditions (randomization arm, enrollment location, receipt of resource navigation) (Table 1). Interviews were conducted between 30 and 45 days after study recruitment to allow time for resource engagement and resolution of any acute medical complaint. All participants again gave verbal informed consent prior to their interview. The first ten interviews conducted were selected for analysis through both traditional in-depth and rapid approaches; these ten interviews were completed between September and November 2022. We evaluated the rigor our of rapid method early in data collection to identify any potential need for modifications before applying it to the entirety of the study, as we planned to use these findings to inform social care implementation in real time. A total of 60 interviews were conducted in the larger qualitative study.

Table 1 Demographic characteristics of study participants included in subsample used for comparison of rapid and traditional qualitative analysis methods

We used the Standards for Reporting Qualitative Research (SRQR) reporting guidelines to inform the presentation of our study results (see Supplemental Material). All study procedures were deemed exempt from review by the Children’s Hospital of Philadelphia Committee for the Protection of Human Subjects (IRB #21–018785).

Traditional in-depth analysis

Interviews were digitally recorded, deidentified, and transcribed verbatim. Transcripts were reviewed and cleaned by the interviewers, then coded in NVivo 12 using an integrated inductive and deductive approach [23]. An initial codebook was generated based on the interview guide and subsequently modified as new themes emerged from the data. Four study team members (RB, LQ, ECC, JDR) independently coded each transcribed interview using the constant comparison method; the 10 transcripts that comprised our sample for this rapid qualitative analysis study were all either double or triple coded [24]. The study team met twice per month along with the Principal Investigator (DC) to review each coded transcript, assess inter-rater reliability, resolve coding disagreements, and revise code definitions. The team members who performed coding were trained by researchers experienced in qualitative methods (DC and RB), who reviewed coding regularly with the research team and coded these transcripts in parallel.

The coded transcript segments were summarized into a matrix organized by code by a secondary analyst (SCK). Applying a thematic analysis approach, coded data were then distilled into themes and sub-themes that could inform social care implementation: caregiver likes and comforts, dislikes and discomforts, and suggestions regarding the study’s process, community resources, and receiving help with social needs in a healthcare setting (Fig. 3). At this stage of coding, themes were intentionally kept granular to rigorously evaluate the degree of overlap between qualitative analysis approaches before further condensing for dissemination purposes.

Fig. 3

Matrix used to distill interview themes in traditional and rapid qualitative analysis

Rapid analysis

An interview notes template was developed that included the interview questions intended to yield data relevant to social care implementation efforts (Fig. 4). During or shortly after each interview, the interviewer (RB, LQ, ECC, or JDR) documented key participant responses directly in the template, returning to the interview audio recording as necessary; templates required approximately 30 min to complete. A secondary analyst (SCK) listened to the interview recordings of the first 10 interviews to assess completeness in each interview note template and add any missing data. The secondary analyst then used this template to create a summary of participant likes and comforts, dislikes and discomforts, and suggestions related to social care expressed in each individual interview. These summary sections were entered into a copy of the analytic matrix used in our traditional analysis, and content was distilled into overarching themes and sub-themes regarding the study’s process, perceptions of using community resources, and thoughts and preferences about receiving help with social needs in a pediatric healthcare setting (Fig. 3). The same approach to discrete theme identification and distillation that was used in traditional analysis was applied to the rapid analysis.

Fig. 4

Sample portion of the interview notes template used for the rapid analysis method

Transcripts from these 10 interviews were used as the main data source for the traditional analysis, while interview recordings and completed summary notes templates were used as the main data source for the rapid analysis.

Comparison of methods

Consistency of themes

The final matrix structures generated for both the traditional and rapid methods were identical to facilitate theme comparison, with the exception of a representative quotes column in the traditional analysis matrix. Key themes and subthemes generated by both methods were compared using a merged analytic matrix to determine consistency; themes identified through each method were grouped by category and listed in parallel to identify points of overlap and difference. When mapping themes onto each other, we followed Taylor et al.’s approach of categorizing themes into matches, partial matches, and no matches (Fig. 5) [25]. A match was defined as an identical theme found in both analyses, while a partial match was two similar but not identical themes found in each analysis. No match was defined as a theme not found in the alternative approach in any capacity.

Fig. 5

Themes identified related to “connecting with community resources: dislikes and discomforts” in both traditional in-depth and rapid approaches. Themes in orange had no match, themes in green are partial matches, and themes in black fully mapped to a theme in the opposite approach. Total themes and subthemes for this section are calculated in the bottom row

Quantity of themes

The number of themes and subthemes generated by each method were quantified and compared (Fig. 5).

Results

Consistency of findings

In our qualitative comparison of findings between rapid and traditional analytic approaches, the rapid analysis produced similar key themes as the traditional analysis in both content and quantity. Themes identified in the traditional approach consistently mapped to themes identified in our rapid analysis; 92.8% of themes generated in the traditional analysis were represented in the rapid analysis as either a match or partial match, demonstrating that our rapid approach was successful in capturing the majority of themes generated by the traditional approach.

Examples of full thematic matches are included in Fig. 5. A total of 64 themes from the rapid analysis and 61 findings from the traditional analysis were categorized as full matches, while 5 themes from the rapid analysis and 3 themes from the traditional analysis were categorized as partial matches. Several nuanced themes from the rapid analysis mapped to a single broader theme identified in the traditional analysis, accounting for the difference in full matches. One example of a partial match is the theme of concern about judgement by others when accessing community resources, found in the rapid analysis. This was not explicitly included as a theme in the traditional analysis, however themes of embarrassment and concern for privacy surrounding resource use and mistreatment by program staff imply similar conclusions as the theme of judgement. Eight themes in total were categorized as having no match: 3 (4.2%) of all rapid analysis themes and 5 (7.2%) of the traditional analysis themes. Thus, while the rapid approach excluded several themes that were found through the traditional approach, it also captured themes that were missed in the traditional approach.

Notably, the quantity of themes found in both methods were similar. The traditional analysis yielded a total of 69 themes and 22 subthemes, while our rapid approach yielded 72 themes and 21 subthemes.

Discussion

Qualitative methods help capture the richness of participant perspectives, strengthening implementation efforts by centering the voices and experiences of those engaging with an intervention [1]. However, traditional approaches to qualitative analysis are time intensive and results from this research can become less relevant by the time of publication, creating barriers to meaningful implementation [5,6,7]. Rapid qualitative methods have emerged to help address this barrier [4]. The purpose of our research was to develop and assess a method of rapid qualitative analysis using a novel structured interview notes template that allows for faster generation and implementation of qualitative research findings. We found this rapid method to be a feasible and rigorous approach to generating themes for the subset of caregiver interviews analyzed. A concern around rapid analytic methods is that the abbreviated analysis would lose richness and quantity of themes generated [9]; however, our rapid method successfully maintained this rigor with 92.8% of themes found in the traditional analysis produced in the rapid method as well. While the rapid method did not capture 5 of the 69 total implementation-focused themes found in the traditional method, it did successfully capture 3 important themes that were not generated in the traditional method.

Our pragmatic rapid qualitative method entailed completion of a structured interview notes template by the interviewer shortly after each interview, which was subsequently reviewed and further summarized by an additional analyst. This approach builds on the method described by Gale et al., maintaining use of a theory-informed summary template to organize implementation-relevant data [6]. Rather than populate the summary template with data extracted from interview transcripts, however, our method uses structured interviewer notes to capture key data. This further consolidates the time from data collection to analysis by integrating interview summary notetaking into the interview process rather than waiting for completed transcripts. Furthermore, our rapid approach promotes inclusion of meaning conveyed through non-verbal cues such as pauses, laughter, and tone [26]. While artificial intelligence-generated text summaries are becoming more widely accessible as an alternative to more resource intensive approaches to condensing data, they cannot yet accurately capture these important elements. This approach also circumvents the cost associated with transcription while keeping rapid coding simple and structured.

As data were coded into a summary template organized by interview question in our method rather than by theoretical construct, interviewers and coders were not required to have significant implementation science expertise to successfully collect and analyze data. As the HEIF and IBM were used to develop the interview guide and interview notes template, data were naturally segmented in line with these frameworks. This resulted in a simpler, lower resource analysis protocol that contributed to the work’s overall efficiency while remaining grounded in an implementation science theory and framework. Furthermore, integrating a theory as well as a framework yielded more comprehensive findings, as we used the IBM to deepen our understanding of constructs within the HEIF. In line with the Gale et al. method, a secondary analyst reviewed summary templates to ensure quality and consistency, and analytic matrices were used to identify implementation-relevant themes and subthemes across participants [6].

Our rapid qualitative approach successfully captured similar implementation-oriented themes as a traditional analytic approach, adding to the existing literature of how rapid qualitative analysis can be a valid and rigorous method for performing qualitative data analysis [4]. However, while a major focus of this work is emphasizing the importance and utility of rapid methods in qualitative implementation research, this rapid approach is intended to guide researchers in identifying key insights to inform implementation efforts in real time as a complement to traditional analysis rather than a replacement. Our rapid analysis had a deliberately narrow focus on identifying implementation factors from the broader interview, enhancing the comprehensiveness of the rapidly collected data and concordance of the rapid and traditional findings. This identifies an important use-case for rapid qualitative methods as an adjunct to traditional qualitative analysis, particularly in hybrid effectiveness-implementation studies or research aimed at informing practice or policy in real time, generating rapid information regarding determinants to implementation success without supplanting the importance of detailed thematic analysis of effectiveness outcomes. Furthermore, traditional methods generate a more comprehensive account of a participant’s viewpoint, reveal findings beyond those that are implementation-specific, and can be hypothesis-generating for future research.

Our team used a secondary analyst to review interview templates for completeness. As is the case with double coding in traditional qualitative analysis, this additional analyst enhanced the rigor of our rapid analysis by assuring that the interview notes templates were completed comprehensively [27]. However, use of a secondary analyst could introduce bias in the comparison of rapid and traditional methods. Although minimal content was omitted during interview note completion by the initial analyst and thus we expect that the secondary analyst’s additional coding had limited impact on the results of our comparison, the potential effect of a secondary coder should be accounted for in future studies. Alternatively, given the completeness of the interview notes templates by the interviewer, we anticipate that the level of detail desired could be attained through additional interviewer training, eliminating the need for a secondary analyst entirely. Completion of both the interview notes template and thematic summary by the interviewer further enhances the pragmatism and efficiency of this approach by conserving staffing resources. Future research could assess the validity of this method when performed by the primary research team without a secondary analyst.

Limitations

It is important to note that qualitative research is subjective by nature and thus can be influenced by bias and differences in interpretation, contributing to limitations to our study. We worked to eliminate this subjectivity by triple coding transcripts, holding frequent team meetings to review coding, as well as by using a secondary analyst to complete analysis of rapid interview note templates. The comparison of themes was conducted by the same secondary analyst, which may introduce potential bias in theme concordance between both methods. This is, however, consistent with previous literature studying rapid qualitative methods [2] and is a pragmatic approach that minimizes the need for additional researchers and resources. While successful in our study, we performed this method to analyze ten caregiver interviews only; more research is necessary to demonstrate that this method is generalizable to other settings.

Conclusion

A major concern of implementation researchers when developing rapid qualitative analytic approaches is balancing scientific rigor and validity with time to actionable results. We describe a rapid yet rigorous method that generates implementation-related findings with comparable depth and breadth as those found in a traditional qualitative analytic approach. Additionally, it is a pragmatic approach that requires minimal training to complete, and generates results in near real-time. Future research could further streamline this method and confirm its validity in other settings and fields of study. While not intended to replace traditional analysis, this approach holds promise to advance the use of rapid qualitative methods in implementation science research.