- Methodology
- Open access
- Published:
Improving measures of context in process evaluations: development and use of the Context Tracker tool
Trials volume 25, Article number: 777 (2024)
Abstract
Background
Process evaluations are increasingly integrated into randomised controlled trials (RCTs) of complex interventions to document their delivery and interactions with local systems and dynamics, helping understand observed health outcomes. Yet process evaluations often struggle to assess relevant contextual determinants, leaving much of the important role of “context” in shaping an intervention’s mechanisms opaque in many studies. A lack of easily adapted data collection methods to help define and operationalise indicators of context likely contributes to this.
Methods
We present a method to help structure measures of context in process evaluations and describe its use in two very different settings. The “Context Tracker” is an innovative tool for use within trials and quasi-experiments to more systematically capture and understand key dimensions of context. It was developed in Zimbabwe as part of a cluster randomised controlled trial and then adapted for a quasi-experimental evaluation in the UK. Both studies provided harm reduction and health services for marginalised and hard-to-reach populations.
Results
We developed the Context Tracker to be both standardised (i.e. formatted and applied in the same way across study sites) and flexible enough to allow unique features to be explored in greater detail. Drawing on the Context and Implementation of Complex Interventions (CICI) and Risk Environments frameworks, we mapped 5 domains across micro, meso and macro levels in a simple table and used existing evidence and experience to predict factors likely to affect delivery of and participation in intervention components. We tracked these over time across study sites using routine programme statistics, observation and qualitative methods. The Context Tracker enables identification and comparison of facilitators and barriers to implementation, variations in engagement with interventions, and how mechanisms of action are (or are not) triggered in different settings.
Conclusions
The Context Tracker is one example of how evidence-based contextual determinants can be used to guide data collection and analysis within process evaluations. It is relevant in low- and high-income settings and applicable to both qualitative and quantitative analyses. While perhaps most useful to process evaluations of complex interventions targeting marginalised communities, the broader approach would benefit a more general research audience.
Background
Rigorous evaluation of complex health interventions is necessary to ensure effectiveness of programmes taken up in policy and practice [1]. Comprehensive process evaluations are increasingly integrated into randomised controlled trials (RCT) and quasi-experiments to document the delivery of intervention activities and their interactions with local systems and dynamics, helping to understand observed health outcomes [2, 3]. Process evaluations examine the implementation of health interventions, test their theoretical foundations by tracking whether delivered components lead to hypothesised changes, and explore the means through which they produce outcomes. There are numerous frameworks to guide process evaluation design, all of which include the need to capture “context” as characteristics and attributes of the setting into which interventions are introduced. The widely used Medical Research Council (MRC) Guidance on process evaluations included context as one of its 3 pillars, alongside implementation and mechanisms of action [2]. However, while specific domains for researchers to operationalise were assigned to the other pillars, context was simply defined as “anything external to the intervention that may act as a barrier or facilitator to its implementation, or its effects” [2].
This initial gap has now been filled by a range of conceptual models offering lists and categories of contextual domains and suggestions for how they might interact with one another to moderate both interventions and outcomes [4, 5]. The revised MRC Guidance acknowledges that context can change during implementation, both as a result of the intervention and for externally driven reasons, e.g. an unrelated policy change [6]. Checklists for reporting design and evaluation of health interventions often equate context with characteristics of the intervention setting [7, 8], which can be further classified, e.g. into geographical, epidemiological, socio-cultural, socio-economic, ethical, legal and political attributes [9]. Realist evaluation approaches highlight the importance of context by defining interventions as new resources and opportunities that, when introduced into a specific context, trigger specific mechanisms of change [10]. By asking “what works, for whom, and in what circumstances?”, realist evaluators blur the separation between an intervention and where it is delivered [11]. Nonetheless, recent reviews have found “context” remains inadequately conceptualised and measured in process and outcome evaluations [12,13,14]. A lack of easily adapted data collection methods and practical guidance to help define and operationalise indicators of context likely contributes to this and authors have commented on the dearth of tools to apply during process evaluations [5, 15].
What guidance exists for incorporating contextual measures into the design of process evaluations often refers to the importance of having a clear logic model that sets out how an intervention’s components are hypothesised to lead to intended outcomes. Logic models (also referred to as logical frameworks, programme theories or theories of change) specify the pathway from an intervention’s activities to effects on systems, behaviours, determinants of health and, ultimately, health outcomes, thus providing the underlying rationale for the intervention [16, 17]. An explicit narrative or diagrammatic map facilitates empirical testing of each step in the pathway, for which indicators and measures are identified in advance. Assumptions, including about contextual facilitators and barriers to success, can be added although, as highlighted by realist evaluators, often are not [18]. Thus while many process evaluations specify data to be collected to track progress along the change pathway, documenting contextual factors tends to rely on ad hoc analysis of information collected for other purposes, often through observations by the research team or interviews with implementors, and thus is limited to factors they consider noteworthy [19]. It is possible, however, to predict key contextual factors in advance based on existing literature and previous experience, particularly local knowledge of implementing partners and participating communities, while remaining flexible and open to unforeseen events or effects [20]. For example, where an intervention addresses a socially sensitive topic, e.g. sexuality education for adolescents, local parental attitudes, religiosity and community trust in school leadership are likely to be key shapers of both delivery and results [21, 22]. Early consideration of potential influencers in each domain of a chosen conceptual framework can increase the likelihood that the process evaluation will capture data that will contribute findings useful for transferring interventions to new settings or delivering at scale outside of carefully controlled research conditions.
Methods
In this paper, we present a method to help plan measures of “context” in process evaluations and describe its use in two very different settings. The “Context Tracker” is an innovative tool for use within trials and quasi-experiments to more systematically capture and understand key dimensions of context. It was initially developed in Zimbabwe as part of a cluster randomised controlled trial (cRCT) and then adapted for a quasi-experimental evaluation of a pilot project in the UK. Although the interventions introduced in each study differed considerably, both addressed infectious disease transmission among criminalised and marginalised (“key population”) communities, which share structural vulnerabilities [23,24,25].
Interventions for marginalised populations can be particularly susceptible to the influence of local context. Health interventions targeting populations with poor relationships with authorities have to navigate contextual barriers related to restrictive legal policy and its enforcement, and consequent suspicion and distrust by community members [26, 27]. For example, drug use and sex work (sale and purchase of sex) are widely stigmatised and criminalised, with individuals involved in these behaviours considered social deviants, vectors of disease and blamed for their disproportionate risk of adverse health outcomes such as blood-borne and sexually transmitted infections, e.g. HIV and hepatitis C [28, 29]. Evaluations of interventions for these communities broadly refer to “context” as legal frameworks and restricted access to health services, but there are highly nuanced differences in how punitive laws, policing strategies and social stigma are locally enacted. These influence facilitators and barriers to implementing activities to reach, engage and provide services and support to marginalised communities, and also shape individuals’ motivation and self-efficacy to interact with the intervention in ways that will trigger its hypothesised mechanisms of action.
As described below, the AMETHIST (Adapted Microplanning to Eliminate Transmission of HIV in Sex Transactions) and SIPP (Safe Inhalation Pipe Provision) studies tested two very different interventions and looked for different health outcomes in diverse populations. Yet the way in which stigmatised, marginalised and often criminalised populations have their agency restricted shares commonalities well-documented in literature, as do the types of interventions known to improve determinants of health and health outcomes for these and similar communities. Harm reduction and peer-led community-level services that work with motivated leaders and their constituencies to reach out to hidden groups, building trust and collaborative relationships, have been shown to be most effective, particularly when participatory and inclusive [30,31,32,33].
The AMETHIST trial was a cluster RCT designed to test whether combined microplanning and self-help groups (SHG) could lead to virtual elimination of HIV transmission in sex transactions in Zimbabwe [34]. The underlying rationale used in the AMETHIST logic model was that providing individually tailored outreach services to female sex workers (FSW) based on their levels of risk, proactively supporting their engagement with HIV prevention and treatment services and organising self-help groups to build social support and problem-solving skills would increase FSW’s knowledge, self-efficacy and motivation to engage with clinical care, thus reducing both vulnerability to HIV acquisition and likelihood of onward transmission [35].
The SIPP study [36] is a quasi-experimental structural intervention seeking proof-of-concept for provision of safe inhalation pipes to people who use crack cocaine in the UK. Rationale for the intervention is rooted in principles of harm reduction [37, 38] and community involvement [39, 40], and thus co-designed with users of crack cocaine. The SIPP logic model hypothesises that provision of a safe inhalation kit with pipe, related paraphernalia and information, supported by online training for health providers and peer-to-peer pipe distribution, counselling and referral to drug treatment services would reduce crack injection frequency and harms from shared pipe use, and thus contribute to reduced transmission of blood-borne and respiratory infections, based on causal evidence for these negative health outcomes [41].
Both studies nested prospective process evaluations within their evaluations using the MRC Guidance to collect data on implementation fidelity, feasibility, acceptability and tracked progress across the hypothesised change pathway set out in each logic model. Each used mixed methods, bringing together routine programme statistics, qualitative interviews and field notes, and questions on exposure and perceptions of intervention activities integrated into endline surveys. Both interventions were further likely to be highly sensitive to localised circumstances, particularly how sex work and drug use are organised and conducted, socially perceived, policed and/or disrupted, thus necessitating comprehensive monitoring of contextual factors and their integration into evaluation findings.
Results
Development of the Context Tracker for AMETHIST
The “Context Tracker” tool was developed and introduced in 2020 when the COVID-19 pandemic restricted travel and interpersonal contact in Zimbabwe, roughly 1 year into the trial [42]. The AMETHIST study initially planned in-person observation and face-to-face qualitative interviews with study staff and participants to capture events, local conditions and trends in each study site. As with many process evaluations, we had not explicitly defined features of context nor specific indicators. However, when confronted with unavoidable changes to both the intervention’s delivery and our research, we had to make adaptations to both. We also wanted to document variations in how national COVID-19 restrictions were being implemented across study sites. We did not consider the pandemic itself, as a global event, to be a contextual factor except in how restrictions were imposed and enforced differently and effects of this on local communities, particularly how these interacted with delivery of the intervention and engagement by our participants. This caused us to think in a more structured way about relevant domains of context (related to COVID-19 and beyond) and how to find a useful structure for systematically capturing these, particularly through remote forms of data collection.
We developed a tool that was both standardised (i.e. formatted and applied in the same way at all 22 study sites) and flexible enough to allow for unique features in each site to be explored in greater detail. The conceptual rationale was that we could map out evidence-based domains likely to affect both delivery of and participation in intervention components and track these over time, routinely capturing developments in each domain for each site without the need to travel. We also embedded flexibility by allowing for unexpected sources of influence, particularly given the unprecedented experience of a global pandemic and growing evidence of how responses in many settings intersected with existing forms of discrimination, stigma and blame [43, 44].
The tool consisted of an Excel database (see Fig. 1) and accompanying interview topic guide that was administered monthly by phone to on-site, purposively selected intervention staff and participants; the spreadsheet was populated during the telephone interview and data extracted, reviewed and synthesised quarterly. The original context domains identified for the study were COVID-related restrictions and their effects; local political or socio-economic developments; infrastructure; institutional/programme issues; and unexpected events. These were selected based on our familiarity with types of differences across the sites and early observations of how COVID-19 interacted with local conditions in ways we predicted would affect delivery and outcomes of the intervention.
The topic guide started with two open-ended questions used to explore each domain: “1) Are there factors/events/happenings that occurred this month that you think influenced (helped/hindered) how you do your work or planned to do your work or how the intervention functioned? How? Why? And 2) What could be done to manage contextual issues that affect the work?” These were followed up every month over 18 months by specific probes to check whether each recorded item was still relevant or had been resolved/changed. Follow-up questions were iteratively developed each month following review of previous entries. By allowing for open-ended responses and longitudinal tracking, the tool could capture the evolving nature of these social dynamics and their impact on the intervention. During the height of the pandemic, however, our topic guide included 8 “standing probes” to capture contextual variations across study sites in socio-economic impacts (changes to sex work), policy (which restrictions were introduced and how enforced), epidemiology (changes to HIV prevention and treatment practices) and accessibility of healthcare (engagement with both intervention and other services); these are listed in Table 1.
Study sites were listed in rows and time periods in columns. Notes were taken during the interview by hand then transferred in bullet-point or summary form into the spreadsheet. Content analysis (both quantitative and qualitative) compared experiences between and within sites, identifying emerging themes for each domain and similarities and differences across time and place. The Context Tracker naturally lends itself to qualitative framework analysis given that it presents key data in “charting” format. The next steps entail mapping and interpreting the organised data to identify patterns, relationships and insights [45]. Figure 1 provides a snapshot from 1 month’s use of the tool, with locations anonymised.
The tool proved useful for integrating data on pre-identified contextual domains with open-ended exploration of emerging issues that might influence the delivery or mechanisms of the intervention. For example, we were able to understand variations in FSW’s access to health care in different sites related to how COVID-19 travel restrictions were locally enforced by linking routine programme data on clinic attendance to reports in the Context Tracker of where public transport had been halted and/or people were apprehended by police when travelling (“COVID restrictions” domain). We also monitored differences in intervention and comparison sites, such as the fact that 40% of intervention clinics experienced regular power cuts compared to only 10% of comparison sites (“infrastructure” domain). We recorded changes in economic activity and how these affected service uptake. In 5 out of 22 trial sites, an increase in illegal mineral and gold mining changed the structure of sex work as a large number of FSW arrived following the influx of new miners but entered into “temporary marriages” rather than selling sex from bars, truck stops or other “hotspots” (“socio-economic changes” domain). This reduced their willingness to attend targeted services that would identify them as sex workers, but increased the likelihood of condomless sex (within more established relationships). It also contributed to higher levels of gender-based violence as competing gangs of illegal miners fought over access to local resources, including sex partners. These developments were followed over subsequent months and we responded by offering a phone-based peer referral system but continued to struggle to reach those FSW whose movements were circumscribed by fear of violence, which finally reduced after a police crackdown on local machete-wielding groups (“unexpected events” domain).
Conceptual refinement of the tool
Following the successful experience of using the Context Tracker in Zimbabwe in which two authors (FM and JB) were involved, CV and JB started planning the SIPP process evaluation. While defining the SIPP logic model and identifying relevant indicators, utility of the AMETHIST approach to prospectively capturing pre-specified contextual determinants became clear, as did the need for greater conceptualisation and refinement of the tool.
Following discussion and informal review of the literature around measures of context in process evaluations, we first adopted Rogers et al. definition of “context” as “a multi-dimensional construct encompassing micro, meso and macro level determinants that are pre-existing, dynamic and emergent throughout the implementation process” [5]. We subsequently selected two frameworks to guide further development of the data collection tool, Context and Implementation of Complex Interventions (CICI) [9] and Risk Environments [46, 47].
These frameworks both reflect socio-ecological models where individuals are embedded in family and peer groups that reside in communities, which form culturally distinct societies shaped by the laws, policies and economies of the country or region [48, 49]. They categorise social layers of influence into three levels: macro (the highest social level, e.g. national law), meso (related to the interventions’ organisation or community, e.g. social support networks) and micro (attributes of individuals or peer groups, e.g. risk behaviour). The Risk Environments framework is specific to the experience of marginalised populations, who are often “hidden” or excluded [46]. It was originally developed to design harm reduction services to prevent HIV among injecting drug users, but has since contributed to analysis of vulnerability of FSW [47, 50], men who have sex with men [51] and transgender women [52, 53]. The Risk Environments framework identifies “macro” and “micro” levels, but many of its micro-level determinants are primarily community-based (availability of services, social networks and norms, etc.) and thus can be considered “meso” level, but these levels serve as guides rather than rigid categories. The Risk Environments lens considers how individual engagement in “risky” practices despite awareness of their harmful consequences are shaped by social determinants categorised as physical, social, economic and policy. For example, the policy of criminalising drug users results in aggressive police enforcement that can affect both health risks (rushed consumption, sharing of scarce injecting equipment) and reduce provision of and access to harm reduction services [54, 55]. These frameworks resonated with issues we considered salient for FSW services in Zimbabwe and provision of drug consumption equipment in the UK, and also complemented each other through focusing on measuring context in the CICI and understanding (and acting on) determinants of risk among marginalised and hard-to-reach populations in Risk Environments.
We selected 5 domains across all 3 levels (Table 2) and pre-populated each cell with potential shapers of how interventions are delivered and experienced, focusing on interventions for key populations. Any given research study should identify which of these (or others) to include and how to access the relevant data (background information, routine statistics, qualitative methods); accompanying topic guides can be tailored accordingly, relevant respondents selected, and data collection frequency decided. While the context domains and levels are likely to apply to most studies, the evidence-based determinants used to pre-populate each cell will vary.
The Context Tracker is oriented primarily at meso (community) and micro (peer group) levels, as health interventions for marginalised populations often target local settings to create an enabling environment for change [26, 56]. Approaches at this level include increasing supply of resources (condoms, clean injecting equipment), improving interactions between marginalised communities and health authorities (sensitising medical staff, offering outreach and peer-led services) and catalysing community mobilisation to strengthen collective action on shared priorities (supporting peer leadership and organisational development). There is strong evidence for the effectiveness of peer-led and community-based interventions for criminalised populations, mostly on HIV-related outcomes [30, 57,58,59]. However, many attempts to replicate successful programmes in new settings have been unsuccessful [26, 60], suggesting contextual attributes still need to be better identified, measured and addressed.
As mentioned, the original tool consisted of an Excel spreadsheet with rows for sites, columns for time periods, and an accompanying interview topic guide as a job aid. The formatting can be adjusted, for instance allocating each site a separate Excel tab with rows and columns recording relevant information at micro, meso and macro levels for each domain. Ideally, the Context Tracker should become a living document to open interactive space for interviewer (social scientist) and interviewees to maintain a regular dialogue on contextual factors. Where possible, several respondents should be consulted to triangulate findings (in AMETHIST, these were outreach and peer staff and FSW participants). It should remain both structured (with set domains and interview schedule) and flexible (guided by iterative questions and allowing for addition of domains when required). The spreadsheet has an element of “built in” charting of the data, facilitating framework analysis for comparison between cases (e.g. sites) and identification of trends over time [45, 61]. Emerging findings can also be fed back into the intervention for course-correction and appropriate adaptations made to intervention delivery, as recommended by the original MRC Guidance on process evaluation [2].
Adaptation and use in SIPP
COVID-19 restrictions were no longer in place by the time of the SIPP study, making it possible to adapt the COVID Tracker for in-person qualitative data collection, including extensive field observation. The research team chose to use a word document rather than Excel spreadsheet for greater ease of inserting long excerpts of written text, with a separate table for each of the 6 study sites (3 intervention and 3 comparison). There was a separate row for each domain, with columns for “micro” and “meso” levels. As all the sites are within the UK, we did not expect national-level (macro) differences. Two further columns were added, “Relevant to intervention (fidelity, feasibility, acceptability)” and “Related to research activities (adherence to study protocol, potential contamination)”. These additional columns allowed for capturing information that seemed directly linked to delivering the intervention or completing the study as planned, for example, where local peer outreach workers appeared to act as “gatekeepers”, limiting the study’s access to some community members, or where there were breakdowns of communication between the research team and implementors that had to be addressed to maintain access to data (see Table 3 for examples). Rather than use an accompanying topic guide, the SIPP research team integrated topics from the Context Tracker into iterative development of other data collection instruments, mostly interviews, focus groups, field notes and extraction from available programme or other local statistics (e.g. local prevalence rate for hepatitis C and other infections among people who use crack).
The SIPP evaluation is still underway, and the Context Tracker updated on a regular basis. Analysis of the data is conducted in real time but will also be examined in parallel with endline survey data on uptake and engagement with the intervention, and qualitative interview and field note data. To date, some of the findings related to different environmental features likely to interact with the underlying SIPP rationale include the fact that in some sites, sex work is closely associated with crack use (“economic” domain), contributing to a different epidemiological and risk profile among the target participants, and differences in availability of public transport, making travel to the drug services easier and less expensive in some sites compared to others (“physical” domain). Furthermore, levels of social cohesion as expressed through whether people who use crack consider there to be a “sense of community” and trust between peers varied across intervention sites (“social” domain). For example, the death of the director of a local service led to power struggles and disagreements on their whole approach to drug treatment; the effects of this disruption on SIPP implementation at that site was captured in real time through the Context Tracker. Finally, we documented how the intervention itself appeared to influence contextual factors such as policing practice, i.e. we documented how in sites that had positive relationships between implementors and the police, people who use crack were newly able to keep their pipes even when arrested for drug use (“policy/legal/health” system domain). Table 3 provides an extract from the SIPP use of the tool.
Discussion
“Context” has been highlighted as an important determinant of how an intervention is delivered, perceived, received and responded to, with growing attention to ensuring it is better defined and measured in process evaluations and implementation science [4, 62,63,64,65]. Numerous conceptual frameworks now exist, and these can be refined and structured in advance for any specific trial based on existing literature. The Context Tracker is one example of how evidence-based contextual determinants drawn from the harm reduction literature can be used to guide data collection and analysis, in this case, for studies targeting marginalised populations that share barriers and facilitators to health interventions despite their considerable diversity.
While the tool itself may prove useful to others planning process evaluations of complex interventions for “key populations” or other marginalised groups, we consider the broader approach likely to have useful applications for a more general research audience. The Context Tracker demonstrates how to pre-plan integration of context measures into process evaluation using any relevant framework combined with existing evidence and prior experience. While we selected the CICI and Risk Environments frameworks, other models would also be useful. A recent scoping review identified 17 separate frameworks that include contextual determinants [4]. For example, Normalisation Process Theory focuses on factors to consider when moving from testing interventions to embedding them into routine practice [66]. The Consolidated Framework for Implementation Research designates “outer” and “inner” settings [62], while Promoting Action on Research Implementation in Health Services (PARiHS) considers “culture”, “leadership” and “evaluation” as the primary dimensions of the environment into which an intervention is introduced [67]. These and other frameworks offer useful conceptual approaches to researchers, but they remain abstract. Combining them with use of the Context Tracker or an adapted version of it could help translate these frameworks into useful data collection strategies “on the ground”.
At the time that we developed the Context Tracker, we had not read the approach described by Craig et al. [68] that offers a useful categorisation of possible domains. This guideline makes clear that “the features of context that need to be taken into account are intervention specific: not every aspect of context is relevant in every case” (p. 6). It suggests that there is less value in rigid standardisation than early consideration of which features are likely to interact with any given intervention. Familiarity with existing literature and the proposed intervention’s underpinning logic model should enable selection of relevant factors. Despite calls for greater consistency in terminology by others [4, 5], we agree with Craig et al. that the aim should not be to standardise conceptualisation of context, but to encourage early reflection during the design of process evaluation measures for each specific intervention study, thus strengthening the quality of data on context. We hope to expand on the utility of Craig et al.’s guidance by offering greater detail on the mechanics of moving from identification of relevant domains to operationalising them during fieldwork.
A strength of the Context Tracker is that it was initially developed by necessity, when circumstances beyond the control of a research team mandated not just a change of data collection method, but a conceptual re-think for how to capture real-time effects of COVID-19 restrictions on both AMETHIST implementation and research activities. This led to an action-oriented solution rooted in the experience of field work in challenging settings. Because the Context Tracker originated in Zimbabwe, it is responsive to realities of conducting research in resource constrained settings, often marked by poor infrastructure, low internet connectivity and unreliable transport. Potential weaknesses include using the tool too rigidly, without adjusting it for different interventions. It is also possible to get overly bogged down in the categorisation of different shapers as “micro” or “meso”, “socio-economic” or “policy” which could lead to unproductive theoretical debates rather than useful guidance in the field. Finally, while designed to facilitate the process of collecting data and simplify its presentation, the Context Tracker in no way reduces the time and effort required by meaningful data analysis and interpretation and would be misused if considered a “short cut” to considering how interventions are embedded in and interact with dynamic social settings.
We intend the Context Tracker to be relevant in low- and high-income settings and applicable to both qualitative and quantitative analyses. While the data collected can be open-ended text conducive to thematic analysis in smaller-scale and more ethnographic research, its database format allows for prespecifying a limited number of contextual variables for each domain, facilitating quantitative analysis as well as categorisation of contextual attributes useful for innovations in statistical analysis of process evaluation data, e.g. moderated mediation analysis [69]. Although initially introduced to overcome the challenges of fieldwork under COVID-19, the adapted and refined Context Tracker has potential to improve quality and robustness of process evaluations conducted both in more predictable and stable circumstances, but can remain useful should a pandemic or other disruption occur requiring rapid transition to remote data collection.
Conclusions
While the use of Theory of Change and similar logic models for complex health interventions has helped improve description of interventions and their underlying mechanisms of action, much of the important role of “context” in shaping and explaining what happens in regard to being able to trigger those mechanisms remains opaque in many studies. The accumulation of useful concepts and frameworks can seem overwhelming and abstract. The Context Tracker offers a novel approach to designing and using a tool within process evaluations that provides some structure without removing the potential for iterative analysis and flexibility to follow unexpected findings. The primary implication for future research is availability of a tool for capturing relevant contextual domains in intervention studies in a more rigorous and systematic way, ensuring consistency in measures over time. This will facilitate the difficult task of interpreting the diverse ways that local environments interact with newly introduced programme components, creating facilitators and barriers to implementation, affecting engagement and uptake, and triggering both hypothesised and unexpected mechanisms of action.
Data availability
Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.
Abbreviations
- AMETHIST:
-
Adapted Microplanning to Eliminate Transmission of HIV in Sex Transactions
- CICI:
-
Context and Implementation of Complex Interventions
- FSW:
-
Female sex worker
- HIV:
-
Human immunodeficiency virus
- MRC:
-
Medical Research Council
- PARiHS:
-
Promoting Action on Research Implementation in Health Services
- PrEP:
-
Pre-exposure prophylaxis
- RCT:
-
Randomised controlled trial
- SHG:
-
Self Help Group
- SIPP:
-
Safe Inhalation Pipe Provision
References
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.
Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.
Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14(1):15.
Nilsen P, Bernhardsson S. Context matters in implementation science: a scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Serv Res. 2019;19(1):189.
Rogers L, De Brún A, McAuliffe E. Defining and assessing context in healthcare implementation studies: a systematic review. BMC Health Serv Res. 2020;20(1):591.
Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374:n2061.
Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139.
Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.
Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implement Sci. 2017;12(1):21.
Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation. 2016;22(3):286–303.
Jack K. What is realist evaluation? Evid Based Nurs. 2022;25(4):111.
Greenhalgh J, Manzano A. Understanding ‘context’ in realist evaluation and synthesis. Int J Soc Res Methodol. 2022;25(5):583–95.
Murdoch J, Paparini S, Papoutsi C, James H, Greenhalgh T, Shaw SE. Mobilising context as complex and dynamic in evaluations of complex health interventions. BMC Health Serv Res. 2023;23(1):1430.
Shoveller J, Viehbeck S, Di Ruggiero E, Greyson D, Thomson K, Knight R. A critical examination of representations of context within research on population health interventions. Crit Public Health. 2016;26(5):487–500.
Grant A, Bugge C, Wells M. Designing process evaluations using case study to explore the context of complex interventions evaluated in trials. Trials. 2020;21(1):982.
Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11(1):63.
De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C, et al. Theory of change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials. 2014;15(1):267.
Ebenso B, Manzano A, Uzochukwu B, Etiaba E, Huss R, Ensor T, et al. Dealing with context in logic model development: reflections from a realist evaluation of a community health worker programme in Nigeria. Eval Program Plann. 2019;73:97–110.
Burchett HED, French RS, Griffin S, de Melo M, Joaquim Picardo J, Kneale D. Developing a theory of change – the importance of rich process data and authors’ insights into context, implementation and mechanisms. Glob Health Promot. 2024;0(0). https://doiorg.publicaciones.saludcastillayleon.es/10.1177/17579759241232387.
Bonell C, Jamal F, Melendez-Torres GJ, Cummins S. ‘Dark logic’: theorising the harmful consequences of public health interventions. J Epidemiol Community Health. 2015;69(1):95.
Chavula MP, Zulu JM, Hurtig AK. Factors influencing the integration of comprehensive sexuality education into educational systems in low- and middle-income countries: a systematic review. Reprod Health. 2022;19(1):196.
Hurst JL, Widman L, Brasileiro J, Maheux AJ, Evans-Paulson R, Choukas-Bradley S. Parents’ attitudes towards the content of sex education in the USA: associations with religiosity and political orientation. Sex Education. 2024;24(1):108–24.
WHO. Consolidated guidelines on HIV prevention, diagnosis, treatment and care for key populations. Geneva: World Health Organization; 2014.
Ippoliti NB, Nanda G, Wilcher R. Meeting the reproductive health needs of female key populations affected by HIV in low- and middle-income countries: a review of the evidence. Stud Fam Plann. 2017;48(2):121–51.
Golub SA, Fikslin RA. Recognizing and disrupting stigma in implementation of HIV prevention and care: a call to research and action. J Int AIDS Soc. 2022;25(S1):e25930.
Makofane K, Lusimbo R, Macharia P, Makanjuola O, Malone S, Mokabedi A, et al. Understanding community as a basis for targeting and shaping service delivery. J Int AIDS Soc. 2021;24(S3):e25712.
Small W, Rhodes T, Wood E, Kerr T. Public injection settings in Vancouver: physical environment, social context and risk. Int J Drug Policy. 2007;18(1):27–36.
Rácz J, Gyarmathy VA, Neaigus A, Ujhelyi E. Injecting equipment sharing and perception of HIV and hepatitis risk among injecting drug users in Budapest. AIDS Care. 2007;19(1):59–66.
Uuskula A, Fischer K, Raudne R, Kilgi H, Krylov R, Salminen M. A study on HIV and hepatitis C virus among commercial sex workers in Tallinn. Sex Transm Infect. 2008;84:189–91.
Vannakit R, Janyam S, Linjongrat D, Chanlearn P, Sittikarn S, Pengnonyang S, et al. Give the community the tools and they will help finish the job: key population-led health services for ending AIDS in Thailand. J Int AIDS Soc. 2020;23(6):e25535.
Yang X, Fang T, Mobarak SAI, Wang J, Wang C, Huang S, et al. Social network strategy as a promising intervention to better reach key populations for promoting HIV prevention: a systematic review and meta-analysis. Sex Transm Infect. 2020;96(7):485.
Haldane V, Chuah FLH, Srivastava A, Singh SR, Koh GCH, Seng CK, et al. Community participation in health services development, implementation, and evaluation: a systematic review of empowerment, health, community, and process outcomes. PLoS One. 2019;14(5):e0216112.
Cyril S, Smith BJ, Possamai-Inesedy A, Renzaho AM. Exploring the role of community engagement in improving the health of disadvantaged populations: a systematic review. Glob Health Action. 2015;8:29842.
Cowan FM, Machingura F, Chabata ST, Ali MS, Busza J, Steen R, et al. Differentiated prevention and care to reduce the risk of HIV acquisition and transmission among female sex workers in Zimbabwe: study protocol for the ‘AMETHIST’ cluster randomised trial. Trials. 2022;23(1):209.
Cowan FM, Machingura F, Ali MS, Chabata ST, Takaruza A, Dirawo J, et al. A risk-differentiated, community-led intervention to strengthen uptake and engagement with HIV prevention and care cascades among female sex workers in Zimbabwe (AMETHIST): a cluster randomised trial. Lancet Glob Health. 2024;12(9):e1424–35.
Harris M, Scott J, Hope V, Busza J, Sweeney S, Preston A, et al. Safe inhalation pipe provision (SIPP): protocol for a mixed-method evaluation of an intervention to improve health outcomes and service engagement among people who use crack cocaine in England. Harm Reduct J. 2024;21(1):19.
Hawk M, Coulter RWS, Egan JE, Fisk S, Reuel Friedman M, Tula M, et al. Harm reduction principles for healthcare settings. Harm Reduct J. 2017;14(1):70.
Ramprashad A, Burnett GM, Welsh C. Harm reduction: not dirty words any more. Psychiatr Clin North Am. 2022;45(3):529–46.
Adams A, Ferguson M, Greer AM, Burmeister C, Lock K, McDougall J, et al. Guideline development in harm reduction: considerations around the meaningful involvement of people who access services. Drug Alcohol Depend Rep. 2022;4:100086.
Greer AM, Luchenski SA, Amlani AA, Lacroix K, Burmeister C, Buxton JA. Peer engagement in harm reduction strategies and services: a critical case study and evaluation framework from British Columbia. Canada BMC Public Health. 2016;16:452.
Harris M. An urgent impetus for action: safe inhalation interventions to reduce COVID-19 transmission and fatality risk among people who smoke crack cocaine in the United Kingdom. Int J Drug Policy. 2020;83:102829.
Matambanadzo P, Busza J, Mafaune H, Chinyanganya L, Machingura F, Ncube G, et al. “It went through the roof”: an observation study exploring the rise in PrEP uptake among Zimbabwean female sex workers in response to adaptations during COVID-19. J Int AIDS Soc. 2021;24(S6):e25813.
Bhala N, Curry G, Martineau AR, Agyemang C, Bhopal R. Sharpening the global focus on ethnicity and race in the time of COVID-19. Lancet. 2020;395(10238):1673–6.
HIV The Lancet. Lockdown fears for key populations. Lancet HIV. 2020;7(6):e373.
Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):117.
Rhodes T. Risk environments and drug harms: a social science for harm reduction approach. Int J Drug Policy. 2009;20:193–201.
Leddy AM, Underwood C, Decker MR, Mbwambo J, Likindikoki S, Galai N, et al. Adapting the risk environment framework to understand substance use, gender-based violence, and HIV risk behaviors among female sex workers in Tanzania. AIDS Behav. 2018;22(10):3296–306.
Baral S, Logie CH, Grosso A, Wirtz AL, Beyrer C. Modified social ecological model: a tool to guide the assessment of the risks and risk contexts of HIV epidemics. BMC Public Health. 2013;13(1):482.
Ma PHX, Chan ZCY, Loke AY. The socio-ecological model approach to understanding barriers and facilitators to the accessing of health services by sex workers: a systematic review. AIDS Behav. 2017;21(8):2412–38.
Sherman SG, Tomko C, Silberzahn BE, White RH, Nestadt DF, Clouse E, et al. The role of local business employees and community members in the HIV risk environment of female sex workers in an urban setting: associations between negative interactions and inconsistent condom use. BMC Public Health. 2021;21(1):2265.
Maleke K, Makhakhe N, Peters RP, Jobson G, De Swardt G, Daniels J, et al. HIV risk and prevention among men who have sex with men in rural South Africa. Afr J AIDS Res. 2017;16(1):31–8.
Glick JL, Russo R, Jivapong B, Rosman L, Pelaez D, Footer KHA, et al. The PrEP care continuum among cisgender women who sell sex and/or use drugs globally: a systematic review. AIDS Behav. 2020;24(5):1312–33.
Wilson EC, Santos GM, Raymond HF. Sexual mixing and the risk environment of sexually active transgender women: data from a respondent-driven sampling study of HIV risk among transwomen in San Francisco, 2010. BMC Infect Dis. 2014;14:430.
Sarat A, Hail-Jares K, Leon C, Shdaimah C. Special Issue: Problematizing Prostitution: Critical Research and Scholarship: Bingley: Emerald Group Publishing Limited; 2016.
Sanders T. Inevitably violent? Dynamics of space, governance, and stigma in understanding violence against sex workers. In: Sarat A, editor. Special issue: problematizing prostitution: critical research and scholarship. Studies in law, politics and society, 71. Bingley: Emerald Group Publishing Limited; 2016. p. 93–114.
DiCarlo MC, Dallabetta GA, Akolo C, Bautista-Arredondo S, Digolo HV, Fonner VA, et al. Adequate funding of comprehensive community-based programs for key populations needed now more than ever to reach and sustain HIV targets. J Int AIDS Soc. 2022;25(7):e25967.
Fuller CM, Galea S, Caceres W, Blaney S, Sisco S, Vlahov D. Multilevel community-based intervention to increase access to sterile syringes among injection drug users through pharmacy sales in New York City. Am J Public Health. 2007;97(1):117–24.
Argento E, Reza-Paul S, Lorway R, Jain J, Bhagya M, Fathima M, et al. Confronting structural violence in sex work: lessons from a community-led HIV prevention project in Mysore, India. AIDS Care. 2011;23(1):69–74.
Erausquin J, Reed E, Blankenship K. Change over time in police interactions and HIV risk behavior among female sex workers in Andhra Pradesh, India. AIDS Behav. 2015;19:1108.
Reed JB, Shrestha P, Were D, Chakare T, Mutegi J, Wakhutu B, et al. HIV PrEP is more than ART-lite: longitudinal study of real-world PrEP services data identifies missing measures meaningful to HIV prevention programming. J Int AIDS Soc. 2021;24(10):e25827.
Schleiff MJ, Kuan A, Ghaffar A. Comparative analysis of country-level enablers, barriers and recommendations to strengthen institutional capacity for evidence uptake in decision-making. Health Res Policy Syst. 2020;18(1):78.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.
Wells M, Williams B, Treweek S, Coyle J, Taylor J. Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials. 2012;13:95.
Howarth E, Devers KJ, Moore GF, O'Cathain A, Dixon-Woods M. Contextual issues and qualitative research. In: Raine R, Fitzpatrick R, Barratt H, editors. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. 2016.
Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health. 2017;3:132–5.
May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. 2009;43(3):535–54.
Bergström A, Ehrenberg A, Eldh AC, Graham ID, Gustafsson K, Harvey G, et al. The use of the PARIHS framework in implementation research and practice-a citation analysis of the literature. Implement Sci. 2020;15(1):68.
Craig P, Di Ruggiero E, Frohlich KL, Mykhalovskiy E, White M, Group obotCIoHRCNIfHRNCGA. Taking account of context in population health intervention research: guidance for producers, users and funders of research. Southampton: NIHR Evaluation, Trials and Studies Coordinating Centre; 2018.
Melendez-Torres GJ, Warren E, Viner R, Allen E, Bonell C. Moderated mediation analyses to assess intervention mechanisms for impacts on victimisation, psycho-social problems and mental wellbeing: evidence from the INCLUSIVE realist randomized trial. Soc Sci Med. 2021;279:113984.
Acknowledgements
We would like to thank all the researchers and participants of the AMETHIST and SIPP studies who helped implement the Context Tracker and contributed to the data collected. We also thank Mark Petticrew, Mitzy Gafos and Sue Napierala for reviewing an early draft of the paper.
Funding
The AMETHIST trial research is funded by the Wellcome Trust (Grant number: 214280/Z/18/Z). The SIPP study is funded by the National Institute for Health and Care Research (NIHR 133118).
Author information
Authors and Affiliations
Contributions
JB conceived and drafted the paper and was co-investigator of both AMETHIST and SIPP evaluation studies. FM led the AMETHIST study process evaluation and developed the initial Context Tracker. CV led the SIPP study process evaluation, including adaptation and application of the Context Tracker. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The AMETHIST study received ethical approval from the Medical Research Council of Zimbabwe (MRCZ/A/2559), Liverpool School of Tropical Medicine (19-115RS) and London School of Hygiene and Tropical Medicine (19123). The SIPP study received ethical approval from the London School of Hygiene & Tropical Medicine Research Ethics Committee (28102).
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Please note that author Cedomir Vuckovic has changed affiliation since the original submission. At the time of the study described, he was Research Fellow at the London School of Hygiene and Tropical Medicine and his role is described in the paper.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Busza, J., Machingura, F. & Vuckovic, C. Improving measures of context in process evaluations: development and use of the Context Tracker tool. Trials 25, 777 (2024). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s13063-024-08623-7
Received:
Accepted:
Published:
DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s13063-024-08623-7