Clinical Study Reports—a systematic review with thematic synthesis: Part 2. Studying benefits, harms, and the benefit to harm balance of pharmacological interventions

Abstract

Background

We define clinical study reports (CSRs) as standardized full reports of the protocols, results, and other pertinent details of clinical studies that are typically submitted by pharmaceutical companies to regulatory authorities when they apply for marketing authorization.

Methods

In this systematic review we searched various databases (Clarivate Web of Science, EMBASE and Ovid Medline, Google Scholar, and PubMed) for publications containing the term “clinical study report/s”, without restrictions.

Thematic synthesis

In the first part of this review we discussed the history of CSRs, their contents and structure, definitions, and relevant terminology. In this second part we discuss the uses of CSRs, concentrating on the individual benefits and harms of pharmacological interventions, and thus the benefit to harm balance. We also discuss adherence to interventions, prepublication of protocols of clinical trials, and how CSRs are written, factors that can all affect estimation of the benefit-harm balance.

Conclusions

When clinical trial data from CSRs are compared with the data in published trial reports, the apparent benefits of pharmacological interventions are less impressive, and more information emerges about harms they can cause. Both of these effects change how the benefit-harm balance of a pharmacological intervention is estimated, generally making it less favourable than was otherwise thought.

For more accurate assessment of the benefit-harm balance of an intervention, full, not abbreviated or synoptic, clinical study reports should continue to be made publicly available by regulatory authorities and manufacturers. Authorities that do not currently make them available should do so.

CSRs should be introduced for assessment of surgical operations, therapeutic devices, and other non-pharmacological interventions in clinical trials.

Background

As users of clinical study reports (CSRs), which provide more comprehensive data on clinical trial protocols and results than publicly available accounts of clinical trials, as published in peer-reviewed journals [1, 2], we are interested to know how often they have been and are being used for research, concentrating on systematic reviews and meta-analyses of the benefits, harms, and the benefit-to-harm balance of pharmacological interventions. We are also interested in knowing about the factors that can affect the benefit-harm balance, such as cost-effectiveness, adherence to the intervention, the quality of reports and how they are written, and the extent of adherence to prepublished protocols. We have therefore carried out a systematic review of publications in which clinical study reports have been mentioned.

In the first part of this review [3] we discussed the history of CSRs, their contents and structure, definitions, and terminology relevant to the availability of CSRs, data sharing, and transparency and confidentiality. In this second part we discuss the uses to which CSRs have been put, especially their use in studying benefits and harms of pharmacological interventions and the benefit-harm balance. We also discuss other factors that can influence estimation of the benefit-harm balance.

Methods

The protocol for this study is available at https://osf.io/pr7fj/.

Following a scoping review in February 2024 we searched a range of databases (Clarivate Web of Science, EMBASE and Ovid Medline, Google Scholar, and PubMed) in April for publications containing the term “clinical study report/s”, without restrictions. In other words, we included any paper that dealt with any aspect of the use of CSRs, without introducing other search terms that would have restricted the search. We also included papers listed in reference lists that were not found by our database searches. The PRISMA flow chart for the searches is shown in Fig. 1. We screened the papers by reading their titles and abstracts and when necessary the full papers. JKA did the original search and IJO checked it independently.

Fig. 1

PRISMA flow chart

Having compiled a list of all papers that dealt in any way with CSRs, we categorized each publication according to whether it dealt directly with benefits, harms (adverse events, adverse effects, or adverse reactions), or the benefit-to-harm balance of a pharmacological intervention or group of interventions. We also categorized the publications according to their emphasis on factors that can affect the benefit-harm balance—adherence, the cost-effectiveness of the intervention, the quality of the CSRs and how they are written, and the extent to which prepublished protocols are used and adhered to.

Deduplication of 1248 hits resulted in 700 publications, of which we excluded 351 that did not deal with CSRs (for example, irrelevant hits that stated in the publication that “this clinical study reports [that]”) or for other reasons (see Fig. 1). This left 349 publications to be analysed. The numbers of each category of publication, including exclusions, are shown in Fig. 1. The cumulative numbers of publications with time are shown in Fig. 2(a).

Fig. 2

The timelines of three categories of publication: a the complete corpus (n = 347, omitting two papers from 2024); b the papers on benefits, harms, and adverse events (n = 94); c the commentaries (n = 79); the arrows mark (1) the date of publication of ICH E3 and (2) the year in which the EMA announced that it intended to make CSRs generally available

We originally intended to categorize each publication according to our primary focus, i.e. how the availability of CSRs affected assessments of the benefits, harms, and the benefit-harm balance of pharmacological interventions, choosing the category that best described the stated main concern of the authors of the publication (Group A in Fig. 1). However, we also found other endpoints in which researchers had been interested, and we have categorized them, according to the authors’ statements of what they had studied, as factors that affect assessment of the benefit-harm balance (Group B) and other topics (Group C).

The complete list of publications we have included is given in the Appendix (Supplementary Material 1).

Thematic analysis

Benefits and harms of pharmacological interventions and the benefit-harm balance

All therapeutic decisions depend on assessment of one or more competing courses of action, all associated with some hazard (that is, potential harm). Three factors affect the benefit-harm balance: an intervention may (a) bring benefit but can also (b) cause harm, and (c) failure to intervene may also be associated with harms. Clinicians therefore have to weigh both the chances of benefits and the competing hazards that carry risks of harms when they consider the overall benefit-harm balance, and when they tell patients about possible courses of action [4].

In all, 59 publications were devoted to accounts of analyses of either benefits alone (n = 27) or both benefits and harms (n = 32) of pharmacological interventions during drug development. Added to the studies on adverse events during trials, discussed below, the total is 94 publications.

Most of the studies targeted a specific medicine or medicines for comparison or a group of medicines, as listed in Table 1.

Table 1 Pharmacological interventions for which CSRs have reported benefits and harms of pharmacological interventions in clinical trials

Figure 2(b) shows the cumulative time-course of publications of the 94 publications on benefits, harms, and adverse events, as well the cumulative time-course of 347 publications in the review in Fig. 2(a) (omitting two from 2024). The numbers of such studies started to increase in 2014, coincident with the EMA’s announcement that it intended to make CSRs available on its website.

Some examples illustrate the need to analyse data from CSRs in assessing the benefit to harm balance of a pharmacological intervention:

• In a comparison of adverse events, serious adverse events, and the reporting of 15 harms criteria in five journal publications with matching CSRs, the CONSORT Harms reporting criteria were satisfied in the methods section of the CSRs more often (70–90%) than in the journal publications (10–50%) [2].

• In a systematic review and meta-analysis of the results of double-blind, randomized, controlled trials of acute treatment for 6 weeks or more with reboxetine versus placebo or SSRIs in adults with major depression, including data from clinical trial registries, trial results databases, and regulatory authority websites, and unpublished data from the manufacturers, the published data overestimated the benefit of reboxetine versus placebo by up to 115% and reboxetine versus SSRIs by up to 23%; harms were also underestimated [5]. The authors concluded that reboxetine was ineffective and potentially harmful.

• When the published results of a trial of epoetin in 1265 patients with cardiac disease receiving haemodialysis were compared with the results in the CSRs, it was found that had the latter been disclosed in the public source, which they were not, concerns about the apparent safety of epoetin and greater doubts about its benefits would have been raised sooner than they actually were [6].

• After the manufacturers of duloxetine had been granted marketing authorization for stress urinary incontinence in women by the EMA, a meta-analysis of the CSRs (6870 pages) of four randomized placebo-controlled trials of duloxetine in 1913 patients, the results of which had been submitted to the Agency, showed that the benefit to harm balance was in fact unfavourable [7].

The number of these studies of benefits and harms is disappointing. A PubMed search from 1995 to 2023, using the “systematic review”, “meta-analysis”, and “randomized controlled trial” filters, yields over 900,000 hits. If even only 10% of those dealt with pharmacological interventions, the proportion that involved analysis of CSRs for benefits and/or harms would be about one per 1000.

Adverse events during clinical trials

We define an adverse event as follows [8]: any abnormal sign, symptom, laboratory test, syndromic combination of such abnormalities, untoward or unplanned occurrence (e.g. an accident or unplanned pregnancy), or any unexpected deterioration in a concurrent illness.

There is no necessary implication in this definition that an adverse event was due to the intervention; it may or may not have been. If it is thought to have been due to the intervention it is called a suspected adverse reaction. A harm, whether due to the intervention or not, may not be detectable by the individual (e.g. neutropenia or a pronged QT interval on an electrocardiogram), in which case, if it is attributable to the intervention, it is called an adverse effect; alternatively, the individual may experience a sign or symptom attributable to the intervention, in which case it is called an adverse reaction [9].

Of 35 reports of the use of CSRs to study adverse events during clinical trials, 27 dealt with adverse events during trials of 23 different specific medicines or types of medicines. These are listed in Table 2, and some examples illustrate the need to analyse adverse events using individual patient data from CSRs rather than summarized data from published sources.

Table 2 Pharmacological interventions for which CSRs have reported adverse events in clinical trials; *combinations of a long-acting muscarinic cholinoceptor antagonist with a long-acting β2-adrenoceptor agonist

• In a comparison between CSRs and published trials of gabapentin and quetiapine, all the CSRs reported all adverse events, while no other sources did, and 22% of the gabapentin sources and 40% of the quetiapine sources reported using selection criteria [10]. Selection criteria, such as "occurring in ≥ 2% [or ≥ 10%] of participants" greatly affected the number of adverse events reported. The authors concluded that selection criteria should be specified in advance to avoid later cherry-picking of results, and even then that using selection criteria would introduce bias into meta-analyses and clinical practice guidelines.

• In a study of 42 RCTs of 13 different drugs used in oncology, the results were available in CSRs in 37 cases, in ClinicalTrials.gov for 36, in the European Clinical Trials Register (EUCTR) for 20, and in journal publications for 32 [11]. Reporting of adverse events was more complete in CSRs than in the other sources. There were also different delays between the primary trial completion date and access to the results: 2.2 (0.64–5.0) years for journal publications; 2.9 (1.2–4.5) years for ClinicalTrials.gov; 4.3 (3.1–7.2) years for CSRs; and 5.4 (4.2–7.3) years for EUCTR.

• After the EMA had declared in 2015 that there was no link between the use of HPV vaccines and serious neurological adverse events, relying on data provided by the drug companies, a systematic review based on CSRs showed that there were significantly more serious neurological harms in the HPV vaccine groups than in the comparator groups [12].

The advantages of independent analysis and reporting of results from CSRs as early as possible, exemplified by the results from these and other studies [1, 10, 13], are obvious.

In one case the structure of CSRs for drug studies was adapted to study adverse events during total knee arthroplasty [14], providing an indirect reminder that CSRs have not yet been introduced into the regulation of devices.

These studies adduced data from studies at all phases of drug development, from phase 1 to phase 4, including phase 1 dosage determination to reduce the risk of adverse events in later phases [15]. However, they were mainly concerned with phases 2 and 3.

Adverse events of special interest (AESIs) are a subset of adverse events that may especially affect the benefit to harm balance, individual patient safety, or public health in general. One would have expected such events to be discussed in CSRs. However, AESIs were not mentioned in any of the studies that we found in our searches.

The benefit to harm balance: the quality of CSRs, publication bias, and other biases

Estimating the benefit to harm balance is not always easy, and is complicated by the fact that publicly available publications describing the results of clinical trials are abbreviated accounts. Adducing the complete results from CSRs often results in an estimate of the benefit to harm balance that is less favourable, as illustrated by the cases described above.

The quality of reports is an important factor in these assessments. It has been suggested that the two most important elements that determine the quality of reports are correctness and completeness [16]. When those are neglected in publicly available publications or CSRs, there is an increased risk of bias in the reporting of trial outcomes. This can lead to a false estimate of the benefit-harm balance.

We found 39 publications whose main concern was some aspect of the quality of publicly available publications and CSRs; many were concerned with the problem of bias, generally by comparing the results of studies as found in the two different types of publication. Some studies that were categorized under other headings did likewise. In all, we found 38 studies that would be suitable for including in a separate systematic review, with possible meta-analysis.

The sources of biases that have been reported include multiple publication, selective publication, and selective reporting; the extent to which these techniques have been used differs between products [17].

For example, in a 2013 comparison of the full CSRs from 101 trials, provided by pharmaceutical companies on request, with their publicly available sources, the CSRs provided complete information on statistically significantly more outcomes than the combined publicly available sources (86% versus 39%, P < 0.001) [18]. Excluding health-related quality of life (57%), CSRs provided complete information on 78–100% of the various beneficial outcomes, while the combined publicly available sources provided only 20–53%. The CSRs also provided considerably more information on harms.

In most cases, when data from CSRs were taken into account the apparent benefits were less impressive and the adverse events more frequent. In other words, the benefit-harm balance became less favourable. Typically, assessments of "unclear" or "low" risk of bias are reclassified as "high" risk of bias when judgements are based on full CSRs, while no previous assessments of "high" risk of bias are reclassified as 'low" or "unclear" [19].

In one study, comparison of the CSRs and the publicly available publications showed that in some cases protocol-defined primary endpoints were either not reported at all or were changed to secondary endpoints, introducing potential biases [20].

There was also a mismatch between the information given in patient consent forms in 50 trials and the more extensive information given in CSRs [18]. Although every consent form included a section on the purpose of the study, patients considered that only 11 did so adequately, seven accurately, and four inaccurately compared with the reference standard.

Cost-effectiveness

The primary criteria to consider when judging the cost-effectiveness of an intervention are that it should be effective and that the benefit-harm balance should be favourable. Since publicly available evidence often overestimates the true efficacy of an intervention, one would expect that cost-effectiveness analysis involving CSRs would be preferred. However, we found only eight relevant publications, of which only six described cost-effectiveness analysis involving CSRs (biologics for psoriasis, letermovir for prophylaxis of cytomegalovirus infection, oxycodone for chronic pain, oxygen for pneumonia in children, tacrolimus for atopic dermatitis, and tyrosine kinase inhibitors in chronic myelogenous leukaemia). In no case was there a comparison between the information obtained from publicly available publications and CSRs. This question remains to be answered.

Other endpoints

Other endpoints that we had not originally intended to study, but which emerged from our searches, are listed in Fig. 1. All of them in some way affect estimation of the benefit to harm balance.

Adherence to interventions: the effect on the benefit-harm balance

Poor adherence to a trial intervention impairs the value of the trial in several ways [21], leading, among other things, to inaccurate estimation of efficacy, underestimation of the risk of harms, and hence inaccurate estimation of the benefit-harm balance.

In many cases clinical trials of pharmacological interventions published in leading medical journals report adherence, but not how it was assessed [22]. We found only one study in which CSRs were used to assess adherence [23]: of 253 clinical trials, all but one included some measurement of adherence, 87% used quantitative methods, and 13% monitored adherence but did not quantify it. There was reportedly greater than 90% adherence across the trials, but there were marked disparities in measurement methods and definitions of adherence. Pill/dose counting on its own, which is unreliable, was used in 53%; more reliable electronic methods and bioanalytical methods were used in only 5.5%.

In two other studies adherence was indirectly studied by looking at trial completion rates and drop-out rates [24, 25].

How to write CSRs

Failure to communicate the results of a CSR can affect the way in which it is interpreted, which in turn affects estimation of the benefit to harm balance.

We found 11 publications that deal in some way with writing CSRs, of which one is worth highlighting. In contrast to the various guidelines on the contents and structure of CSRs, which we discussed in Part 1 of this review, the document called CORE Reference (Clarity and Openness in Reporting, which is based on ICH E3) gives guidance on writing CSRs [26]. In a so-called Terminology Table it defines eight key features (objective, hypothesis, measurement, procedure, assessment/evaluation, variable, endpoint, and estimand, i.e. the thing to be estimated) and gives guidance on the expected contents under those headings. There is also information on how to write a development safety update report (DSUR) [27] and on methods of quality control [28]. Automated systems can be used to ease the burden [29].

We have not (yet) seen a paper describing the use of generative large language models to produce CSRs [30], but doubtless that will come, and there is in fact a preprint describing an evaluation of such models in producing clinical trial documents, highlighting deficiencies in clinical thinking and logic and appropriate use of references [31]. There is also a patent for a system for automatically authoring scientific documents using a machine learning model and natural language processing with minimal intervention by users [32].

Protocols for studies of CSRs

It has become de rigueur to publish prewritten protocols for clinical trials. That is because protocols are often altered during the conduct of a trial, and the ways in which they are altered can affect the ways in which results are collected and analysed, potentially altering estimation of the benefit to harm balance.

We found only 13 papers that we classified as protocols for studies involving CSRs, and 10 others that were grant proposals or descriptions of inventions. Nine of the former declared their intention to use CSRs, either in studies of various pharmacological interventions or in one case a surgical procedure. In another four cases a single study was described and the ambiguous intent to write a “clinical study report” at the end was stated. Other publications that we have included under this heading were descriptions of patents and grant-funded research.

Having discovered these protocols, and although it was not part of our original plan, we also searched the PROSPERO website (https://www.crd.york.ac.uk/PROSPERO) for protocols of studies whose authors intended to incorporate CSRs into their search strategy. We found 56 protocols in which CSRs were mentioned, and in 51 of them the authors declared their intentions to include CSRs in their reviews. Five that were subsequently published were already on our list of included publications. However, in two other cases, the published reports did not mention the use of CSRs, in contrast to the protocols. The other 44 studies have not yet been published, although some have reportedly been completed; in some cases publication may have been delayed by covid-19.

It is encouraging to see that some researchers are beginning to mention CSRs in their protocols, although without the final published reports we cannot say whether the authors’ intentions to collect and use them were eventually fulfilled.

Other publications

Commentaries

Commentaries on different aspects of CSRs form the largest single category of publication in our classification (n = 78), except for the combined papers on benefits, harms, and adverse events (n = 94). Several authors have commented on events as they have evolved, particularly in relation to official documents and the availability or non-availability of CSRs, or the related topics of the quality of the CSRs and the transparency of data. Specific recommendations about the use of CSRs have been highlighted in the Cochrane Handbook for Systematic Reviews of Interventions [33].

It may be that the amount of discussion of the topic, which has outweighed the amount of published evidence, influenced the decisions of regulators after many years to require pharmaceutical companies to make their CSRs publicly available. This view is supported by the timelines shown in Fig. 2: the commentaries were proportionately more frequent between 1999 and 2005 [Fig. 2(c)] than both the papers in the total corpus [Fig. 2(a]) and the combined papers on benefits, harms, and adverse events [Fig. 2(b)]. Their numbers also rose more steeply in 2015, when interest in CSRs started to grow, and since the growth spurt and the introduction of regulatory actions they have tailed off a little more quickly.

Other uses of CSRs

In 28 publications other uses were described besides those detailed above, most of which can affect estimation of the benefit to harm balance. The purposes for which CSRs were used included:

• classification of research protocols [34];

• to study test validation [35];

• to study the pathology of diseases—for example, Alzheimer's disease [36];

• to identify shortcomings of clinical trials [37];

• to study polymorphisms in therapeutic targets [38];

• to study patient-reported outcomes in trials [39];

• to contribute to the information available in the Open Trials database [40];

• to identify standards of care [41];

• to evaluate the efficacy of risk minimization measures [42];

• to evaluate the pharmacokinetics of medicines [43, 44];

• to identify subgroups for potential analysis and hypothesis generation [45]; in three cases other studies were initiated as a result of studies of CSRs;

In two cases CSRs were used to determine the efficacy of acupuncture [46, 47].

Journals

The publications included in this review have appeared in just over 150 different journals and in a few other types of publications, such as official websites and books. About 120 journals have published only one of the papers. Table 3 shows the numbers of papers published in the ten journals that published the most, and the breakdown of all the papers by the most commonly mentioned topics.

Table 3 The ten journals that have published the most papers in the total corpus featuring clinical study reports (e.g. the BMJ has published 29 papers) and the topics they cover (e.g. general medical journals have published 95 papers)

Of course, we cannot tell whether the journals that are publishing few papers on the subject of CSRs, or none, are rejecting them or are not being given the opportunity to do so. Nor have we tried to assess the quality of the papers that have been published. However, it is disappointing, for example, that we have found only seven Cochrane reviews featuring CSRs. Certainly, fewer papers on the subject of CSRs have been published than the subject deserves.

Conclusions

We draw several conclusions from this review, summarized here.

• Consideration of CSRs can alter estimation of the benefit-harm balance of a therapeutic intervention; CSRs should therefore ideally be included when the benefit to harm balance of any therapeutic intervention that has been subjected to clinical trial is being estimated.

• Full clinical study reports should continue to be made publicly available by regulatory authorities and manufacturers, and authorities that do not currently make them available should do so; abbreviated versions should be strongly discouraged, particularly since modern technology is making it easier for pharmaceutical companies to produce full CSRs.

• Any computerized aids (e.g. large language models) used in preparing a CSR should be declared in the preamble to the document.

• Reports should include details of both beneficial outcomes and adverse events/suspected adverse reactions.

• Statements about the intended use or non-use of CSRs should be included in protocols for systematic reviews, as described elsewhere [48].

• All systematic reviews and meta-analyses of pharmacological interventions should ideally include relevant data from CSRs; those who report the results of clinical trials should state whether they have attempted to obtain CSRs; if they have not attempted or have attempted and failed, they should explain why; this could be added to the PRISMA statement, i.e. requiring authors to state whether they have or have not included CSRs in their analysis. This does not imply that they must do so, but would encourage greater use of CSRs.

• CSRs could be extended to trials of pharmacological interventions other than those carried out by pharmaceutical companies and at times other than during drug development; they could also be used to study non-pharmacological interventions, including surgical operations.

• Cost-effectiveness analyses of pharmacological interventions should include relevant data from CSRs; further research is needed on the question of whether, and if so to what extent, the use of CSRs in cost-effectiveness analysis affects the quality of the analysis.

• There is a need for better assessment of adherence to pharmacological interventions in clinical trials.

• CSRs should be introduced into the regulation of medical devices.