Cost-benefit of outcome adjudication in nine randomised stroke trials

Background Central adjudication of outcomes is common for randomised trials and should control for differential misclassification. However, few studies have estimated the cost of the adjudication process. Methods We estimated the cost of adjudicating the primary outcome in nine randomised stroke trials (25,436 participants). The costs included adjudicators’ time, direct payments to adjudicators, and co-ordinating centre costs (e.g. uploading cranial scans and general set-up costs). The number of events corrected after adjudication was our measure of benefit. We calculated cost per corrected event for each trial and in total. Results The primary outcome in all nine trials was either stroke or a composite that included stroke. In total, the adjudication process associated with this primary outcome cost in excess of £100,000 for a third of the trials (3/9). Mean cost per event corrected by adjudication was £2295.10 (SD: £1482.42). Conclusions Central adjudication is a time-consuming and potentially costly process. These costs need to be considered when designing a trial and should be evaluated alongside the potential benefits adjudication brings to determine whether they outweigh this expense.


Introduction
In randomised stroke trials, central adjudication of outcomes is common. 1 Adjudicators are typically blinded independent experts who review individual participant data and provide an assessment of outcome(s). Clinicians at local sites often assess the same outcomes, but these data are commonly discarded. Central adjudication should reduce both random and systematic bias, but studies have shown that it has limited impact on the estimated treatment effect in randomised trials. [1][2][3] Thus, it is important to estimate the cost of the adjudication process, in order to establish whether the cost outweighs the perceived benefit of the process.
The cost of adjudication could be estimated as the direct cost paid to the adjudicator, which is often a menial amount (£10-15 per adjudication). However, this does not take into account the time it takes the adjudicator to undertake their assessment of the individual participant data. Furthermore, preparing data for the adjudicators, blinding and obtaining source information and general set-up costs are other expenses associated with the adjudication process that are borne by the co-ordinating centre (often a clinical trials unit). Thus, the cost of the adjudication process can be made up from (1) direct costs paid to the adjudicators, (2) time costs of the adjudicators, and (3) co-ordinating centre costs.
The aim of this study was to estimate the cost of central adjudication in randomised stroke trials and present this cost in terms of the number of events corrected by the central adjudication process.

Data collection
Authors of stroke trials included in a systematic review 1 were invited to provide data on the cost of outcome adjudication in an online questionnaire (see Supplemental Material, Cost of Adjudication Questionnaire). Data collected included time required to adjudicate events, adjudicators' approximate salary in today's costs and how disagreements were handled.

Benefit of adjudication
Our measure of benefit of the adjudication process was the number of events corrected after adjudication. This outcome provides an estimate of the agreement between the adjudicators and site investigators and quantifies the extent to which the adjudication process changes the trial outcome. However, this measure does not take into account whether the adjudication process affects the treatment effect estimate. We collected the number of events corrected after adjudication in our previous systematic review. 1

Cost of adjudication
All costs were taken from a societal perspective and were reported in various currencies (Australian dollars, Euros, Pounds Sterling and New Zealand Dollars). If costs were not entered as pounds sterling, then all other currencies were converted into pounds sterling on 16 May 2019.
We calculated adjudicator hourly salary, assuming that adjudicators worked 1680 h a year (40 h a week for 42 weeks of the year). Total time was determined using the amount of time it took for each adjudication, the number of events adjudicated and whether events were adjudicated multiple times. In addition to staff salary costs assumed to be borne by employers rather than study funders, we also assumed that adjudicators were directly paid £10 per event from study funds. This is not paid in all studies; however, it enables a comparison between studies and, where paid, is typical of the amount in UK studies. Costs of resolving disagreements were based on staff time only.
For each study, we assumed a fixed set-up cost to create a website for data sharing (brain scans, video footage, etc.) and for adjudicators to submit their final assessments. We estimated this to require 40 h of a database programmer (cost £650). We also assumed that each adjudicated event required 4 h of administrative time at the co-ordinating centre (cost £40 per adjudicated event).

Statistical analysis
Continuous variables were summarised with mean and standard deviation, or median and interquartile range. Categorical variables were described with frequency counts and percentages. The cost per event corrected was determined for each trial and for all trials.

Results
Authors from nine trials, comprising 25,436 participants, completed the questionnaire, and these nine trials are included in this study [4][5][6][7][8][9][10][11][12] (Table 1). All outcomes adjudicated were either stroke or a composite including stroke, with two trials including transient ischaemic attack as part of their outcome of stroke (see Supplementary Material, Supplementary Table I). In the majority of the trials (7/9, 77%), adjudicators reviewed only those events identified by the site investigators, but for two of the studies, adjudicators assessed a larger number of events by adjudicating either suspected events or all participants (see Supplementary Material, Supplementary Table I). The time taken per adjudication range from 5 min to 2 h. However, the trial with the lowest time had three adjudicators assess each event ( Table 2). All other trials assessed each event either once or twice. The trial with highest adjudicator salary cost was around twice that of the lowest.
Three of the trials (33%) had no disagreements (see Supplementary Material, Supplementary Table II). For trials that had disagreements between adjudicators, the cost and time associated with this was far less than that associated with the original adjudication (Table 3). Two studies had no events corrected after adjudication, although the total cost of the adjudication process for both of these was less than £4,000. Three trials had a total estimated cost of the adjudication process in excess of £100,000. On average, across the seven trials with corrected events, adjudication cost £2,295.10 per event corrected (SD: £1482.42).

Discussion
In this analysis of nine randomised trials, we found that, on average, adjudication costs approximately £2300 per corrected event, with the total cost of adjudication exceeding £100,000 for three trials. Our study assumed a fixed cost of £40 per adjudication for the co-ordinating centre, as it was not possible to collect accurate data retrospectively on a trial-by-trial basis. Another study estimated the co-ordinating centre cost for the adjudication process in a thromboprophylaxis trial, to be approximately £55 per adjudicated event 13 compared to our assumption of £40. Thus, we may have marginally underestimated the total cost of the adjudication process.
The nine trials included in this study all had binary primary outcomes, and when these were composites, they were constructed from binary components. There may be different types of outcomes that have greater or worse cost-benefit of adjudication than the outcomes described in this study. For example, outcomes with greater objectivity, such as disabling stroke, may have less need for adjudication when compared to more subjective outcomes, such as functional status measured on the modified Rankin Scale or causality of adverse events. 14 In addition, the method of adjudication could have large cost-benefit implications. For studies where adjudicators only assess site-reported events, the cost will be lower than studies in which all participants are adjudicated. However, when adjudicators only assess site-reported events, the adjudicators cannot identify additional events that could have been missed by the site investigators, potentially limiting the benefit of adjudication. Furthermore, the blinding status of the site investigators has the potential to influence the benefit that can be gained from adjudication, 15 and this information should be used when designing a clinical trial to understand the potential costs and benefits adjudication can bring.
This study has a number of limitations. First, the nine trials included span a 30-year time interval and cover a range of academic and industry funded trials, as well as including funding agencies that have varied sources of financial support (e.g. charities, government agencies). Therefore, the variability in these factors may have contributed to the disparity seen between the time taken for adjudication, which may have influenced the estimated cost of adjudication in this study. Second, this study relies on retrospectively collected cost data. Some of the trials included in this study performed their adjudications decades ago, so it is possible that these data are not as accurate as data collected prospectively. However, this is the first study that we are aware of that has attempted to estimate the entire cost of the adjudication process. As eluded to above, further research is warranted that prospectively collects the costs of central adjudication in an ongoing trial. A previous study showed that adjudications can take months to complete, 14 and a prospective study could accurately measure the time taken for the multitude of processes that are involved in implementing adjudication in a clinical trial. Furthermore, combining cost data with the approach of Kahan et al. 16 could give further understanding to the cost-benefit of increasing the number of adjudicators or varying the method of adjudication.
To conclude, central adjudication has been shown to control for differential misclassification and is important for trials where blinding is inadequate or impossible. 15 However, it can be costly, and these costs need to be estimated when designing a trial to evaluate whether the benefits of central adjudication outweigh the potential expense.