Be skeptical about unexpected large apparent treatment effects:

Abstract

The preliminary results of the twelfth Medical Research Council acute myeloid leukemia trial show no evidence of a survival advantage for five courses of therapy compared to four courses in a randomized comparison involving 1078 patients (hazard ratio 1.09, 95% confidence interval [CI] 0.87–1.37, p=0.4). However, the data presented to the independent data monitoring and ethics committee (DMEC) at both its reviews in 1998 suggested large benefits for the additional course with hazard ratios of 0.47 and 0.55 (95% CIs 0.29–0.77 and 0.38–0.80, p=0.003 and p=0.002, respectively). Despite these highly significant findings, the DMEC did not recommend closure of the randomization, a decision vindicated by the subsequent reversion to a null result. The main reason for not closing the randomization was that the treatment effects observed in 1998 (53% and 45% reductions in the odds of death) were considered too large to be clinically plausible, despite the p-values associated with them. Investigations have not identified any clinical explanations, such as different types of patients in the early and later parts of the trial, to explain the loss of benefit as the trial progressed. Thus, the most likely current explanation for the large benefit observed early on is the play of chance. Lessons to be learned from this example are that: fixed stopping rules based on some predetermined p-value should not be used and the decision to close a randomization or not should take account of other factors such as the medical plausibility of the magnitude of the treatment effect; chance effects do occur and happen more frequently than many clinicians realize; it is important that DMEC members are experienced in the interpretation of clinical trial evidence and aware of the dangers of early stopping without wholly convincing evidence.