Maybe hilarity is a bit strong.
I’ve just been looking at the recently published (in JAMA) Guidelines for the Content of Statistical Analysis Plans in Clinical Trials. Most of the action is in the eAppendix2 “Explanation and Elaboration of Essential Items,” where they go through the essential items in exhaustive and sometimes mind-numbing detail.
It’s all based very much on traditional frequentist designs so there is much that I don’t agree with, but Item 8: “Description of specific objectives or hypotheses” particularly caught my attention. They give an example:
The null hypothesis is that there is no difference in time to first blood stream infection between the standard and impregnated (antibiotic and heparin combined) groups. The alternative hypothesis is that there is a difference between the two groups.
The primary objective of this trial is to determine the effectiveness of heparin bonded or antibiotic impregnated CVCs (combined) compared with standard CVCs for preventing hospital acquired blood stream infection
Secondary objectives are:
a. To determine the cost effectiveness of heparin bonded or antibiotic impregnated CVCs compared with standard CVCs, based on the primary outcome and costs of acute care from the perspective of the NHS.
b. To determine the effectiveness of type of CVC in 3-way comparisons of heparin bonded versus antibiotic impregnated versus standard CVCs for preventing hospital acquired blood stream infection, based on culture, quantitative bacterial DNA, and clinical measures of infection.”
What’s wrong with that? Mainly that the Research Hypothesis doesn’t address any of the study objectives. That seems a bit of a problem! Everyone is so used to equating statistical significance with effectiveness, and treating effectiveness as a binary outcome (it works or it doesn’t), that this just passed without comment.
Guidelines for the Content of Statistical Analysis Plans in Clinical Trials. Gamble, C., Krishan, A., Stocken, D., Lewis, S., Juszczak, E., Doré, C., … Loder, E. (2017). Jama, 318(23), 2337. link to publication