Unintentional hilarity in statistical analysis plan guidelines

Maybe hilarity is a bit strong.

I’ve just been looking at the recently published (in JAMA) Guidelines for the Content of Statistical Analysis Plans in Clinical Trials.  Most of the action is in the eAppendix2 “Explanation and Elaboration of Essential Items,” where they go through the essential items in exhaustive and sometimes mind-numbing detail.

It’s all based very much on traditional frequentist designs so there is much that I don’t agree with, but Item 8: “Description of specific objectives or hypotheses” particularly caught my attention.  They give an example:

Research hypothesis
The null hypothesis is that there is no difference in time to first blood stream infection between the standard and impregnated (antibiotic and heparin combined) groups. The alternative hypothesis is that there is a difference between the two groups.

Study objectives
The primary objective of this trial is to determine the effectiveness of heparin bonded or antibiotic impregnated CVCs (combined) compared with standard CVCs for preventing hospital acquired blood stream infection

Secondary objectives are:
a. To determine the cost effectiveness of heparin bonded or antibiotic impregnated CVCs compared with standard CVCs, based on the primary outcome and costs of acute care from the perspective of the NHS.
b. To determine the effectiveness of type of CVC in 3-way comparisons of heparin bonded versus antibiotic impregnated versus standard CVCs for preventing hospital acquired blood stream infection, based on culture, quantitative bacterial DNA, and clinical measures of infection.”

What’s wrong with that? Mainly that the Research Hypothesis doesn’t address any of the study objectives.  That seems a bit of a problem!  Everyone is so used to equating statistical significance with effectiveness, and treating effectiveness as a binary outcome (it works or it doesn’t), that this just passed without comment.

 

Guidelines for the Content of Statistical Analysis Plans in Clinical Trials. Gamble, C., Krishan, A., Stocken, D., Lewis, S., Juszczak, E., Doré, C., … Loder, E. (2017). Jama, 318(23), 2337. link to publication

2 thoughts on “Unintentional hilarity in statistical analysis plan guidelines

  1. Under the accept/reject paradigm one would still be informed in the sense of being able to state with given confidence that the effectiveness is non-zero.

    I was trained to formulate such hypotheses as “statistical” ones, not research ones though. I suppose the secondary, implicit goal is to provide an estimate of the effectiveness via point+CI which probably would be done anyway if the researchers are following the typical analysis cookbook.

    Like

  2. Yes, my point was really that the objective is about HOW effective something is, which implies estimating some quantity, but the “research hypothesis” is “is the difference zero?”

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s