Do “researcher degrees of freedom” invalidate my analysis?

Clearly report what you did and both results. Readers will make up their own minds about the strength of the findings.

The situation that you have encountered is very common. While testing a prior hypothesis with predefined methods is the right thing to do, in practice it is unavoidable that you will sometimes find problems that you need to correct after running an initial analysis.

Also, if your analysis produces an unexpected result, it is quite logical that this will lead to extra checking of what you have done, and further analyses to confirm it.

There's nothing wrong with doing extra things to refine the analysis. The problem would be refining the analysis and then failing to report on earlier versions.

A multiple testing correction would not typically be used for only two analyses. Furthermore, it is a judgement call whether this counts as two tests or one. If the second analysis was really the "correction" of something flawed, that is not quite the same as repeating two different but valid approaches. And there is not a clear line as to when that becomes true.

In summary, there is no problem with what you have done, as long as you report it, but your concern with doing this the right way is a very good thing--please keep it up! Inadequate reporting of this sort of thing is quite common, and it undermines the quality of scientific evidence.


@dan1111's answer is excellent if you do not have the resources to repeat your experiment, but under ideal circumstances, no matter how your model was created, an independent replication of the experiment would be the best possible evidence of your model's strength.

Much like in experiments involving fitting models to data (such as protein crystallography or data mining), it is perfectly acceptable to refine/train a model after obtaining your initial data, if the model is then fitted with independent results from a different dataset.

For example, you could repeat the data collection (and apply your new filter), and then run your model on the new dataset. If the new dataset confirms the previous one, your model would then be pretty solid.

In any case, no matter which path you take, you should present all of your results in your final publication. If you do in fact have the resources to repeat the experiment, the writeup could go as follows:

I collected dataset A, and then performed analysis using filter 1. However, the results were not statistically significant.

Upon realising that there were a number of flaws in filter 1, I then modified the filter, removing (anomalous values) and changing the filter to filter 2. Upon repeating the analysis, the results were statistically significant.

To reduce the likelihood of "researcher degrees of freedom" affecting the model, I then collected an independent dataset B. Upon repeating the analysis using filter 2, I found that the results replicated the findings of Filter 2 on Dataset A.

Tags:

Statistics