Pre-registering your research: Extra effort, but what's the pay-off?
Open Science
Registering your hypothesis and analysis plan online before starting your study – why should researchers bother? Henk van Steenbergen, a researcher in cognitive psychology, decided to give it a try. 'I used it as an exercise in open science.'
'I hate doing my tax returns and outsource them whenever I can. Unfortunately, researchers increasingly have to jump through all sorts of bureaucratic hoops before they can even begin a new project,' says Henk van Steenbergen, Associate Professor in the Cognitive Psychology unit. 'Preregistration felt like yet another hoop to jump through, so I was a bit apprehensive.' Nonetheless, he decided to preregister a study – the results of which will soon be published in Psychological Medicine – several years ago. 'Because of Covid, the start of the project was massively delayed. Most labs were closed and many research projects were on hold anyway, so we thought: we might as well preregister the study. I decided to treat it as a practice run for open science.' What did Van Steenbergen learn from this experience?
More reliable research
'Preregistration means you state: this is my hypothesis and this is the analysis plan I intend to use. You record this on a website with a timestamp, after which it cannot be changed. You do this before you begin data collection – or at least before you have seen the full dataset. In the publication, you stick to this plan unless there is a very good reason to deviate from it. That restriction often makes research more reliable. Another advantage of preregistration is that it allows you to use one-tailed rather than two-tailed tests. If hypotheses are preregistered, a directional (one-tailed) test controls the risk of a false positive result just as well as a two-tailed test, while increasing the power to detect an effect.'
‘You essentially commit to the analysis you specified in advance, which makes your research more reliable’
One-tailed tests still not widely accepted
'In this study, we looked at the neurochemical basis of the broadening of attention that occurs when you receive a reward. Previous studies have shown that this broadening of attention is linked to mental flexibility and resilience. By using a pharmacological manipulation, we demonstrated that the chemicals that normally help reduce pain – the well-known endorphins – also make your mind more open and flexible. We first submitted our paper to a journal that has positioned itself strongly as an advocate of open science in recent years. Strangely enough, the article was rejected straight away, largely because we had reported one-tailed tests. This shows that even with preregistration, one-tailed tests are still not fully accepted, not even by the editors of a journal where open science is the norm. At Psychological Medicine, by contrast, the open science element of our study was greatly appreciated during the review process. So, although open science is now standard practice in psychology, there are still differences of opinion about certain details.'
A stronger start of your study
'One major benefit of preregistration was that I discussed the analysis plan with my co-authors in advance. For this study, I collaborated with a colleague who had developed the computer task and used it in a previous study. Because of preregistration, we had to agree on how to analyse the data. This led to an interesting email exchange before we could finalise the plan. I also showed the preregistration to another colleague who suggested adding a secondary hypothesis. I think this careful process upfront ultimately made the study stronger.'
'Preregistration is not suitable for every type of data'
Don't make pre-registration mandatory
'I don’t think preregistration should be compulsory. It simply doesn’t suit all types of data. I sometimes hear colleagues say: "My data are so complex, I cannot possibly predict what I will encounter." And they are right. For very complex datasets, preregistering analyses is often still too big a leap. But in those cases, you could at least preregister your hypotheses. Some colleagues also worry that preregistration will lock them into a single analysis approach. But deviations are perfectly fine – as long as you justify them transparently. We also encountered unexpected issues in our study that required a slightly different approach. And you can always run additional analyses – as long as you label them as exploratory, there is no problem.'
Use a template
'If it doesn’t take too much effort, I would say: always consider preregistering. And about my aversion to forms? For our preregistration we used a Word template created by Anna van ’t Veer and colleagues. You can adapt the document to your preferences, and it is easy to share and revise with co-authors. So preregistration can actually be quite straightforward – and if your study is a good candidate for it, why not?'
Open Science Week
During the National Open Science Week, our own OSCL and the faculty Academia in Motion teams will host several events, and highlight how open, transparent, equitable, and inclusive practices are shaping the future of academia. Check out the programme to find the next step that fits your open science journey.