Research into the academic system
Research into research improves the academic system. CWTS studies and evaluates the academic system in various ways. One way is to provide information on the productivity and impact of academics by documenting how many publications they or their universities produce and how often these publications are cited.
Apples and oranges
Last year, the UK Research Excellence Framework reported that British biomedical researchers had written twice as many top publications in 2014 as in 2008. Doubling the number of top publications in the space of six years would be a fantastic achievement. CWTS researcher Thed van Leeuwen and a number of British colleagues delved into this remarkable study. When he examined the data on which the claim was based, Van Leeuwen saw that the 2014 study did not include publications in journals with lower impact factors, whereas the 2008 one did. This is like looking at the average of your whole school report one year and at the average of only your higher marks the next. It looks like you have massively improved, but you are essentially comparing apples and oranges. Van Leeuwen worked out that the British biomedical research had really improved ‘only’ to the order of 10 to 25%.
This incident highlights the importance of transparent research. The correct definition and calculation of indicators is important, but so too is high-quality data, the cornerstone of research. Van Leeuwen: ‘The quality of data collection is hugely important. This part of the research process is often underestimated and undervalued. Collecting research data is often seen as boring, routine and definitely less “exciting” than its analysis. But data and all that relates to it is actually of crucial importance in conducting academic research. ‘The old chestnut “Garbage in, garbage out” couldn’t be more true here.’
Measuring academic productivity
CWTS specialises in bibliometrics, one aspect of which involves producing statistics on the number of publications that universities and academics publish and the number of times that these publications are cited. Bibliometrics provide information on the academic output and impact of universities and academics. The figures are often used in university rankings that are published all over the world. As quantitative information is of increasing importance to policymakers and administrators in higher education, the figures must be reliable and it must be possible to compare them (over time too).
When conducting bibliometric research, CWTS researchers stumble upon errors and imprecise data. Van Leeuwen: ‘When we analyse universities, we often find that researchers have been careless with the university’s address. Then an institution appears in different formats in our data files.’ The CWTS ensures that the different formats used for an institution or author are brought under a single header and also draws up standards for whether university hospitals are counted as part of a university.
‘Another problem, if we look at individuals, is with homonyms and synonyms: there can be more than one John Brown, so different researchers can share the same name. But there are also different ways to spell John Brown. Researchers often publish under different variants of their full name.’
The findings of the bibliometric research conducted by the CWTS are used in the CWTS Leiden Ranking, which provides an overview of the performance of a large number of universities worldwide and is based on bibliometric statistics such as number of frequently cited publications. What makes the Leiden Ranking unique is that alongside a transparent methodology, CWTS uses advanced indicators to give an idea of the scientific impact of universities. CWTS thus tries to give the most precise possible picture of the scientific performance of one university compared with another.