Research assessment

Research assessment is a broad concept. It can be said that it concerns the assessment of the research as well as the metrics of research contributions and results and research impact. It embraces both qualitative and quantitative methodologies including the use of bibliometric indicators, mapping and peer review.

It is important for researchers to be aware of the research impact since it helps them to:

• Strengthen their position when they claim tenure or promotion.
• Be more able to make future funding requests by showing the importance of their research.
• Understand the public and learn how to address it.
• Recognize who uses their work and ensure that it is properly acknowledged.
• Recognize partners relevant or not to their subject area.
• Manage their academic reputation.

Indicators and metrics

Metrics and indicators, most of which come from commercial providers/creators, can often be misunderstood or misused. Despite the fact that the indicators may be correct and accurate, the science and research environment and all the involved stakeholders make it difficult and, sometimes, unreliable to use indicators exclusively. In any case, they should be examined and used in a careful and proper way.

Below there is a review and a description of some of the indicators. The metrics indicators can provide information at either journal or article level and most of them are providers’ products, such as Thomson Reuters, Elsevier, and Google. While Web of Science (Thomson Rheuters) and Scopus (Elsevier) are commercial tools, accessible through subscription, Google Scholar is freely available but it lacks quality control. It is advisable that data from Google Scholar be used following a meticulous check and data cleaning.

Principles of Research Assessment

The fact that the assessment of research and researchers has been associated with metrics indicators for years has created a lot of difficulties and disadvantages. Therefore, it became obvious that the research assessment should focus on the actual content of the research and not, for example, on the journal titles. All the parties involved, such as researchers, funders, publishers and institutions, realized the need to improve the means and methods to assess research. This resulted in the development of good practices in research assessment, such as the ones described in the San Francisco Declaration on Research Assessment and the Leiden Manifesto. What is more, the European Commission has made extensive reference to research assessment and the changes that should be made to the system in the report it published concerning the future of Scholarly Publication and Communication in 2019. It recommends an assessment system which would be objective, neutral and austere and calls universities and research centres to develop a transparent and fair assessment system by adopting the principles outlined in the San Francisco Declaration and the Leiden Manifesto.

In 2012, the Declaration on Research Assessment (DORA) was drawn up during the Annual Meeting of the American Society for Cell Biology in San Francisco, USA, which covers every scientific field and is available in Greek, as well. The declaration focuses on the following subjects:

• the need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations
• the need to assess research on its own merits rather than on the basis of the journal in which the research is published; and
• the need to capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact),

while the general recommendation is “Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”.

The Leiden Manifesto for Research Metrics was drawn up by five experts who suggested 10 principles for research metrics and published in Nature in 2015. The principles are roughly outlined below:

1. Quantitative evaluation should support qualitative, expert assessment.
2. Measure performance against the research missions of the institution, group or researcher.
3. Protect excellence in locally relevant research.
4. Keep data collection and analytical processes open, transparent and simple
5. Allow those evaluated to verify data and analysis.
6. Account for variation by field in publication and citation practices.
7. Base assessment of individual researchers on a qualitative judgement of their portfolio.
8. Avoid misplaced concreteness and false precision.
9. Recognize the systemic effects of assessment and indicators.
10. Scrutinize indicators regularly and update them