Sub-Plenary 1-6

Research Evaluation at the Crossroads of Science:
The Effects and Implications of the Machinery
of Research Evaluation

 

Organizers:
Eliana Minelli, Università Carlo Cattaneo – LIUC, Italy
Maria Rucsandra Stan, University of Milan-Bicocca, Italy
 
Panelists:
Lars Engwall, Uppsala University, Sweden
Alis Oancea, Kellogg College, University of Oxford, United Kingdom
Emanuela Reale, Research Institute on Sustainable Economic Growth (IRCrES), Italy
Alesia Ann Zuccala, University of Copenhagen, Denmark
 

Research evaluation mechanisms have a significant influence on publishing patterns and behaviours (Korytkowski & Kulczycki, 2019; Moed, 2008; Cronin, 2013), resource allocation (Carlsson, 2009; Neave, 1991; Gläser, 2007), and academic careers (Marini, 2017; Marzolla, 2015). Furthermore, evaluation has potential governance effects, as governance modes “reflect the missions of higher education and research” (cf. EGOS Colloquium 2023, Sub-Plenary 1-1: Collegiality in Higher Education and Beyond: Rethinking the Organized World), influencing the hierarchical relationships between actors, but also at level of the scientific community, reconfiguring the academic work (Molas-Gallart, 2012).
 
The research evaluation traditions of European countries have often followed neo-liberal principles, aimed at steering at a distance while ensuring the autonomy of universities, as well as promoting accountability through strengthening the capacity to plan, control and measure achievements (Neave, 1998; Westerheijden, 2007; Rosa & Amaral, 2007). The literature has addressed the critical aspects of research evaluation arising from the transformation of the institutional and social contexts, from the perspective of the transnational authority of science (Whitley, 2003) and emerging interdisciplinary models of research (Altbach & Lewis, 1996; Moscati, 2001; Rostan, 2011). Nonetheless, the effects of research evaluation on careers, hierarchies, governance and management at institutional level and on research quality at the societal level are still underexplored.
 
Based on the evidence from an Italian research program of national interest, the sub-plenary will reflect on the principles underlying evaluation systems and focus on national assessment practices, exploring the implications for the specific university mission and career trajectories of researchers, as well as in terms of funding allocation. We encourage the audience to share their experiences from their respective systems, fostering a diverse and comprehensive discussion. We will conclude the sub-plenary by exploring the wider relevance of research evaluation beyond academia, particularly highlighting its impact on society and how the current evaluation system either facilitates or hinders the delivery of high-impact research.
 
This sub-plenary aims at delivering three ‘key takeaways’:

• Reflect on the ontological and epistemological foundations of the current evaluation mechanisms;

• The role and responsibilities of the researcher in research evaluation discourses;

• The need for a more nuanced approach to research evaluation, which considers both intrinsic and extrinsic characteristics of quality and recognized the complexity of defining and assessing research beyond academia.

 
The sub-plenary will complement the Colloquium’s theme of “Crossroads for Organizations: Time, Space, and People” by examining the impact of research evaluation on organizations and individuals. Furthermore, the research aligns with the Colloquium’s focus on adapting organizations to the interconnected “onlife” world, where individuals and technology shape and transform organizational dynamics. While the sub-plenary primarily focuses on higher education, it aims to foster discussions about the broader impact beyond academia and embrace diverse perspectives of the numerous disciplines that converge at these “crossroads”. In this regard, we will revisit previous organizational theories emphasizing the role of research organizations, including universities and research centres, in promoting innovation and collaboration to adapt to dynamic environments.
 

---

References

---

• Altbach, P.G., & Lewis, L. (1996): “The academic profession in international perspective.” In: P.G. Altbach (ed.): The International Academic Profession: Portraits of Fourteen Countries. Princeton, N.J.: Carnegie Foundation for the Advancement of Teaching.
• Carlsson, H. (2009): “Allocation of Research Funds Using Bibliometric Indicators – Asset and Challenge to Swedish Higher Education Sector.” INFOtrend, 64 (4), 84–90.
• Cronin, B. (2013): “Slow Food for Thought.” Journal of the American Society for Information Science and Technology, 64 (1), 1.
• Gläser, J. (2007): “The Social Orders of Research Evaluation Systems.” In: R. Whitley & J. Gläser (eds.): The Changing Governance of the Sciences. Sociology of the Sciences Yearbook, Vol. 26. Dordrecht: Springer, 245–266.
• Korytkowski, P., & Kulczycki, E. (2019): “Examining how country-level science policy shapes publication patterns: The case of Poland.” Scientometrics, 119 (3), 1519–1543.
• Marini, G. (2017): “New promotion patterns in Italian universities: Less seniority and more productivity? Data from ASN.” Higher Education, 73 (2), 189–205.
• Marzolla, M. (2015): “Quantitative Analysis of the Italian National Scientific Qualification.” Journal of Informetrics, 9 (2), 285–316.
• Moed, H.F. (2008): “UK Research Assessment Exercises. Informed judgments on research quality or quantity?” Scientometrics, 74 (1), 153–161.
• Molas-Gallart, J. (2015): “Research evaluation and the assessment of public value.” Arts and Humanities in Higher Education, 14 (1), 111–126.
• Moscati, R. (2001): “Italian university professors in transition.” Higher Education, 41 (1), 103–129.
• Neave, G. (1991): “A Changing Europe: Challenges for Higher Education Research.” Higher Education in Europe, 16 (3), 3–27.
• Neave, G. (1998): “The Evaluative State Reconsidered.” European Journal of Education, 33 (3), 265–284.
• Rosa, M.J., & Amaral, A. (2007): “A Self-Assessment of Higher Education Institutions from the Perspective of the EFQM Excellence Model.” In: D.F. Westerheijden, B. Stensaker & M.J. Rosa (eds.): Quality Assurance in Higher Education. Trends in Regulation, Translation and Transformation. Higher Education Dynamics, Vol. 20. Dordrecht: Springer, 181–207.
• Rostan, M., & Vaira, M. (eds.) (2011): Questioning Excellence in Higher Education: Policies, Experiences and Challenges in National and Comparative Perspective. Higher Education Research in the 21st Century Series, Vol. 3. Rotterdam: Sense Publishers.
• Westerheijden, D.F. (2007): “States and Europe and Quality of Higher Education.” In: D.F. Westerheijden, B. Stensaker & M.J. Rosa (eds.): Quality Assurance in Higher Education. Trends in Regulation, Translation and Transformation. Higher Education Dynamics, Vol. 20. Dordrecht: Springer, 73–95.
•