Sub-Plenary 1-6

Research Evaluation at the Crossroads of Science:
The Effects and Implications of the Machinery
of Research Evaluation

 

Thursday, July 4, 2024, 16:00–17:30 CEST

U6 Building – “AGORÀ” | Room: U6-P0-09 | Piazza dell’Ateneo Nuovo, 1 | 20126 Milano


Organizers:
Eliana Minelli, Università Carlo Cattaneo – LIUC, Italy
Maria Rucsandra Stan, University of Milan-Bicocca, Italy
 
Panelists:
Lars Engwall, Uppsala University, Sweden
Alis Oancea, Kellogg College, University of Oxford, United Kingdom
Emanuela Reale, Research Institute on Sustainable Economic Growth (IRCrES), Italy
Alesia Ann Zuccala, University of Copenhagen, Denmark
 

This sub-plenary session will engage scholars in a deep reflection on research evaluation systems and practices and their implications for the development and wellbeing of our societies. Studying the effects and impact of research evaluation is crucial as it enhances research quality, ensures research integrity, guides resource allocation, accurately assesses research impact, and promotes inclusivity and innovation within the research environment.

Research evaluation mechanisms have a significant influence on publishing patterns and behaviours (Korytkowski & Kulczycki, 2019; Moed, 2008; Cronin, 2013), resource allocation (Carlsson, 2009; Neave, 1991; Gläser, 2007), and academic careers (Marini, 2017; Marzolla, 2015). Furthermore, evaluation has potential governance effects, as governance modes “reflect the missions of higher education and research” (cf. EGOS Colloquium 2023, Sub-Plenary 1-1: Collegiality in Higher Education and Beyond: Rethinking the Organized World), influencing the hierarchical relationships between actors, but also at level of the scientific community, reconfiguring the academic work (Molas-Gallart, 2012).
 
The research evaluation traditions of European countries have often followed neo-liberal principles, aimed at steering at a distance while ensuring the autonomy of universities, as well as promoting accountability through strengthening the capacity to plan, control and measure achievements (Neave, 1998; Westerheijden, 2007; Rosa & Amaral, 2007). The literature has addressed the critical aspects of research evaluation arising from the transformation of the institutional and social contexts, from the perspective of the transnational authority of science (Whitley, 2003) and emerging interdisciplinary models of research (Altbach & Lewis, 1996; Moscati, 2001; Rostan, 2011). Nonetheless, the effects of research evaluation on careers, hierarchies, governance and management at institutional level and on research quality at the societal level are still underexplored.
 
Based on the evidence from an Italian research program of national interest, the sub-plenary will reflect on the principles underlying evaluation systems and focus on national assessment practices, exploring the implications for the specific university mission and career trajectories of researchers, as well as in terms of funding allocation. We encourage the audience to share their experiences from their respective systems, fostering a diverse and comprehensive discussion. We will conclude the sub-plenary by exploring the wider relevance of research evaluation beyond academia, particularly highlighting its impact on society and how the current evaluation system either facilitates or hinders the delivery of high-impact research.
 
This sub-plenary aims at delivering three ‘key takeaways’:

  • Reflect on the ontological and epistemological foundations of the current evaluation mechanisms;

  • The role and responsibilities of the researcher in research evaluation discourses;

  • The need for a more nuanced approach to research evaluation, which considers both intrinsic and extrinsic characteristics of quality and recognized the complexity of defining and assessing research beyond academia.

 
The sub-plenary will complement the Colloquium’s theme of “Crossroads for Organizations: Time, Space, and People” by examining the impact of research evaluation on organizations and individuals. Furthermore, the research aligns with the Colloquium’s focus on adapting organizations to the interconnected “onlife” world, where individuals and technology shape and transform organizational dynamics. While the sub-plenary primarily focuses on higher education, it aims to foster discussions about the broader impact beyond academia and embrace diverse perspectives of the numerous disciplines that converge at these “crossroads”. In this regard, we will revisit previous organizational theories emphasizing the role of research organizations, including universities and research centres, in promoting innovation and collaboration to adapt to dynamic environments.
 


References


  • Altbach, P.G., & Lewis, L. (1996): “The academic profession in international perspective.” In: P.G. Altbach (ed.): The International Academic Profession: Portraits of Fourteen Countries. Princeton, N.J.: Carnegie Foundation for the Advancement of Teaching.
  • Carlsson, H. (2009): “Allocation of Research Funds Using Bibliometric Indicators – Asset and Challenge to Swedish Higher Education Sector.” INFOtrend, 64 (4), 84–90.
  • Cronin, B. (2013): “Slow Food for Thought.” Journal of the American Society for Information Science and Technology, 64 (1), 1.
  • Gläser, J. (2007): “The Social Orders of Research Evaluation Systems.” In: R. Whitley & J. Gläser (eds.): The Changing Governance of the Sciences. Sociology of the Sciences Yearbook, Vol. 26. Dordrecht: Springer, 245–266.
  • Korytkowski, P., & Kulczycki, E. (2019): “Examining how country-level science policy shapes publication patterns: The case of Poland.” Scientometrics, 119 (3), 1519–1543.
  • Marini, G. (2017): “New promotion patterns in Italian universities: Less seniority and more productivity? Data from ASN.” Higher Education, 73 (2), 189–205.
  • Marzolla, M. (2015): “Quantitative Analysis of the Italian National Scientific Qualification.” Journal of Informetrics, 9 (2), 285–316.
  • Moed, H.F. (2008): “UK Research Assessment Exercises. Informed judgments on research quality or quantity?” Scientometrics, 74 (1), 153–161.
  • Molas-Gallart, J. (2015): “Research evaluation and the assessment of public value.” Arts and Humanities in Higher Education, 14 (1), 111–126.
  • Moscati, R. (2001): “Italian university professors in transition.” Higher Education, 41 (1), 103–129.
  • Neave, G. (1991): “A Changing Europe: Challenges for Higher Education Research.” Higher Education in Europe, 16 (3), 3–27.
  • Neave, G. (1998): “The Evaluative State Reconsidered.” European Journal of Education, 33 (3), 265–284.
  • Rosa, M.J., & Amaral, A. (2007): “A Self-Assessment of Higher Education Institutions from the Perspective of the EFQM Excellence Model.” In: D.F. Westerheijden, B. Stensaker & M.J. Rosa (eds.): Quality Assurance in Higher Education. Trends in Regulation, Translation and Transformation. Higher Education Dynamics, Vol. 20. Dordrecht: Springer, 181–207.
  • Rostan, M., & Vaira, M. (eds.) (2011): Questioning Excellence in Higher Education: Policies, Experiences and Challenges in National and Comparative Perspective. Higher Education Research in the 21st Century Series, Vol. 3. Rotterdam: Sense Publishers.
  • Westerheijden, D.F. (2007): “States and Europe and Quality of Higher Education.” In: D.F. Westerheijden, B. Stensaker & M.J. Rosa (eds.): Quality Assurance in Higher Education. Trends in Regulation, Translation and Transformation. Higher Education Dynamics, Vol. 20. Dordrecht: Springer, 73–95.
  •  

Lars Engwall is Professor Emeritus of Business Studies at Uppsala University, Sweden. He has published widely on institutional change and the diffusion of management ideas, in particular the role of management education and of the media.

Eliana Minelli is an Associate Professor at Carlo Cattaneo University LIUC (Castellanza, Italy) in Organization and Management. She has published on research assessment and evaluation in peer-reviewed journals, including Quality in Higher Education and Higher Education Policy, and has also published books on the same topic. Since 2019, Eliana has coordinated the LIUC unit of the PRIN Project “The effects of evaluation on academic research: knowledge production and methodological issues”.
 
Alis Oancea is a Professor of Philosophy of Education and Research Policy at the University of Oxford, United Kingdom, and Project Lead (Research on Research) at the Economic and Social Research Council (ESRC) Centre for Global Higher Education (CGHE). Alis is the Joint Editor of the Oxford Review of Education and Chief Editorial Advisor on Routledge Open Research (Education). Alis has published in numerous peer-review journals, including Oxford Review of Education, European Educational Research Journal, Studies in Higher Education, and Research Evaluation.
 
Emanuela Reale has been Director of the Research Institute on Sustainable Economic Growth (IRCrES CNR) until 2023 and is currently Associate Senior at IRCrES in Turin, Italy. Her main areas of interest are higher education policy, governance, R&D funding, STI indicators, research evaluation, and impact assessment. Emanuela is member of the Board of the Consortium of Higher Education Researchers (CHER), of the European Network of Indicator Designers (ENID), and President of the European Forum for Research and Innovation (EU-SPRI).
 
Maria Rucsandra Stan is a currently a post-doctoral researcher in Strategy and Entrepreneurship at University Milan-Bicocca, Italy. During her PhD she has participated in the PRIN Project ‘The effects of evaluation on academic research: knowledge production and methodological issues’.
 
Alisa Ann Zuccala is an Associate Professor at the Royal School of Library and Information Science, University of Copenhagen, Denmark. Her research specialization is in the field of bibliometrics, scholarly communication, research evaluation, and in recent years she has focused primarily on the evaluation of research outputs across the humanities. Alisa Ann has published in numerous peer-reviewed journals, including Journal of Informetrics, Scientometrics, and Research Evaluation.