Overview
This paper analyses traditional evaluative practices on the example of European Capitals of Culture (ECoC), namely Matera-Basilicata 2019. In an in-depth case study, we focus on diverse phases of its becoming as an ECoC, the title year and long term-impacts. We contrast the evaluative practices with empirical data from the field, using differences in perception to delineate a possible development of evaluation. Supported by the discourse on a skeptical turn in evaluation, we refer to aesthetic perspectives in and on evaluating ECoC. Artists in residence and city writers are taken as genuine role models and learning fields.

Issue and Argument
For nearly 40 years by now more than 60 cities have been designated as ECoC. Since 2009, the notion of ‘impact’ raises issues from urban regeneration to new economic opportunities, from social inclusion to self-identity, and from quality of life to dimensions of citizenship, including participation, subjective experience and learning among visitors (Piber et al., 2018).

First legal frameworks, like Decision 1499/1999/EC and Decision No. 1622/2006/EC of the European Parliament and the Council were not very elaborated on evaluation. Unsurprisingly, the study by Beatriz Garcia and Tamsin Cox for the European Parliament (European Parliament, 2013) asks “(…) to develop a far more robust and objective data collection mechanism, with clearer specifications as to what should be collated by each city, and, importantly, how this should be collated and subsequently presented.” (p. 196)

In 2021, the Commission published a compilation of recommendations made by the ex post evaluations for ECoC 2007-2019 (European Commission, 2021). It presents how little has changed in status quo and future ambitions: “The amount of evidence on the impact of ECoC is still relatively limited.” (p. 10) Consequently, the Commission recommends to invest early into gathering baseline data to support a strong narrative of the ECoC’s development and to strengthen monitoring and reporting by annual implementation plans and milestones which is partly reflected in Decision No. 445/2014. It is relevant for all ECoC taking place from 2020-2033. It installs a hierarchy of goals, clarifies the expert Panel’s monitoring-role, and is more elaborate on evaluation issues.

References
Evaluation research is at the crossroads. Since decades, diverse metaphors are offered to demonstrate a dynamic development provoked by critique (Lincoln, Lynham and Guba, 2018, for an overview on paradigmatic controversies). ‘Stages’ are proposed, for instance, by Shadish, Cook and Leviton (1991). Others propose the term ‘generations’, like Guba and Lincoln (1989) with their 4th generation evaluation from a constructivist perspective, or Brousselle and Buregeya (2018) propagate a 5th generation. ‘Waves’ (Vedung 2010; Picciotto 2020) are as well a metaphorical description of alternative perspectives on evaluation which increase in importance. However, some problems remain unresolved and seem to be remarkably resilien. Peter Dahler-Larsen (2019) identifies six on a theoretical and practical level, asking for a ‘skeptical turn’. Out of a skeptical turn-perspective, inductively generated insights from our empirical case study support a more context-sensitive methodology and method of evaluating ECoC. Voices from the field substantially change reflection upon fundamental aspects of evaluation, like unfulfilled promises regarding utility, cost- effectiveness, accountability, improvement of governance, or reliability of forecasts. It reminds us to find different responses from before. ECoC are multi-stakeholder-oriented and complex settings, which need a perspective beyond the conventional repertoire of evaluation. With an aesthetic approach we try to overcome the difficulties of conventional evaluation frameworks, approaching to a better understanding of dynamics of change along the timeline an ECoC as temporary organization is subjected to.

Methodology
Methodically grounded in qualitative social research, we combined participatory observations, document analysis and (semi-structured) interviews. Ten interviews with decision makers, project managers, marketing and communication managers as well as other stakeholders like entrepreneurs, artists and journalists were backed by interviews with EU-evaluation experts, researchers and cultural sector organizers. Key questions addressed impact dimensions, evaluation strategy, framework and implementation, cooperation within the ECoC-network. Seminal quotes were summarized, analyzed, and combined with document analysis (Bid Books, program folders and accompanying material, monitoring- and evaluation-reports and a broader look at the scientific literature).

Takeaway and results
Some preliminary conclusions are allowed for at that stage: The empirical data demonstrate more layers of perception than traditional evaluations focus on. They allow for putting metrics into context. The quotations reveal many nuances, controversial judgements and a spectrum of impression, critique, expression, positioning and emotions traditional evaluations are not capable to represent. A different sensorium of perception is needed in order to deal with ambiguity, ambivalence, and to find a contextualized balance within this setting. This makes us skeptical regarding the strategy to invest in more of data and to tighten evaluation and monitoring following the logic we already know.

Not much literature on evaluating ECoC reflects on limitations, or presents a comprehensive critique of the evaluative approaches usually in use. Neither do we find much on aesthetics as perspective, impetus or approach to construct an adequate ECoC-evaluation. We rely on the perceptual and epistemological perspective of aesthetic theory, the aesthetics of the beautiful and sublime as well the aesthetics of art (Habermas, 1981/1987; Lyotard, 1979; Kant, 1878/1963; Gadamer, 1960/1990; Feyerabend, 1979). The term of an ‘aesthetic lens’ is chosen by us to develop a better understanding of complex circumstances as well as complex impacts of an evaluand. Finally, we refer to two interesting examples, which can be used as role models for conceptualizing an aesthetic lens for evaluative purposes: Interventions from artists (art residencies) and the function of city/town-writers and poets (Barry and Meisiek, 2010 Schein, 2001; Lee et al., 2018; Moore, 2024). With both examples we tend to broaden the conceptualization of an aesthetic evaluation, asking, for instance, how much evaluation is enough (Raimondo, 2021), if ‘holistic’ models (Marra, 2019) are misleading, how much evaluation is needed to learn (Furubo, 2019; Sandahl, 2019 on ‘evaluation underload’), or what kind of ‘use’ to expect (Ivaldi et al., 2015). In short: Less evaluation for art and culture, and more evaluation by art and culture and its protagonists may be a promising path to follow.

Literature
Barry D and Meisiek S (2010) Seeing More and Seeing Differently: Sensemaking, Mindfulness, and the Workarts. Organization Studies 31(11): 1505-1530.
Benjamin LM (2021) Beyond programs: toward a fuller picture of beneficiaries in nonprofit evaluation. In: Dahler-Larsen P (ed) A research Agenda for Evaluation. Cheltenham: Edward Elgar, pp. 81-103.
Brouselle A and Buregeya J (2018) Theory-based evaluations: Framing the existence of a new theory in evaluation and the rise of the 5th generation. Evaluation 24(2): 153–168.
Dahler-Larsen P (2019) The skeptical turn in evaluation. In: Furubo JE and Stame N (eds) The Evaluation Enterprise. A Critical View. New York: Routledge, pp. 58-80.
Feyerabend P (1979) Erkenntnis für freie Menschen. Frankfurt am Main: Suhrkamp.
Furubo, JE (2019) Understanding the Evaluation Enterprise. In: Furubo JE and Stame N (eds) The Evaluation Enterprise. A Critical View. New York: Routledge, pp. 3-31.
Gadamer HG (1960/1990) Wahrheit und Methode: Grundzüge einer philosophischen Hermeneutik. Tübingen: Mohr.
Guba EG and Lincoln YS (1989) Fourth Generation Evaluation. Newbury Park, CA: Sage.
Habermas J (1981/1987) Theorie des kommunikativen Handelns. Part 1: Handlungsrationalität und gesellschaftliche Rationalisierung. Frankfurt am Main: Suhrkamp.
Ivaldi S, Scaratti G and Nuti G (2015) The practice of evaluation as an evaluation of practices. Evaluation
21(4): 497–512.
Kant I (1878/1963) Kritik der Urteilskraft. Stuttgart: Reclam. Kushner S (2000). Personalizing Evaluation. London: Sage.
Lee B, Fillis I and Lehman K (2018) Art, science and organisational interactions: Exploring the value of artist residencies on campus. Journal of Business Research 85: 444-451.
Lincoln YS, Lynham SA and Guba EG (2018) Paradigmatic Controversies, Contradictions, and Emerging Confluences, Revisited. In: Denzin NK and Lincoln YS (eds) The Sage Handbook of Qualitative Research. 5th ed. Los Angeles: Sage, pp. 108-150.
Lyotard JF (1989) Der Widerstreit. München: Wilhelm Fink. (French original: Le Différend. Paris: Les editions de Minuit 1983).
Matera-Basilicata 2019 Foundation (2018) Progress Report Matera European Capital of Culture 2019. Second Formal Monitoring Meeting. Bruxelles.
Matera-Basilicata 2019 Foundation (2021) Matera is a Producer of Culture. Matera European Capital of Culture Monitoring Report.
Matera 2019 Committee (2014) Matera candidate city european capital of culture 2019.
Palmer-Rae Associates (2004) European Cities and Capitals of Culture. Study prepared for the European Commission. Part I & II. Brussels: Palmer-Rae Associates.
Piber M, Demartini P and Biondi L (2018) The management of participatory cultural initiatives: learning from the discourse on intellectual capital. Journal of Management and Governance 23(1): 1-24.
Picciotto R (2020) From disenchantment to renewal. Evaluation 26(1): 49-60.
Raimondo E (2021) What if less were more? Exploring new pathways for institutionalization of evaluation in international organizations. In: Dahler-Larsen P (ed) A research Agenda for Evaluation. Cheltenham: Edward Elgar, pp. 43-62.
Sandahl R (2019) Evaluation Overload and Evaluation Underload. In: Furubo JE and Stame N (eds) The Evaluation Enterprise. A Critical View. New York: Routledge, pp. 111-125.
Schein EH (2001) The Role of Art and the Artist. Reflections: The SoL Journal 4(2): 81-83.
Shadish WR, Cook, TD and Leviton L (1991) Foundations of Program Evaluation. Thousand Oaks, CA: Sage.
Vedung E (2010) Four waves of evaluation diffusion. Evaluation 16(3): 263–77.