with Friedrich Heinemann & Zareh Asatryan
German Federal Ministry of Finance
Project description
In the Multiannual Financial Framework (MFF) 2021-2027, about 30 percent of all EU budgetary resources are allocated to cohesion. Additional amounts are made available to the European budget by Next Generation EU (NGEU) through REACT EU. The central goal of EU cohesion policy remains regional convergence. However, despite the substantial use of funds, convergence success is not assured. While there have been successes in East-West convergence, North-South divergence persists stubbornly. Against the background of these developments and methodological advances in the evaluation of EU regional policy, this project addresses, among others, the following questions: How is the "European added value" of an EU regional policy programme to be conceptually defined? How should the design and effectiveness of central rules of EU cohesion policy be assessed? What adjustments to the regional policy toolbox are necessary in terms of scope, instruments and rules?
Project output:
with Friedrich Heinemann, Zareh Asatryan, Julia Bachtrögler-Unger, Franceso Corti, Maximilian von Ehrlich, Ugo Fratesi, Clemens Fuest, Valentin Lang & Martin Weber
ZEW Discussion Paper No. 24-034 // 2024
Abstract
By international comparison as well as compared to other EU policies, the EU‘s Cohesion Policy (CP) evaluation system is far developed and institutionalized. This paper analyses the remaining gaps and shortcomings in the CP evaluation system against principles established by the OECD and others and provides recommendations on how to further improve it. The presence of a broad and imprecise CP objective function emerges as a key challenge for evaluations. The evaluation culture is not equally developed among all Member States and regions. In quite some cases, an unfavorable equilibrium is found which is characterized by limited evaluation capacities, poor methods, and a formalistic approach to evaluations. Programme evaluations in the Member States are usually commissioned by national or regional managing authorities who have a vested interest in promoting the success of their programmes. Evaluations are carried out by evaluators who are functionally independent, but often lack factual independence. There is also limited international competition in the market for evaluations commissioned by national or regional authorities. Evaluation methods applied in CP programme evaluations mostly lag behind academic advancements and evaluation reports often do not transparently describe their methodological limitations. As the EU body responsible for implementing CP across all 27 Member States, the Commission may also have an overly optimistic perspective on CP. Finally, there is little evidence that evaluation findings are used for decision-making processes, funding allocation and the design of programmes. The paper offers a number of recommendations how to advance the evaluation system: (1) Reorient CP reforms towards a more focused set of objectives; (2) Specify evaluation obligations more precisely in the Common Provision Regulation and set out a ‘charter for evaluators’; (3) Introduce an ‘evaluate first’ requirement when preparing or updating programmes; (4) Promote the use of counterfactual methods; (5) Explicitly link funding decisions at programme and policy level to evaluation results; (6) Implement measures to stimulate a European market for CP evaluations; and (7) establish a standing European Advisory Panel on CP evaluation to foster independent third-party reviews.
with Zareh Asatryan and Friedrich Heinemann
ZEW Discussion Paper No. 24-037 // 2024
R&R @ ITAX
Abstract
Independent and high-quality evaluations of government policies are an important input for designing evidence-based policy. Lack of incentives and institutions to write such evaluations, on the other hand, carry the risk of turning the system into a costly beauty contest. We study one of the most advanced markets of policy evaluations in the world, the evaluations of EU Cohesion Policies by its Member States (MS). We use large language models quantify the findings of about 2,300 evaluations, and complement this data with our own survey of the authors. We show that the findings of evaluations are inconsistent with those of the academic literature on the output impacts of Cohesion Policy. Using further variation across MS, our analysis suggests that the market of evaluations is rather oligopolistic within MS, that it is very fragmented across the EU, and that there is often a strong involvement of managing authorities in the work of formally independent evaluators. These factors contribute to making the findings of the evaluations overly optimistic (beautiful) risking their overall usefulness (evidence-based policy). We conclude by discussing reform options to make the evaluations of EU Cohesion Policies more unbiased and effective.