Evidence-based Instruments

 

To what extent do government ministries regularly evaluate the effectiveness and/or efficiency of public policies and use results of evaluations for the revision of existing policies or development of new policies?

EUOECD
 
Ex post evaluations are carried out for all significant policies and are generally used for the revision of existing policies or the development of new policies.
10
UK
Evidenced-based decision-making is deeply rooted in the United Kingdom’s tradition of firm and efficient legislation, and ex post evaluations are as vital a part of public policymaking as impact and sustainability checks. The OECD ranks the United Kingdom second among its 40 members for its approach. Specialist analytical functions have recently been brought together to form the Analysis Function, further strengthening their role in policymaking and service delivery. Analytical approaches to evaluation are set out by the Magenta Book and the Green Book, supported by the Cross-Government Evaluation Group coordinated by HM Treasury. More recently the Behavioral Insights Team (formerly a team within the Cabinet Office, but now an independent entity) and the What Works Network (coordinated by the Cabinet Office) promote the increased use of evaluation methods, especially randomized controlled trials.

The notion here is to make government more efficient, and so to quickly identify and overcome unnecessary regulatory hurdles. In 2011, the previous coalition government launched the “Red Tape Challenge,” encouraging citizens to report unhelpful or burdensome legislation that could be cut or reformed. However, the main focus of the United Kingdom’s commitment to ex post evaluation is surely on facilitating business.

The Regulatory Policy Committee (RPC) assesses the quality of the government’s RIA analyses and further examines all published ex post evaluation. If the RPC submits a recommendation to the government, it is expected to be implemented into law. Further, businesses can directly address the RPC if they disagree with or feel disadvantaged by a specific governmental regulatory assessment.
Citations:
OECD Regulatory Policy Outlook 2018
9
Finland
Consultation with experts and stakeholders is a natural phase in the Finnish lawmaking process. In addition, the public is invited to comment on draft proposals online. Furthermore, all proposals for changing statutes must be accompanied by an assessment of their impact across several aspects of society (e.g., the economy and environment). However, the OECD has pointed out that although ex post evaluations are frequently carried out, Finland lacks a systematic strategy for the ex post evaluation of regulations.
Citations:
“Säädösehdotusten vaikutusten arvionti,” Oikeusministeriö, http://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/76082/saadosehdotusten_vaikutusten_arviointi_ohjeet.pdf?sequence=1&isAllowed=y
OECD: “Better Regulation in Europe: Finland.” http://www.oecd.org/gov/regulatory-policy/45054846.htm.

OECD (2018), OECD Regulatory Policy Outlook 2018, OECD Publishing, Paris, https://doi.org/10.1787/9789264303072-en.

Opinion service webpage: lausuntopalvelu.fi

Governments Registry for Projects and Initiatives (http://valtioneuvosto.fi/hankkeet).
Switzerland
Article 170 of the constitution states that “(t)he federal parliament shall ensure that the efficacy of measures taken by the confederation is evaluated.” Ex post evaluations have been strongly developed and are, to varying degrees, standard in most policy fields. Evaluations are best established in the fields of development cooperation, public health, education and economy. Ex post evaluations are a most important source of information for the revision and development of policies when used by the administration. Administration experts draft laws and reports based on the available empirical evidence, which mostly consists of policy evaluations. However, the administration formulates drafts that are subject to pre-parliamentarian and parliamentarian policymaking processes that include many relevant actors that do not prioritize evidence. The main goal of policymaking in Switzerland is acceptance rather than evidence-based policy. However, as many evaluations focus on learning within the administration rather than executive control and new legislation, the impact of evaluations remains significant in Switzerland. The Swiss evaluation community is one of the most professional in Europe and evaluations are of good quality. The role of ex post evaluations in Switzerland can thus be considered important.
Citations:
Fritz Sager, Thomas Widmer, Andreas Balthasar (eds.) 2017: Evaluation im politischen System der Schweiz. Entwicklung, Bedeutung und Wechselwirkungen, Zürich: NZZ Verlag
 
Ex post evaluations are carried out for most significant policies and are used for the revision of existing policies or the development of new policies.
8
Denmark
Existing regulations of RIAs do not require ex post evaluations. Such evaluations are part of the ongoing political process. Political agreements either explicitly or implicitly settle policies for some period. Though various events and developments may require subsequent action, which would provide a chance for citizens, media outlets and policy entrepreneurs to advocate policy reforms. This happens regularly in the Danish political system and very often the government or the opposition will suggest new policies. Sometimes expert committees will be set up to analyze the issues. Most significant policies are regularly debated and policy reforms are common.
The annual budget preparation is one occasion for evaluating policies. The parliament’s Rigsrevision (auditor general) also issues an annual report, which may lead to policy reforms. In some cases, an assessment is made an explicit part of a political agreement (e.g., labor market policy).
Citations:
Vejledning om konsekvensanalyser Maj 2005 (afsnit 3.2, 7.1 og 7.2 revideret i 2018). https://modst.dk/media/19528/vejledning-om-konsekvensanalyser-maj-2005-afsnit-3-2-7-1-og-7-2-revideret-i-2018.pdf (Accessed 10 October 2018),
Jørgen Grønnegård Christensen et al, Politik og forvaltning. 4. udg. Reitzels Forlag, 2017, p.300.
Norway
The Norwegian government makes use of evaluations in most policy sectors and issue areas. Evaluations are carried out by external experts or internal ministerial review bodies. Evaluations are sometimes intended to measure the effect of reforms, although more frequently evaluations serve as a starting point for a future reform process. There is broad support for evidence-based policymaking and the results of policy evaluations tend to attract considerable attention.
Sweden
Ex post evaluations take various forms in the Swedish system. For the past 10 to 15 years, performance measurement and management has been an integral part of public management in Sweden, as in most other countries. Also, audits conducted by the Swedish supreme audit institution (Riksrevisionen) are important evaluation instruments. There is, however, a tendency to focus more on institutions and cost efficiency (the audit approach) than on programs and impact (the evaluation approach) – a trend that is increasingly noticeable across many Western countries. That said, both approaches are useful as feedback on public policy.
Citations:
Pierre, J., B. G. Peters and J. de Fine Licht (2018), “Is auditing the new evaluation? Can it be? Should it be?,” (tillsammans med B. Guy Peters och Jenny de Fine Licht), International Journal of Public Sector Management 31:726-39.

B. G. Peters and J. Pierre (2019), ““From evaluation to auditing and from programs to institutions?: Causes and consequences of the decline of the program approach,” Governance: An International Journal of Policy, Administration, and Institutions.
7
Canada
As with other aspects of Canada’s RIA regime, ex post evaluation differs between laws and regulations. While laws are not subject to systematic ex post evaluation, departments and agencies are expected to regularly review existing regulations to assess their impacts and develop refinements.

In 2016, the Treasury Board of Canada introduced a new “Policy on Results.” Evaluations of programs, policies and priorities under the policy is to be a “systematic and neutral analysis of evidence related to relevance, effectiveness and efficiency of policies and programs,” and should “generally employ social science research methods.” Those evaluations can be done through the Treasury Boards’ resource alignment reviews and internally by departments themselves.

In practice, it is frequently the Office of the Auditor General of Canada (OAG) that evaluates government programs and initiatives ex post. The OAG is formally charged with so-called performance audits, which aim to provide an independent, objective and systematic assessment of whether government programs are being run with due regard for the economy, efficiency and environmental impact. The OAG has considerable discretion regarding which programs it will examine, and takes requests from parliamentary committees, members of parliament, citizens, civic groups and other parties to conduct audits in specific areas. It conducts between 25 and 30 performance audits each year and publishes the results.
Citations:
Treasury Board of Canada, Cabinet Directive on Regulation, available at https://www.canada.ca/en/treasury-board-secretariat/services/federal-regulatory-management/guidelines-tools/cabinet-directive-regulation.html

Treasury Board of Canada, Policy on Results, available at https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=31300
Estonia
The Strategy Unit of the Government Office is responsible for the quality of policymaking, including the evaluation of policy effectiveness and development of a knowledge base for future reforms. For 2014 – 2020, European Social Fund resources have been allocated for these activities, which has resulted in an increased number of studies of ex post and ex ante policy impact analyses. Nevertheless, the overall framework remains a work in progress, as the evaluations do not cover all significant policies and are not systematically used for the development of new policies. Since the Government Office has limited analytic capacity, the studies are produced by external national and international research teams.
France
There is no practice of systematic evaluation, except for policies or laws in which the respective constitutive act stipulates the need for an evaluation. However, over the past 25 years, the Court of Accounts, which previously exerted a legalistic type of oversight, has transformed its mission and adapted its methods so as to evaluate public policies from a political, social, economic and financial point of view. The Court’s reports have become reference documents not only for the political authorities (government and parliament), but also for the opposition, the media and the broader public. The reports are usually characterized by rich analysis and accurate criticisms, and the recommendations are usually well received. The parliament and the government rarely challenge the courts’ conclusions and recommendations, which often become the basis for new legislation. Since Sarkozy’s time in office, the nominee for president of the court has always been a former politician from the opposition (at the time of appointment). This pattern has strengthened the legitimacy of the court, and allowed for the adoption of more policy-oriented evaluations. This dimension is not negatively perceived, as the Court is not seen as biased in its conclusions; indeed, its pragmatic suggestions are seen as useful in the preparation of new legislation.
Germany
In general, government ministries are solely responsible for policy results and, as a consequence, evaluate the effectiveness of policies within their domain. Evaluations are often part of a ministry’s daily procedures.

Traditionally, evaluation of legislation in Germany has an ex ante character. Ex post evaluations are not yet universally undertaken. These study the causal impacts of policies based on credible experimental methods, for example, through comparisons of a treatment group with a control group. Today, thorough ex post analyses are used in the fields of labor market, education and family policies. A milestone for ex post labor-market research was the introduction of a legal obligation to evaluate the impact of active labor-market policies in 1998. Since then, important legislation such as labor-market and social security reforms (Hartz reforms), and later the introduction of minimum wages have undergone far-reaching ex post evaluations.

In December 2018, the government agreed on a program for better lawmaking and bureaucracy reduction (“Bessere Rechtsetzung und Bürokratieabbau”). Application of this program is still in the early stages. Thus, while some standards for high-quality ex post evaluation have been set for some important policy areas, use of these methods has yet to be implemented across the broader policy field.
Citations:
Boockmann, B., Buch, C. M., Schnitzer, M. (2014): Evidenzbasierte Wirtschaftspolitik in Deutschland: Defizite und Potentiale, IAW Discussion Paper Nr. 103, April 2014.

Normenkontrollrat (2018):
https://www.normenkontrollrat.bund.de/nkr-de/aktuelles/bundesregierung-setzt-wichtige-impulse-fuer-buerokratieabbau-und-bessere-rechtsetzung-1558986
Japan
Government ministries evaluate their policies on an ex post basis. The Administrative Evaluation Bureau (AEB) conducts inspections, and each ministry carries out independent evaluations of the effects of its own policies. The AEB supports such activities, for instance by encouraging ministries to share methodologies and experiences. It also works to standardize and prioritize policy evaluations, and reviews ministry and agency evaluations.

However, Japan is ranked comparatively low in an OECD ex post evaluation index for 2014. In 2017, a new rule was introduced requiring regulatory enforcement ministries and agencies, which previously had primarily evaluated regulations themselves, to engage in ex post evaluation.

At this stage, it is difficult to judge the quality of the ministerial evaluation efforts, or whether such activities have indeed become a major source of information in the reformulation of policies and projects. Such evaluations do not play a significant role in public-policy discussions.
Citations:
Council for Promotion of Regulatory Reform, Third Report by the Council for Promotion of Regulatory Reform – For New Era to Come, Provisional Translation, 4 June 2018

Ministry of Internal Affairs and Communications (Administrative Counseling Division), Japanese Ombudsman System, Tokyo, March 2018

OECD, Government at a Glance 2017
Lithuania
Government ministries sometimes evaluate the effectiveness and/or efficiency of public policies, but most evaluations are related to the use of EU funds; it is mandatory to evaluate the implementation of operational programs financed from EU structural and investment funds. For instance, 63 evaluations were performed during the programming period from 2007 to 2013. Many evaluations were executed during the 2014 to 2020 period. For instance, annual evaluation plans contain about 10 evaluations each year.

The implementation of recommendations derived from these evaluations is monitored on a regular basis, but a 2013 study revealed that only about 60% of all recommendations provided by evaluators had been implemented by Lithuanian ministries or other state institutions. This average rate of implementation was attributed to insufficient institutional and staff capacities in the administration; this in turn reduces the demand for evaluations, hinders quality-assurance efforts and limits the use of evaluation results. The administration has also showed limited progress in implementing State Audit Office recommendations.

Although Lithuanian authorities are also in theory supposed to assess the implementation of national budget programs, no such evaluations have been completed since 2016. This is in large part due to the ongoing restructuring of the strategic-planning and budget-formulation system. Although a new draft law on strategic management refers to the Strategic Planning Methodology in its sections on evaluation, it remains unclear how national evaluations will be conducted in the future.
South Korea
In a recent OECD assessment of ex post evaluation, Korea was ranked third behind Australia and United Kingdom. While there is an effective evaluation system in place, the effect on policy revision has been limited to date. In recent years, real-estate policy has been broadly unsuccessful, contributing to historically high and skyrocketing prices by repeatedly imposing low-quality short-term regulations without effective ex post evaluation. Soaring prices have produced adverse effects by increasing economic inequality.
Citations:
OECD, Government at a Glance Database,
https://www.oecd-ilibrary.org/sites/9789264303072-en/1/2/2/index.html?itemId=/content/publication/9789264303072-en&_csp_=46cd5ed37844c9dc6cf21c84716ab307&itemIGO=oecd&itemContentType=book
6
Australia
Ex post evaluation of public policies is not a mandated or standard part of policy implementation in Australia. However, Australia was ranked first in ex post evaluation in the OECD Government at a Glance database. While an effective evaluation system is in place, the effect on the revision of policies has so far been limited. Periodically, policies are announced with explicit accommodation for their evaluation, but this is the exception rather than the norm. In general, evaluation is more likely to occur for policies developed at the departmental or agency level (as opposed to the top level of government) or for policies that are not politically important/sensitive.
For policies that are evaluated, the quality of evaluation is variable, and rarely is the “gold standard” of a randomized controlled trial used.
Citations:
https://de.slideshare.net/OECD-GOV/effective-ex-post-evaluation-purpose-and-challenges
Chile
Since 1997, the Ministry of Finance’s Budget Office (Dirección de Presupuestos, DIPRES) has had the power to assign specific budgets to line ministries for the contracting of external consultants to carry out ex post evaluations of their government programs (Evaluación de Programas Gubernamentales, EPG). Programs or institutions to be evaluated are agreed on with the Congress annually, with the instruction coming via ministerial decree. The evaluation results are normally made publicly and freely available.
Citations:
http://www.dipres.gob.cl/598/w3-article-111762.html

Evaluación Ex Post: Conceptos y Metodologías (Dipres 2015)
http://www.dipres.gob.cl/598/articles-135135_doc_pdf.pdf
Czechia
Ex post evaluations are a regular part of the RIA process. However, they are usually carried out internally by individual departments and, save for measures financed with EU funds, have not been done in a systematic fashion so far. With the goal of changing this situation, the Government Office presented a comprehensive proposal in mid-2018 for improving the quality of ex post evaluations. An interdepartmental working group is currently drafting new formal guidelines for ex post evaluations that would make findings public and render the whole process more transparent.
Israel
In 2015, the OECD Regulatory Policy Outlook stated that Israel had made progress in improving regulatory policy across several fields, including ex post evaluations. Though the report also determined that Israel had no general policy on ex post evaluation and that government offices and ministries operate without clear guidelines. Since then, Israel has invested in improving its ex post evaluations.

In 2018, an OECD report, “The Long View: Scenarios for the World Economy to 2060,” concluded that Israel had improved many regulatory fields, including ex post evaluations. This, to our understanding, is a continuation of the process mentioned in the OECD report from 2017, which highlighted the general improvement in regulatory processes. The report also concluded that Israel has improved its consultation processes by opening the processes up to the public and integrating them into the RIA system. While Israel still focuses on reducing the regulatory burden, it seems that it is working harder on providing a solid basis for a general government regulatory policy.

The 2018 OECD Regulatory Policy Outlook stated that while Israel continues to make significant progress in improving its regulatory policy, government ministries are conducting legislative planning in isolation with limited interministerial coordination. This is one of the key factors behind the inflation in regulatory activity. The government is working on reforming the planning system.
Citations:
“Israel could gain from increased competition, public spending, OECD says “, Times of Israel, 2018 (Hebrew):
https://www.timesofisrael.com/israel-could-gain-from-increased-competition-public-spending-oecd-says/

“OECD economic scenarios to 2060 illustrate the long-run benefits of structural reforms,” OECD Website,
http://www.oecd.org/economy/oecd-economic-scenarios-to-2060-illustrate-the-long-run-benefits-of-structural-reforms.htm

“OECD Regulatory Policy Outlook 2015,” OECD Website,
https://www.oecd.org/gov/regulatory-policy/Israel-web.pdf

“OECD Regulatory Policy Outlook 2018“, OECD WEBSITE, 2018
https://www.oecd-ilibrary.org/docserver/9789264303072-en.pdf?expires=1540849342&id=id&accname=ocid195467&checksum=2F91C900404B34AA4EE7659685539C79

“Government at a Glance 2017 – Israel,” OECD Website,
https://www.oecd.org/gov/gov-at-a-glance-2017-israel.pdf
Latvia
Currently ex post evaluation is carried out for development planning documents, as prescribed in the Development Planning System Law. In addition, the “methodology for developing and evaluating the results and performance indicators for ministries and other central state institutions” provides general guidelines for ministerial reporting. However, there is currently no common approach to the evaluation of legislation post-implementation, although institutions are allowed to order research studies (including ex post impact studies) at their own discretion.

Recognizing the need for a unified approach and clear regulation, the Cabinet of Ministers approved the State Chancellery’s concept report on ex post evaluations in 2016. The report considered several potential approaches, before recommending that two pilots should be carried out in 2017, which would then be used to finalize a new policy in 2018. This policy is currently pending.
Citations:
1. State Chancellery (2016), Annual Report (in Latvian), Available at: https://www.mk.gov.lv/sites/default/files/page/attachments/valsts_kancelejas_2016.gada_parskats.pdf, Last assessed: 05.11.2019

2. State Chancellery (2017), Annual Report (in Latvian), Available at: https://www.mk.gov.lv/sites/default/files/page/attachments/valsts_kancelejas_gada_parskats_2017_final_6.pdf, Last assessed: 05.11.2019

3. Development Planning System Law (in Latvian), Available at: https://likumi.lv/doc.php?id=175748, Last assessed: 05.11.2019

4. Methodology for developing and evaluating the results and performance indicators for ministries and other central state institutions (regulation), Available at: https://likumi.lv/doc.php?id=200935, Last assessed: 05.11.2019

5. Cabinet of Ministers (2016), Report on Ex-post Evaluation Implementation (in Latvian), Available at: http://tap.mk.gov.lv/lv/mk/tap/?pid=40386136, Last assessed: 05.11.2019
Malta
In recent years, ex post evaluations have been carried out for most significant policies. Various tools are used, and supported by enhanced digital processes. Improvements in ministerial coordination have also contributed to this development, along with ongoing review by the Office of the Principal Permanent Secretary. A “mystery shopper” for government departments was introduced in 2017, tasked with identifying shortcomings in service delivery so that they could be remedied accordingly. The National Audit Office performs audits to determine whether government entities have adequate systems of internal controls in place, with follow-up audits conducted to determine whether identified weaknesses have been dealt with. The 2018 follow-up audit report paints a mixed picture, with recommended improvements fully or partially implemented in some cases, and no changes made in others. The 2019 OECD report on Regulatory Policy and Governance indicates that Malta is well below the OECD average in terms of the ex post evaluation of regulations. Additionally, Malta has no entity that can take legal or regulatory action against consultants who present flawed reports, or who mislead the Environment and Resources Authority or Planning Authority.
Citations:
Follow Up Reports by the National Audit Office 2018 http://nao.gov.mt/en/recent-publications
Malta Today 03/07/2018 Environment Impact Assessments still unregulated after 20 years
Mexico
Overall, Mexican policies are supposed to subjected to ex post evaluation and, at least at the national level, a comparatively strong culture of ex post evaluation has grown over the last two decades. This phenomenon is rooted in two different ideological streams. On the one hand, the traditional planning euphoria from a left-leaning, corporatist system has embraced the idea of ex post evaluation as an integral part of a well-mastered policy cycle. On the other hand, market liberal reforms and the influence of international finance organizations have introduced forms of new public management, with rigorous ex post evaluation seen as a crucial way to guarantee the efficiency and effectiveness of public policies.

Since 2012, mandatory guidelines require the use of ex post evaluations. As a consequence, Mexico has established one of the most developed evaluation policies in the field of social policy, which is executed by a constitutionally anchored evaluation agency, CONEVAL. Created in 2014, CONEVAL is established as an autonomous constitutional organization with a very high level of technical and management autonomy. Its task is to coordinate and manage the ex post evaluation of national social policies, and it does so with a very high level of sophistication. It is consists of the head of the Ministry of Social Development, six well-known academics chosen by the National Social Development Commission and an Executive Secretary who is in charge of the council. In the international realm, CONEVAL has been an institutional innovation in poverty measurement and the evaluation of social public policies. So far, the independent provision of scientifically based evaluations has had a substantial impact on technically improving social policy programs in Mexico.

While CONEVAL represented a substantial move forward in the evaluation of social policies, a major setback occurred in mid-2019 when the head of CONEVAL was replaced after having criticized cuts to the agency’s budget. Given his prior critiques on the governments’ austerity policies, the replacement was perceived as a serious blow to the agency’s independence. It remains to be seen whether CONEVAL will continue to act as a critical but constructive evaluation agency.

Beyond the field of social policy, however, other policy fields are subjected to far less scientific ex post evaluations and, at the subnational level, much more remains to be done. However, the ongoing presence of many organizations of international development cooperation in Mexico as well as promising dynamics at the subnational level is favorable to the development of an evaluation-friendly environment.
New Zealand
Despite the dominant role of ex-ante RIAs, steps are also taken to implement ex post RIAs. However, the quality of ex-post evaluation of public policies remains unclear. They are not mandatory and there is no established methodology for conducting them. Introducing systematic reviews of regulation could potentially help strengthen the policymaking process. There is little evidence to assume that this has changed in the review period of the SGI 2020.
Citations:
https://treasury.govt.nz/publications/legislation/regulatory-impact-assessments
Gill, Derek 2016. Rgulatory Coherence: The Case of New Zealand. ERIA Discussion Paper Series 2016-12. Wellington: University of Wellington.
OECD Regulatory Policy Outlook 2015 Country profile New Zealand. https://www.oecd.org/gov/regulatory-policy/New%20Zealand-web.pdf
Kupiec, Tomasz 2015. Regulatory Impact Analysis Practice in New Zealand in the Light of Models of Evaluation Use – Inspiration for the Polish Government. „Management and Business Administration. Central Europe” 23(2), pp. 109–128.
Slovenia
Ex post evaluations are regularly carried out for the most significant policies, but rarely for all other policies. When carried out, ex post evaluations are primarily used for the improvement of existing policies rather than for the development of new policies.
Spain
The 2015 law on the common administrative procedure includes a provision regarding ex post evaluation, making it compulsory to evaluate the application of rules and norms in order to ascertain whether they have met the objectives pursued, and whether costs have been justified.

The oversight body for ex post evaluations, the Office on Regulatory Coordination and Quality became active in 2018. In theory, the office is responsible for the ex post evaluation of all public policies. However, during the period under review, no data regarding the activities of the office was available. In 2018, new evaluation procedures were approved in order to consider the impact of regulatory drafts on competition and SMEs, and new thresholds were introduced for the conduct of ex post evaluations.

Nevertheless, according to OECD recommendations published in 2019, Spain should develop standard evaluation techniques for ex post evaluation since the ex post review system is still in its early stages and has not yet been implemented systematically.

Most autonomous communities have developed additional, systematic ex post evaluation programs for their public policies. However, the practical impact of these measures has to date been limited.

The Independent Authority for Fiscal Responsibility (AIReF) is responsible for the evaluation of public spending and the efficiency of public policies. The AIReF analyses key expenditure and investment items such as education, healthcare and public works. The reports are not binding, but if the administration chooses not to follow the recommendations, it must justify this decision. In 2018, the AIReF was given greater access to information and an increased budget.
Citations:
Ley 39/2015. Royal Decree 105/2018

OECD, 2019 INDICATORS OF REGULATORY POLICY AND GOVERNANCE: SPAIN
http://www.oecd.org/gov/regulatory-policy/indicators-of-regulatory-policy-and-governance-2019-spain.pdf
Netherlands
The General Audit Chamber (Algemene Rekenkamer) scrutinizes ex post policy evaluations by ministerial departments. Since 2000, the chamber has reported its findings to parliament on the third Wednesday in May each year. In 2012, the government introduced the Regulation for Regular Evaluation Studies, which specifies research criteria for assessing policy efficiency, goal achievement, evidence-based policymaking and subsidy-based policies. Yet, time and again, the chamber has reported deficits in goal achievement and weaknesses in goal formulation, which undermine the quality of ex post evaluation research. Other weaknesses in policy evaluation studies include the lack of citizen perspectives, inability to accurately calculate societal costs and benefits, overreliance on input from implementing organizations for evidence and lack of public access to many evaluations. In line with the general trend toward more instrumental advice, over the last couple of years, the General Audit Chamber has focused its attention on specific points in departmental agendas.
Moreover, there are a wide range of additional non-obligatory evaluations produced by ministerial departments, parliament, government-sponsored knowledge institutes, the ombudsman, implementation bodies and quasi-independent non-governmental bodies. Since evaluation findings are just one factor in designing new or adjusting existing policies, it is not clear how much policy learning actually occurs. A recent study commissioned by the minister of finance assessed past evaluations and their use. The study confirmed that although “no other country evaluates so many of its policies,” policymaking civil servants and members of parliament are less sensitive to the outcomes of previous policies than to images and incidents (in the press). Moreover, obstruction and disinterestedness contribute to methodological weaknesses in many of the evaluation studies, this assessment found. For example, although the government agreement stipulates that a new policy decriminalizing the use of hashish may be experimentally tested at the local level, interference in the study’s design has already made a politically unbiased evaluation of results as good as impossible.
Dutch ex post evaluators closely follow international trends of “evidence informed” and “behavioral knowledge” evaluation studies. There has been a tendency to move away from a focus on single, case-specific ex post evaluation studies to a focus on the construction of broader, more balanced departmental knowledge portfolios, in which ex post evaluation studies are embedded as elements in a larger body of knowledge accessible to policymakers and other participants in policy subsystems. It is not yet clear to what extent such trends in evaluation research really inform evaluation practices.
Citations:
A. Knottnerus, Van casus-specifieke beleidsevaluatie naar systematische opbouw van kennis en ervaring, Beleidsonderzoek Online, May 2016

Meyken Houppermans, ‘Wat is de toegevoegde waarde van de onafhankelijk
deskundige bij beleidsdoorlichtingen?’, Beleidsonderzoek Online juni 2018,
DOI: 10.5553/BO/221335502018000005001

A. Korsten, Wat ervan terecht komt. Zicht op beleidsevaluatie, April 2013 (arnokorsten.nl, accessed 31 October 2018)

NRC Next, 28 August 2019. Gemeenten die meedoen aan experiment met legale wietteelt uitgelekt. (NRC.nl, accessed 8 November 2019)

SEO Economisch Onderzoek, december 2018. Beleidsdoorlichtingen belicht (SEO-rapport nr. 2018-110, sep.nl, accessed 8 November 2019

NRC Next, Gemeenten krijgen toch wat ruimte om wiebelend te regelen. ((NRC.nl, accessed 8 November 2019)
 
Ex post evaluations are rarely carried out for significant policies and are rarely used for the revision of existing policies or the development of new policies.
5
Italy
Italian governments and parliaments rarely conduct in-depth ex post evaluations of policies before promoting a revision of existing or the development of new policies. Any new bill is accompanied by a technical report, which typically discusses existing policies and explains the benefits of the new policy. But only rarely is this the result of an evaluation process conducted through a public and open debate. A good example was the revision of the Fornero pension reform of 2011 adopted by the first Conte government. This revision (the so-called Quota 100) resulted from an election-campaign pledge by the Northern League, but its implementation was not preceded by any serious cost-benefit analysis. The second Conte government’s decisions to retain both the expensive pension revision and the citizen’s income were also made without any serious ex post evaluation of these two policies.
USA
The United States has extensive and highly sophisticated capabilities for evaluating the impact of public policies – within government, in a large sector of think tanks and consulting firms, and in the extensive public policy-oriented academic community. In normal times, however, U.S. government is relatively inflexible, and slow to respond to evidence of the need for change. This is a widely recognized consequence of the separation-of-powers constitutional system, which was designed to inhibit policy change. In particular, the institutional barriers to policy change enhance the ability of organized groups to resist reforms that they oppose.

The lack of interest in information about the consequences of policies reached a new and quite extraordinary level during the first two years of the Trump presidency. In some cases, the Republican leadership overrode congressional rules by refusing to wait for CBO analysis before voting on bills. Policy journalists have described an unprecedented indifference among senior government officials to evidence of the actual effects of policy. Effective use of evaluation information is limited to issues that are not on presidential or partisan agendas, and independent regulatory commissions (with multiple commissioners, serving fixed terms) that are not yet controlled by Trump appointees.
4
---
---
3
Austria
Ex post evaluation is a rather unknown field in Austrian politics. The lack of any systematic ex post evaluation tradition and the tendency of political actors to prioritize the next election over all other perspectives makes it highly unlikely that the present government or parliament will establish a structure of ex post evaluations. The absence of long-term strategies, beyond traditional vague ideologies (like social justice or defending Austrian identity), prevent any reasonable systematic ex post evaluation. Though one exception concerns electoral campaigning. Following the priority given to electoral strategies, parties in Austria reflect systematically on the reasons for any specific electoral outcome, which may be viewed as a specific version of ex post evaluation. This did not change under the previous coalition government, which collapsed in 2019.

The only systematic ex post evaluation involves the Austrian Court of Audit’s control over particular financial aspects of specific government or government-sponsored projects.
Bulgaria
The rules for impact assessments in Bulgaria established in 2016 require an ex post evaluation of policies and their effects within five years of their implementation. By the end of the review period in 2019, only two such evaluations had been published through the government’s public-consultation portal.
Croatia
The process of Europeanization and Croatia’s EU membership have opened the space for the use of evaluation studies in Croatian public administration at the central and local government levels. Methods and theoretical approaches to evaluation are exchanged through the Croatian Evaluation Network, which is comprised of experts interested in evaluation practice. However, Croatian policymakers are yet to introduce effective ex post evaluation mechanisms for the already passed legislative and regulatory measures. Thus, ex post evaluations of significant policies are rarely carried out, and are even more rarely used by policymakers as a source of evidence and inspiration. If ex post evaluations are carried out, the success indicators tend to be too general and insufficiently precise.
Hungary
There is no formal framework for carrying out ex post evaluations in Hungary. Such evaluations are rarely carried out since the Orbán governments have been more interested in exercising political control than in the effectiveness of their measures.
Ireland
The extent of overspending in healthcare totaled more than €2 billion over the last four years according to the Irish Fiscal Advisory Council. Such budgetary over-runs suggests that there is little ex post evaluation of policy in this significant budgetary area. Each year the government is presented with an over-run that is tacitly accepted and paid for out of buoyant tax revenues. However, if tax revenues fall, the ability of the government to fund such over-runs will create significant political tensions.
Poland
According to the RIA rules, ex post evaluations can be requested by the Council of Ministers. In practice, however, primary laws and subordinate regulations are only rarely evaluated. Even when such procedures are carried out, the outcome of the review is not used to improve the quality of laws.
Citations:
OECD (2018): OECD Regulatory Policy Outlook 2018. Paris, 222-223 (https://www.oecd.org/gov/regulatory-policy/poland-country-note-regulatory-polic y-2018.pdf).
Portugal
A recent study noted that ex post evaluation is very weakly institutionalized in Portugal. The ex post evaluations that take place are largely the result of external pressures (notably, EU requirements) and circumscribed to a limited set of policy areas – usually, those that legally require ex post evaluation.
Citations:
Ana Diogo (2018), “Hacia un sistema integrado de M&E para los objetivos de desarrollo sostenible en Portugal” [dissertation], available online at: https://www.somosiberoamerica.org/investigaciones/la-evaluacion-de-la-agenda-2030/

A Avaliação de Políticas Públicas by the GEE – O Gabinete de Estratégia e Estudos, do Ministério da Economia
Romania
While the institutionalization of ex post evaluations has been announced several times, they have remained the exception rather than the rule. If such evaluations have been done, their impact on decision-making has been intransparent.
Slovakia
Ex post evaluations have not been required by law so far. If they have been carried out, they have focused mainly on administrative burdens. The RIA 2020 Better Regulation Strategy, approved by the cabinet in January 2018, envisaged more comprehensive ex post evaluations. However, implementation of the strategy has been slow.
Citations:
OECD (2019): Indicators of Regulatory Policy and Governance Europe 2019: Slovak Republic. Paris (http://www.oecd.org/gov/regulatory-policy/indicators-of-regulatory-policy-and-g overnance-2019-slovak-republic.pdf).
 
Ex post evaluations are generally not carried out and do not play any relevant role for the revision of existing policies or the development of new policies.
2
Belgium
The typical strategy is to pick the data that justify the decisions that have been made. For instance, to justify the usefulness of a subsidy, the government may argue that a large number of people demanded the subsidy, not whether the subsidy managed to achieve its political objective. This approach has led to counterproductive decisions in the areas of, for example, education, energy conservation, subsidies for solar panels and immigration. There may be some scattered ex post evaluations undertaken on the initiative of individual line ministries, but these evaluations have no direct impact on the revision of existing policies since they are not seriously considered by ministerial cabinets, where all strategic policy choices are initiated and arbitrated.
Cyprus
Given the deficiencies in strategic planning, ex post evaluation has not been part of government practice. However, in the framework of commitments to reform the public administration, the government in recent years solicited the drafting of studies by the World Bank and others. These studies on the administration and line ministries aimed mainly at restructuring and reforming the administration and its procedures, while also reviewing their general policies. Specific policies in ministries are not evaluated and progress achieved so far remains limited.

The administration acknowledges that sectoral reviews should form part of strategic planning.
Citations:
Public Administration Reform – Restructuring ministries and administration http://www.reform.gov.cy/en/public-administration-reform/restructuring-in-the-public-sector
Greece
Ex post evaluations are not yet part of the Greek government’s regulatory-management toolbox. There is frequent turnover among ministers, and it is rare for a minister to stay in office long enough to evaluate a measure he or she adopted. After government reshuffles and certainly after a change in government, incoming ministers rarely are interested in evaluating the policies pursued by their predecessors. Notwithstanding, there are in the public administration and central headquarters of ministries, higher-ranking civil servants who could perform such evaluations if asked to do so. The potential is there, but it is rarely used. However, the Greek government has in some cases asked international organizations (e.g., the OECD) or think tanks to evaluate policy impacts. The Syriza-ANEL government was rather reluctant to ask for external advice. In contrast, the New Democracy government in power since July 2019 has shown itself to be more open to the prospect of establishing proper ex post policy-evaluation processes.
Iceland
No regular ex post evaluations of the effectiveness or efficiency of public policies appear to be conducted by any government ministry. However, that does not mean that no such evaluations take place.
Luxembourg
Legislation is rarely modified through a process of ex post evaluation. One counterexample is provided by the Violence Protection Act, which was evaluated in 2009 after five years in force. High-quality evaluation studies are rarely carried out. There is also a shortage of personnel able to design legal texts. Rigorous state-commissioned evaluation of legislation is thus unlikely in the near future.
Citations:
“Evaluation „Fünf Jahre Gewaltschutzgesetz im Grossherzogtum Luxemburg.” http://mega.public.lu/fr/publications/publications-ministere/2010/evaluation-gewaltschutzgesetz/Evaluation_Gewaltschutzgesetz.pdf. Accessed 19 Oct. 2019.
Turkey
Currently, the capacity of public policy assessment institutions in Turkey lack knowledge regarding evidence-based instruments, both theoretically and practically. RIA practice was introduced by Law No. 5018 on Public Financial Management and Control (2003), and other relevant regulations. In this context, a cost-benefit analysis is required for all public agencies. However, there are various social and political barriers to the evaluation of public policies. For example, the parliament and judiciary cannot effectively supervise and review executive actions. It is unlikely that an effective evaluation mechanism will be developed in the near future. However, there are a few academic studies that have evaluated certain public policies.

The Annual Presidential Program of 2019 makes references to monitoring and assessment but not to the systematic ex post evaluation of public policies. There are a few ex post analyses of public policies, mainly in the health sector, that address citizen/patient satisfaction, self-assessment and the assessment of action plans.
Citations:
TC Cumhurbaşkanlığı, 2019 Yılı Cumhurbaşkanlığı Yıllık Programı, https://www.sbb.gov.tr/wp-content/uploads/2018/11/2019_Yili_Cumhurbaskanligi_Yil lik_Programi.pdf (accessed 1 November 2019)

TC Sağlık Bakanlığı Faaliyet Raporları, https://sgb.saglik.gov.tr/dokumanlar/faaliyet-raporlari (accessed 1 November 2019)

Aydın Usta, “Yeni Kamu Yönetiminde Politikaların Değerlendirilmesi: Tipolojiler, Boyutlar ve Ölçütler, ” Sayıştay Dergiṡi̇, 94/Temmuz-Eylül 2014, 5-27.

İbrahim Arap and Veysel Erat, “Bir Kamu Politikasının Analizi: Türkiye’de Geçici Köy Koruculuğu,” Mülkiye Dergisi, 39 (4), 2015: 73-108.
1
---
---
Back to Top