The Institutionalisation of Evaluation in Asia-Pacific 3031369173, 9783031369179

Table of contents :
Preface and acknowledgements
Contents
Abbreviations
Part I Introduction
1 The Institutionalisation of Evaluation: Theoretical Background, Analytical Framework and Methodology
1 Introduction
2 State of Research and Theoretical Background
2.1 State of Comparative Research on the Institutionalisation of Evaluation
2.2 Theoretical Background
2.3 Particularities of the Asian Pacific Region
3 Institutionalisation of Evaluation—A Systemic Perspective
3.1 Institutionalisation of Evaluation in the Political System
Institutional Rules
Practices
Organisations and Organisational Structure
3.2 Institutionalisation of Evaluation in the Social System
3.3 Institutionalisation of Evaluation in the System of Professions
3.4 Summary
4 Methodological Remarks
4.1 Comparison in Research on Evaluation
4.2 Case Selection and Research Process
4.3 Epistemology and Reflection
Appendix 1: Glossary for the Evaluation Globe—Compendium on the Institutionalisation of Evaluation
Appendix 2: Analytical Framwork: Compendium on the Institutionalisation of Evaluation
References
Part II National Developments
2 Evaluation in Australia
1 General Country Overview
1.1 Brief History of Evaluation
2 Institutional Structures and Processes
2.1 Evaluation Regulations
2.2 Parliamentarian and National Audit Structures
2.3 Organisational Structure
2.4 Evaluation Practice and Use
3 Societal Dissemination and Acceptance of Evaluation in Civil Society
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Discussion of Evaluation
3.3 Participation of Civil Society and Demand for Evaluation
4 Professionalisation of Evaluation
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
4.4 Existence and Compliance with Standards and Quality Obligations
5 Conclusion and Future Outlook
References
3 Evaluation in Bangladesh
1 General Country Overview
2 Institutional Structures and Processes (Political System)
2.1 Evaluation Regulations
2.2 Parliamentarian and National Audit Structures
2.3 Organisational Structure
2.4 Evaluation Practice and Use
Evaluation Practice and Use at the Government Level
Evaluation Practice at the Sector/Industry Level
3 Societal Dissemination/Acceptance (Social System)
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Discourse on Evaluation and Evaluation Results
3.3 Participation of Civil Society in Evaluations
3.4 Demand for Evaluation Within Civil Society
4 Professionalisation of Evaluation (System of Professions)
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
5 Conclusions
References
4 Evaluation in China
1 General Country Overview
2 Institutional Structures and Processes (Political System)
2.1 Evaluation Regulations
2.2 Parliamentarian and National Audit Structures
2.3 Organisational Structure
2.4 Evaluation Practice and Use
3 Societal Dissemination/Acceptance (Social System)
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Discourse on Evaluation and Evaluation Results
3.3 Participation of Civil Society in Evaluations
4 Professionalisation of Evaluation (System of Professions)
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
4.4 Compliance with Quality Standards and Obligations
5 Conclusions
References
5 Evaluation in India
1 Introduction
1.1 Governance
1.2 Monitoring and Evaluation in India—An Overview
1.3 Development Planning
2 Institutional Structures and Processes
2.1 Constitution of India
2.2 National Laws and Administrative Orders
2.3 Comptroller and Auditor General (C&AG) of India (Supreme Audit Authority of India)
2.4 National Evaluation Policy
2.5 Parliament and Evaluation
2.6 Evaluation Structures in Government
2.7 Evaluation Practice
2.8 Evaluation Use
3 Societal Dissemination/Acceptance (Social System)
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Public Discourse
3.3 Participation of Civil Society in Evaluation
3.4 Demand for Evaluations by Civil Society Organisations
4 System of Professions (Professionalisation of Evaluation)
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
4.4 Existence and Compliance with Standards and Quality Obligations
5 Conclusions
References
6 Evaluation in Japan
1 General Country Overview
2 Institutional Structures and Processes (Political System)
2.1 Evaluation Regulations
2.2 Organisational Structure
2.3 Parliamentary and National Audit Structures
2.4 Evaluation Practice and Use
3 Societal Dissemination/Acceptance (Social System)
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Discourses on Evaluation and Evaluation Results
3.3 Participation of Civil Society in Evaluations
3.4 Demand for Evaluation Within Civil Society
4 Professionalisation of Evaluation (System of Professions)
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
4.4 Compliance with Quality Standards and Obligations
5 Conclusions
References
7 Evaluation in Korea
1 General Country Overview
2 Institutional Structures and Processes (Political System)
2.1 Evaluation Regulations
2.2 Parliamentarian and National Audit Structures
2.3 Organisational Structure
2.4 Evaluation Practice and Use
3 Societal Dissemination/Acceptance (Social System)
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Discourse on Evaluation and Evaluation Results
3.3 Participation of Civil Society in Evaluations
3.4 Demand for Evaluation Within Civil Society
4 Professionalisation of Evaluation (System of Professions)
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
4.4 Compliance with Quality Standards and Obligations
5 Conclusions
References
8 Evaluation in Nepal
1 General Country Overview
1.1 Governance System
2 Institutional Structures and Processes (Political System)
2.1 Evolution of Evaluation in Nepal
2.2 Evaluation Regulations
2.3 Parliamentarian and National Audit Structures
Parliament Mechanisms
National Parliamentarian’s Forum for Development Evaluation Policy (NPFDEP)
2.4 Organisational Structure
2.5 Evaluation Practice and Use
Evaluation Practice
The Evaluation Practice at Decentralised Level
Internal vs. External Evaluation Practices
2.6 Use of Evaluations
3 Societal Dissemination/Acceptance (Social System)
3.1 Institutionalised Use of Evaluations by Civil Society
Overview of Civil Societies in Nepal
3.2 Evaluation Use in CSOs
3.3 Evaluation Database/repository and More
3.4 Public Perception and Discourse on Evaluation and Evaluation Results
3.5 Participation of Civil Society in Evaluations
3.6 Demand for Evaluation Within Civil Society
3.7 Access to Open Data
4 Professionalisation
4.1 Academic Study Courses and Training Practices
4.2 Non-Academic Training
4.3 Journals and Communication Platforms
4.4 Professional Organisations
4.5 Compliance with Quality Standards and Obligations
4.6 Partnership
5 Conclusions
References
9 Evaluation in Pakistan
1 General Country Overview
2 Institutional Structures and Processes (Political System)
2.1 General History of Evaluation in Pakistan
National Policies, Frameworks, and Systems on Evaluation
Evaluation Regulations at Federal Level
Evaluation Regulations at Provincial Level
2.2 Parliamentarian and National Audit Structures
2.3 Organisational Structures
2.4 Evaluation Practice and Use
3 Societal Dissemination/Acceptance (Social Systems)
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Public Discourse
3.3 Participation of Civil Society in Evaluations
3.4 Demand for Evaluations by Civil Society Organisations
4 Systems of Professions (Professionalisation of Evaluation)
4.1 Academic Study Courses and Training Practices
Journals and Communication Platforms
Professional Organisations
Existence and Compliance to Standards and Quality Obligations
5 Conclusion
Appendix A
References
10 Evaluation in the Philippines
1 General Country Overview
2 Institutional Structures and Processes
2.1 Evaluation Regulations
2.2 Parliament and National Audit Structure
2.3 Organisational Structure
2.4 Evaluation Practice and Use
2.5 Use of Evaluations
3 Societal Dissemination/Acceptance (Social System)
3.1 Institutionalised Use of Evaluations by Civil Society
3.2 Public Perception and Public Discourse
3.3 Participation of Civil Society in Evaluations
3.4 Demand for Evaluation by Civil Society Organisations
4 Professionalisation (System of Professionalisation)
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
4.4 Compliance with Standards and Quality Obligations
5 Conclusions
References
11 Evaluation in Sri Lanka
1 General Country Overview
2 The Governance System in the Country
3 Institutional Structures and Processes (Political System)
3.1 Evaluation Regulations
3.2 Parliamentarian and National Audit Structures
3.3 Organisational Structure
3.4 Evaluation Practice and Use
Internal vs. External Evaluation
Use of Evaluations
4 Social System
4.1 Use of Evaluations by Civil Society
4.2 Public Perception and Discourse on Evaluation and Evaluation Results
4.3 Participation of Civil Society in Evaluations
4.4 Demand for Evaluation Within Civil Society
5 Professionalisation of Evaluation
5.1 Academic Study Courses and Training Practices
5.2 Journals and Communication Platforms
5.3 Professional Organisations
5.4 Compliance with Quality Standards and Obligations
6 Conclusion
References
12 Evaluation in Taiwan
1 General Country Overview
2 Institutional Structures and Processes (Political System)
2.1 Evaluation Regulations
2.2 Parliamentarian and National Audit Structures
2.3 Organisational Structure
2.4 Evaluation Practice and Use
3 Societal Dissemination/Acceptance (Social System)
3.1 Use of Evaluations by Civil Society
3.2 Public Perception and Discourse on Evaluation and Evaluation Results
3.3 Participation of Civil Society in Evaluations
3.4 Demand for Evaluation Within Civil Society
4 Professionalisation of Evaluation (System of Professions)
4.1 Academic Study Courses and Training Practices
4.2 Journals and Communication Platforms
4.3 Professional Organisations
4.4 Compliance with Quality Standards and Obligations
5 Conclusions
References
Part III Transnational Organisations
13 The Asian Development Bank and Evaluation in Asia and the Pacific
1 Asian Development Bank Organisational Structure
2 Institutionalisation of Evaluation
2.1 ADB Self-Evaluation
2.2 Independent Evaluation by the Independent Evaluation Department
3 Contributions to the Professionalisation of Evaluation
4 Other Products
5 Conclusions
References
14 UNICEF Support to Institutionalising Evaluation in East Asia and the Pacific Region
1 Introduction
2 Overview of the Organisational Structure
3 Institutionalisation of the Evaluation Function Within UNICEF’s Organisational Structure
3.1 The Evolution of UNICEF’s Evaluation Function
3.2 The Current Structure of Evaluation Function at UNICEF
4 Relationship to National Policies in EAP Countries and Their Evaluation Practice
4.1 Decentralised Evaluation Function
4.2 SDGs and Country-Led Evaluation
4.3 National Evaluation Capacity Development
5 Relationship with Civil Society
6 Contributions to the Professionalisation of Evaluation
6.1 Global Framework
6.2 Regional NECD Initiatives
7 Future Vision for UNICEF to Strengthen Evaluation Function in EAP Region
Appendix
References
15 The Influence of Transnational Organisations: A Comparison of Effects in America, Asia and Europe
1 Introduction
2 Evaluation Networks and the Role of Transnational Organisations
3 Institutionalisation of Evaluation in Transnational Organisations
3.1 Implementation of Monitoring and Evaluation in Organisation
3.2 Monitoring and Evaluation Practice of Transnational Organisations
4 Transnational Organisations Supporting National M+E Activities
4.1 National Monitoring and Evaluation Systems
4.2 Voluntary Organisations for Professional Evaluation
4.3 Evaluation Capacity Development
5 Conclusions
References
Part IV Synthesis
16 The Institutionalisation of Evaluation in Asia Pacific: A Synthesis
1 Introduction
2 The Institutionalisation of Evaluation in the Political System
3 The Institutionalisation of Evaluation in the Social System
3.1 Introduction
3.2 Results
3.3 Institutionalised Use of Evaluation by Civil Society
3.4 Public Perception and Discussion of Evaluation
3.5 Civil Societies’ Demand for Evaluations
4 Summary
5 The Institutionalisation of Evaluation in the System of Professions
5.1 Introduction
5.2 Results
6 Summary
7 Challenges and Driving Forces
7.1 Challenges
7.2 Driving Forces
Evaluation in the Context of Planning Processes
Evaluation in the Context of New Public Management and Programme Planning and Budgeting Systems
Evaluation in the Context of International Cooperation
Evaluation in the Context of the SDGs
Evaluation in the Context of Organisational Development
References
Correction to: Evaluation in Australia
Correction to: Chapter 2 in: R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_2

Citation preview

The Institutionalisation of Evaluation in Asia-Pacific Edited by Reinhard Stockmann Wolfgang Meyer · Niklas Zierke

The Institutionalisation of Evaluation in Asia-Pacific

Reinhard Stockmann · Wolfgang Meyer · Niklas Zierke Editors

The Institutionalisation of Evaluation in Asia-Pacific

Editors Reinhard Stockmann Center for Evaluation Saarland University Saarbrücken, Germany

Wolfgang Meyer Saarland University Saarbrücken, Germany

Niklas Zierke Center for Evaluation Saarland University Saarbrücken, Germany

ISBN 978-3-031-36917-9 ISBN 978-3-031-36918-6 (eBook) https://doi.org/10.1007/978-3-031-36918-6 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023, corrected publication 2024 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Cover credit: Ashway/Alamy Stock Photo This Palgrave Macmillan imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface and acknowledgements

The series of publications on the institutionalisation of evaluation from a global point of view is making good progress. Following the publication of the findings from sixteen European countries in English1 and German,2 and eleven countries in North, Central and South America,3 the findings from eleven countries in Asia-Pacific are now presented here. In all these volumes, moreover, articles by transnational organisations are published, showing how the latter have contributed systematically to the building of evaluation capacities and the establishment of organisational and national evaluation structures. In this volume, the Asian Development Bank and UNICEF present their activities in the Asian and Pacific region. In fact, an article by the UNDP was also planned, since it too has been very active in evaluation capacity building in the region for many years, but sadly it was not able to be realised. The chapter on transnational organisations includes a comparative article, which analyses the role they play in the institutionalisation of evaluation from a global point of

1 Stockmann, R., Meyer, W., & Taube, L. (2020). The Institutionalisation of Evaluation

in Europe. Cham/Switzerland: Springer Nature Switzerland/Palgrave Macmillan. 2 Stockmann, R., Meyer, W. (2021). Die Institutionalisierung der Evaluation in Europa. Münster: Waxmann Verlag. 3 Stockmann, R., Meyer, W., & Szentmarjay, L. (2022). The Institutionalisation of Evaluation in the Americas. Cham/Switzerland: Springer Nature Switzerland/Palgrave Macmillan.

v

vi

PREFACE AND ACKNOWLEDGEMENTS

view. As always, we conclude our Globe volumes with a synthesis of the findings reported from the individual countries. Even if a global synthesis which compares the individual topics and countries with one another will not be possible until the last volume in this series appears—on the institutionalisation of evaluation in Africa— some central findings are already beginning to take shape now, and some of them are surprising. Perhaps the most striking of these is that the institutionalisation of evaluation is policy-driven in all the countries and throughout all the regions, levels of development and political, social and cultural contexts. The political system determines the demand for evaluation, the evaluation topics, and the degree to which institutionalised evaluation is developed and used. The finding that—in all the countries investigated in all four continents—evaluation does not play an important role in a country’s social system—i.e. particularly in civil society—is no less significant. Civil-society organisations hardly act anywhere as promoters of evaluation, and they hardly use evaluation internally at all (except in the case of development cooperation). And there is a third finding that appears to have been confirmed worldwide: the professionalisation of evaluation has indeed reached a remarkable level in some countries and regions, but astonishingly that development has come about regardless of the demand for it from the political and social system. It remains to be seen whether these findings can be confirmed in the countries of Africa. Work on the Africa Globe is already in progress, and it is due out in 2024. To succeed, such a mammoth project requires the support of many. As regards the production of this book, above all, we are much indebted to Julia Schu and Angelika Nentwig. With this volume, we expand the global analysis of the degree of institutionalisation of evaluation to include the countries of Asia-Pacific. What is more, the comparative global perspective adopted here makes a contribution to identifying the causes of common and differing developments and identifying the factors that encourage and impede the institutionalisation of evaluation in politics, society and science. Bürstadt and Saarbrücken June 2023

Reinhard Stockmann

Contents

Part I Introduction 1

The Institutionalisation of Evaluation: Theoretical Background, Analytical Framework and Methodology Niklas Zierke, Reinhard Stockmann, and Wolfgang Meyer

3

Part II National Developments 2

Evaluation in Australia Brad Astbury and Scott Bayley

65

3

Evaluation in Bangladesh Md Faisal Ibne Wali, Sharif N. As-Saber, Muhammad Wali Ullah, Muhammad Maruf Ibne Wali, and Nabila Kamal

95

4

Evaluation in China Wenxuan Yu

125

5

Evaluation in India Rashmi Agrawal, Banda Rao, and Rajib Nandi

145

6

Evaluation in Japan Keiko Nishino, Yoko Ishida, and Kiyoshi Yamaya

195

7

Evaluation in Korea Nowook Park

219

vii

viii

CONTENTS

8

Evaluation in Nepal Asela Kalugampitiya and Gana Pati Ojha

235

9

Evaluation in Pakistan Amna Aaqil, Amna Mahmood, Anam Shoaib, and Sehr Jamil

273

10

Evaluation in the Philippines Asela Kalugampitiya, Ana Erika Lareza, and Romeo Santos

323

11

Evaluation in Sri Lanka Asela Kalugampitiya, Soma de Silva, and Chamara Senaratna

353

12

Evaluation in Taiwan Chi-Yeh Yung, Jan Fell, and Jiann-Jong Guo

387

Part III 13

14

15

The Asian Development Bank and Evaluation in Asia and the Pacific Maya Vijayaraghavan

419

UNICEF Support to Institutionalising Evaluation in East Asia and the Pacific Region Koorosh Raffii and Xin Xin Yang

435

The Influence of Transnational Organisations: A Comparison of Effects in America, Asia and Europe Wolfgang Meyer

465

Part IV 16

Transnational Organisations

Synthesis

The Institutionalisation of Evaluation in Asia Pacific: A Synthesis Reinhard Stockmann, Wolfgang Meyer, and Niklas Zierke

Correction to: Evaluation in Australia Brad Astbury and Scott Bayley

503 C1

Abbreviations

ACNC ACSPRI ADB ADP ADP AES AEW AGP AIC AIHW ANAO APA APEA APEA APS ARC ARFES BAI BCAS BCURE BES BGMEA BIGM BISP BKMEA BRAC

Australian Charities and Not-For-Profits Commission Australian Consortium for Social and Political Research Asian Development Bank Annual Development Programme Annual Development Plan Australian Evaluation Society Asian Evaluation Week Auditor General Pakistan Australian Institute of Criminology Australian Institute of Health and Welfare Australian National Audit Office Annual Performance Agreement Asia Pacific Evaluation Association Asian Pacific Evaluation Association Australian Public Service Administrative Reform Council (Gyosei Kaikaku Kaigi) ASEAN Regional Framework on Evaluation Standards Board of Audit and Inspection Bangladesh Centre for Advanced Studies Building Capacity to Use Research Evidence Bangladesh Evaluation Society Bangladesh Garment Manufacturers and Exporters Association Bangladesh Institute of Governance and Management Benazir Income Support Programme Bangladesh Knitwear Manufacturers and Exporters Association Bangladesh Rural Advancement Committee ix

x

ABBREVIATIONS

BRTC C&AG CAO CBGA CBO CBS CCP CCW CDE CDPR CEMS CEP CEPD CERP CIPP CLE CMA CO COA COE SA COE-Nepal CPD CPE CPE CSI CSO CSR CUSA CW CY DAC DBM DCC DEAR DENR DFABM DHM DMEO DMF DPMM EAD

Bangladesh Road Transport Corporation Comptroller and Auditor General of India Cabinet Office Centre for Budgeting and Accountability Community-Based Organisation Center for Business and Society at LUMS Chinese Community Party Citizen Congress Watch (公民監督國會聯盟) Center for Drug Evaluation (財團法人醫藥品查驗中心) Consortium for Development Policy Research Concurrent Evaluation and Monitoring Scheme Costed Evaluation Plan Council for Economic Planning and Development (經濟建設委 員會) Centre for Economic Research in Pakistan Context, Inputs, Process and Product Country-led Evaluation Chinese Management Association (社團法人中華民國管理科學 學會) Country Office/Country Offices Commission on Audit Community of Evaluators South Asia Community of Evaluators, Nepal Country Programme Document Country Programme Evaluation Centre for Program Evaluation Centre for Social Impact Civil Society Organisation Corporate Social Responsibility Council for United States Aid (美援運用委員會) Covenants Watch (人權公約施行監督聯盟) Control Yuan (監察院) Development Assistance Committee Department of Budget and Management District Coordinating Committee Department of Evaluation and Applied Research Department of Environment and Natural Resources Department of Foreign Aid and Budget Monitoring Department of Hydrology and Meteorology Development Monitoring and Evaluation Office Design and Monitoring Framework Department of Project Management and Monitoring Economic Affairs Division

ABBREVIATIONS

EAP EAPRO ECARO ECG ECNEC ECOI EIS EISI EJA ELD EO EPB ePMS EPoD ESARO EvalAgenda EvalNet EY FAO FDI FMIP FOI FUR GDP GED GEROS GNI GOGP GPEA GPEC GPFE GPRA HARTI HEEACT HEC HEI HMGN HPNSDP IAM ICAR ICCCAD

xi

East Asia and the Pacific East Asia and the Pacific Regional Office Europe and Central Asia Office Evaluation Cooperation Group Executive Committee of National Economic Council Evaluation Community of India Environmental Impact Statement Evidence Information Systems Integration Evaluation Journal of Australasia Empowering Learning Development Evaluation Office Economic Planning Board electronic Project Monitoring System Evidence for Policy Design Eastern and Southern Africa Regional Office Global Evaluation Agenda 2016–2020 Evaluation Network established and managed by NPC Executive Yuan (行政院) Food and Agriculture Organisation Foreign Direct Investment Financial Management Improvement Programme Freedom of Information Follow-Up and Review Gross Domestic Product General Economic Division Global Evaluation Reports Oversight System Gross National Income Government of Gandaki Province Government Performance Evaluation Act Government Performance Evaluation Committee Global Parliamentarians Forum for Evaluation Government Performance and Results Act of 1993 Hector Kobbekaduwa Agrarian Research and Training Institute Higher Education Evaluation and Accreditation Council of Taiwan (財團法人高等教育評鑑中心基金會) Higher Education Commission Higher Education Institution His Majesty’s Government of Nepal Health, Population and Nutrition Sector Development Programme Institute of Administrative Management Indian Council of Agricultural Research International Centre for Climate Change and Development

xii

ABBREVIATIONS

ICDDR IDEAS IDR IED IEET IEO IIED ILO IMED IMEP InDec INDIS INGO IOCE IOs IPA IPDET IPPG IPPME IRIPE IRMA ISST JCPAA JES JICA J-PAL JRF JY KI KIPF LACRO LFA LGUs LSG LSGA LUMS LY M&E MCES MDAC MDGs MEAL

International Centre for Diarrhoeal Disease Research Institute of Development and Economic Alternatives International Development Research Council Independent Evaluation Department Institute for Engineering Education Taiwan (中華工程教育協會) Independent Evaluation Office, UNDP International Institute for Environment and Development International Labour Organisation Implementation Monitoring and Evaluation Division Integrated Monitoring and Evaluation Plans Indonesian Development Evaluation Community Integrated National Development Information System International Non-Governmental Organisation International Organisation for Cooperation in Evaluation International Organizations Institute of Public Administration International Programme for Development Evaluation Training Institute for Public Policy and Governance International Perspective Participatory Monitoring and Evaluation Inter-Regional Initiative for Professionalisation of Evaluation Institute of Rural Management Anand Institute of Social Studies Trust Joint Committee of Public Accounts and Audit Japan Evaluation Society Japan International Cooperation Agency Abdul Latif Jameel Poverty Action Lab Judicial Reform Foundation (財團法人民間司法改革基金會) Judicial Yuan (司法院) Key Informant Korea Institute of Public Finance Latin America and Caribbean Regional Office Logical Framework Analysis Local Government Units Leading Small Group Local Self-Governance Act Lahore University of Management Sciences Legislative Yuan (立法院) Monitoring and Evaluation Multi-Country Evaluation Specialists Ministerial Development Action Committee Millennium Development Goals (UN) Monitoring, Evaluation, Accountability and Learning

ABBREVIATIONS

MEL MEN MENARO MES MfDR MfR MGNREGA MIC MIC MIS MOE MoEF MoF MOFA MoPDR MOU MPA MPI MRP NAARM NAB NABO NAO NAP NASC NCCAP NCDHR NCF NDAC NDC NEAS NEC NEC NECD NEDA NEP NEPAN NEPF NEPS NES NES NFA NFP

Monitoring Evaluation and Learning Mongolia Evaluation Network Middle East and North Africa Regional Office Malaysian Evaluation Society Managing for Development Results Managing for Results Mahatma Gandhi-National Rural Employment Guarantee Act Middle-Income Countries Ministry of Internal Affairs and Communications Management Information System Ministry of Education (教育部) Ministry of Economy and Finance Ministry of Finance Ministry of Foreign Affairs Ministry of Planning, Development and Reform Memorandum of Understanding Master of Public Administration Ministry of Plan Implementation Management Response Plans National Academy for Agricultural Research Management National Accountability Bureau National Assembly Budget Office National Audit Office National Action Plan Nepal Administrative Staff College Philippine National Climate Change Action Plan National Campaign on Dalit Human Rights National Curriculum Framework National Development Action Committee National Development Council (國家發展委員會) National Education Assessment System National Economic Council National Evaluation Capacity National Evaluation Capacity Development National Economic and Development Authority National Evaluation Policy Nepal Participatory Action Network National Evaluation Policy Framework National Evaluation Policies and Systems National Evaluation System Nepal Evaluation Society National Finance Act Not-for-profit

xiii

xiv

ABBREVIATIONS

NGO NIAA NILERD NITI Aayog NITI NLC NMES NPC NPC NPFDEP NPM NPO NPR NPS NRSC NSW OAG OCAG ODA OECD OED OJT OoR OPMCM P&D PAC PaCT PASS PBS PC PCM PCP PDCA PDP RM PDP PEA PEN PEO PERI PGPA PHILDEV

Non-Governmental Organisation National Indigenous Australians Agency National Institute of Labour Economics Research and Development National Institution for Transforming India National Institute for Transforming India Nepal Law Commission National Monitoring and Evaluation Systems National Planning Commission National People’s Congress National Parliamentarians Forum for Development Evaluation Policy New Public Management Non-profit Organisation Nepalese Rupee National Pension Scheme Nepal Red Cross Society National Social Watch Office of Attorney General Office of the Comptroller and Auditor General Official Development Assistance Organisation for Economic Co-operation and Development Operations Evaluation Department On-the-Job Training Office of Research-Innocenti Office of the Prime Minister and Council of Ministers Planning and Development Public Accounts Committee Partnership for Cleaner Textile (PaCT) programme Poverty Alleviation and Social Safety Division Pakistan Bureau of Statistics Planning Commission Project Cycle Management Pakistan’s Center for Philanthropy Plan-Do-Check-Act Philippine Development Plan Result Matrices Philippine Development Plan Pakistan Evaluation Association Pakistan Evaluation Network Programme Evaluation Organisation Punjab Economic Research Institute Public Governance, Performance and Accountability Act Philippine Evaluators for Development

ABBREVIATIONS

PIA PIDE PIL PIP PJERE PM PMDU PMES PMES PMIC PMO PMTA PPAF PPAS PPMI PPRA PRC PRIA PRILIT PSA PSC PSC PSDP QCA R&D RBM&E RBNEP RCT REA REF RMG RO ROC ROSA RSPN RTH RTI SAARC SAI SAJEP SCAAP SDGs

PRIA International Academy Pakistan Institute of Development Economics Public Interest Litigation Public Investment Programme Pakistan Journal of Educational Research and Evaluation Prime Minister Prime Minister’s Performance Delivery Unit Pilipinas Monitoring and Evaluation Society Project Monitoring and Evaluation Systems Prime Minister’s Inspection Commission Prime Minister’s Office People’s Mid-Term Appraisal Pakistan Poverty Alleviation Fund Policy and Programme Assessment System Pakistan Planning & Management Institute Public Procurement Regulatory Authority’s People’s Republic of China (中華人民共和國) Participatory Research in Asia Policy Research Institute for Land, Infrastructure, Transport, and Tourism (Kokudo Koutu Seisaku Kenkyusho) Philippine Statistics Authority Parliament Select Committee on Evaluation Parliamentary Standing Committee (Sri Lanka) Public Sector Development Programmes Qualitative Comparative Analysis Research and Development Results-Based Monitoring and Evaluation Result-Based National Evaluation Policy Randomised Control Trials Regional Evaluation Advisor Regional Evaluation Function (of UNICEF EAP Region) Readymade Garments Regional Office/Regional Offices Republic of China (中華民國) South Asia Regional Office Rural Support Programmes Network Right to Hearing Right to Information South Asian Association for Regional Cooperation Supreme Audit Institution of India South Asian Journal of Evaluation in Practice Special Commonwealth Assistance for Africa Sustainable Development Goals

xv

xvi

ABBREVIATIONS

SHIPDET SIG SIMNA SLEvA SLPFE SOME SPI SUCCESS SWC TA TCGA TERU TMAC ToC ToR TRS TWAEA UGC UN UNDP UNEDAP UNEG UNICEF UNICEF UNSDCF USAID VNRs VOPE WASH WCARO WNTA YEE

Shanghai International Programme for Development Evaluation Training Special Interest Group Social Impact Measurement Network Australia Sri Lanka Evaluation Association Sri Lanka Parliamentarians Forum for Evaluation Society of Monitoring and Evaluation Social Progress Index Sindh Union Council and Community Economic Strengthening Support Social Welfare Council Technical Assistance Taiwan Corporate Governance Association (中華公司治理協會) Tertiary Education Research Unit Taiwan Medical Accreditation Council (臺灣醫學院評鑑委員會) Theory of Change Terms of Reference Target Responsibility System Taiwan Assessment and Evaluation Association (社團法人台灣評 鑑協會) University Grant Commission United Nations United Nations Development Programme UN Evaluation Group in Asia and Pacific United Nations Evaluation Group United Nations Children’s Fund United Nations International Children’s Emergency Fund United Nations Sustainable Development Cooperation Framework United States Agency for International Development Voluntary National Reports Voluntary Organisation for Professional Evaluation Water, Sanitation and Hygiene West and Central Africa Regional Office Wada Na Todo Abhiyan Young and Emerging Evaluators

PART I

Introduction

CHAPTER 1

The Institutionalisation of Evaluation: Theoretical Background, Analytical Framework and Methodology Niklas Zierke, Reinhard Stockmann, and Wolfgang Meyer

1

Introduction

Evaluation—understood as the “systematic investigation of the merit or worth of an object (program) for the purpose of reducing uncertainty in decision making” (Mertens, 1998; cf. Appendix 1)—has gained widespread acceptance in various domains of social and political life and is now being adopted and used by countries and organisations all over the world, making it a truly global phenomenon. Its worldwide

N. Zierke (B) · R. Stockmann Center for Evaluation (CEval), Saarland University, Saarbrücken, Germany e-mail: [email protected]; [email protected] R. Stockmann e-mail: [email protected] W. Meyer Working Group Evaluation, Saarland University, Saarbrücken, Germany e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_1

3

4

N. ZIERKE ET AL.

spread has been gaining momentum in recent decades, in part due to intensified globalisation, greater global interconnectedness and technical progress such as digitalisation and digital transformation. For governments and organisations, it has become easier to share ideas and engage in dialogue about best evaluation practices, resulting in manifold knowledge transfers. The global diffusion of evaluation has also contributed to increased knowledge-sharing and collaboration among governments and organisations to drive further progress. This is accompanied by a global evaluation-specific knowledge generation, in which universities, other higher education institutions as well as non-academic consultancies play an important role. In addition to an established academic tradition that now spans more than half a century, evaluation systems have meanwhile become established worldwide in a variety of ways at the nation-state and organisational levels. The instrument promises its users, in particular, to make an evidence-informed decision—that is, to ground their actions in reliable scientific data. Although challenges in the development of evaluation can be identified time and again—which is why its history is often described as “wave-like” rather than a linear success story (cf. Vedung, 2010; Wollmann, 2017)—it has nevertheless become an indispensable component of social action in various world regions and different social spheres today. This is not surprising, as evaluations have multiple positive benefits. According to Stockmann (2013, p. 74), evaluations are useful for various purposes. First, at the level of project and programme management, lessons can be learned about implemented measures and thus project or programme planning can be optimised. Furthermore, policy evaluations can improve various aspects of Good Governance, for example, by informing about coherence, relevance, impacts and sustainability of policies, and by enabling transparency and participation. As a result, political governance can be improved and the legitimacy and credibility of policies can be strengthened. Ultimately, if the results are taken up in public discourse, evaluation can contribute to the enlightenment of society and facilitate a dialogue about socio-political processes (cf. Stockmann, 2022, 2024). The expansion of evaluation also led to increased attention in academia and the parallel emergence of the research field of “research on evaluation”. Building on this research tradition, the idea of the Evaluation Globe research project emerged with the aim of producing an international and intra- and interregional systematic comparative study of the institutionalisation of evaluation at the nation-state level across countries

1

INTRODUCTION

5

in all world regions. Primarily, the project pursues a descriptive concern, as it aims to take stock of the current state of institutionalisation of evaluation in three different societal systems—namely, the political, the social and the professional system—in a comparative perspective. Another topic to be discussed is identifying and reflecting on the key drivers of development. The results of two volumes, on Europe (Stockmann et al., 2020) and the Americas (Stockmann et al., 2022), have already been published; those on Asia-Pacific are presented here, and those on Africa will follow. The present volume brings together 37 authors who have prepared case studies on eleven countries in the Asia-Pacific region and three regionally significant international organisations. The introduction first briefly discusses the general theoretical framework, which has already been presented in detail in the European volume (Meyer et al., 2020), and then focuses on some methodological and epistemological aspects of the research project, in particular, comparative evaluation research and the comparative approach taken in the Evaluation Globe. Some regional specificities of the Asia-Pacific region will also be highlighted, as was the case in the Americas volume. The following section outlines the current state of research and the theoretical underpinnings of the project. The analytical framework and its main categories are then introduced, followed by methodological considerations, such as the selection of countries or the implemented review and editing process. After the introduction, the eleven country case studies and three chapters on international organisations are presented. These are then discussed in a final synthesis in light of the research question, including an interregional comparative section. Finally, a comparative chapter on the influence of international and transnational organisations in the institutionalisation of evaluation in the Asia-Pacific region is included.

2

State of Research and Theoretical Background 2.1

State of Comparative Research on the Institutionalisation of Evaluation

A first standard reference in international comparative research on the institutionalisation of evaluation is the “International Atlas of Evaluation” published in 2002 by Jan-Eric Furubo, Ray C. Rist and Rolf Sandahl,

6

N. ZIERKE ET AL.

comparing 21 countries and three transnational organisations. In the case studies, country experts are asked to provide information on nine key indicators. Unlike the Latin American countries, which are not considered, five countries of the Asia-Pacific region are included: Australia, China, Japan, New Zealand and South Korea. However, the country focus is clearly on Europe and other countries in the Global North, as also can be seen in the country selection for Asia-Pacific. As a result, the study indicates New Public Management as one of the main drivers of the developments. Overall, the authors distinguish between external and internal determinants. External factors include membership in the EU and internal factors entail political constellations, the financial situation and the constitutional framework (Furubo et al., 2002). Steve Jacob, Sandra Speer, and Jan-Eric Furubo updated these findings a decade later. The new version shows that overall, there has been an improvement in mean total scores of all nine indicators. The authors interpret this as a maturation of the evaluation discipline. In a large number of countries, the institutionalisation of evaluation has continued with most significant positive changes in Finland, Japan, Spain, and Switzerland. The study cannot identify processes of de-institutionalisation: “This relative stability can be explained by the fact that once implemented, evaluation is vigorously rooted in the political and administrative environment” (p. 25). However, according to the authors, a change in data collection methods compared to the original study probably also led to measurement artifacts, through which a decline in institutionalisation was observed in a few countries (Jacob et al., 2015). A methodology that is consistent across the volumes is therefore an important concern of the present research project. Finally, unlike in the Evaluation Globe project, the term “evaluation” is not defined in these two landmark studies, but left to a wide range of interpretation, posing a challenge for systematic comparison. Moreover, they do not follow a systemic perspective as the Evaluation Globe does. Barbara Rosenstein (2013, 2015) presented a comprehensive overview of the situation of national evaluation policies, looking at a high case number of countries (115 and 109 countries respectively were studied), achieving a profound and extensive global mapping. The most recent update of the report looked at 113 countries (Rosenstein & Kalugampitiya, 2021). The study also covers a large number (16) of countries in the Asia-Pacific region: New Zealand, Sri Lanka, India, Indonesia, Japan, South Korea, Malaysia, Mongolia, Nepal, The Philippines, Bhutan,

1

INTRODUCTION

7

Vietnam, Bangladesh, Singapore, Australia, and Pakistan are included in the analysis. On a cross-regional basis, the authors indicate that 35 countries have a National Evaluation Policy (NEP) (of which the legislative process was still ongoing in 10 countries at the time of the study) and 21 countries have regular evaluation practices even without having a NEP. Of the 16 countries examined in the Asia-Pacific region, 9 countries have a NEP and one is in the legislative process. Methodologically, the study was based on a desk study that took into account extensive material, which is an optimal starting point for research with other designs and methods such as in the Evaluation Globe project. For example, an UNDP study already extended the findings for 10 countries in the region by combining a desk study, literature analyses, and an online survey (UNDP, 2015). In another international study, Stockmann and Meyer (2016) looked at the professionalisation of evaluation worldwide, compiling a differentiated body of work with a decidedly international perspective that also included teams of authors from the Asia-Pacific region. Nevertheless, this approach lacked a systematic research approach that would allow for a detailed examination of the institutionalisation of evaluation. Apart from these international comparative studies, the field of research that applies a region-specific focus has been growing in recent years. This can be said with respect to various regions of the world, including the Asia-Pacific region, although compared to research focusing on American, European, or African countries, the canon of literature is comparatively small. However, the research field is relatively dynamic, and recently important intra-regional comparative research on the institutionalisation of evaluation in the Asia-Pacific has been published as well: In his dissertation project, Asela Kalugampititya (2021) provides an innovative intra-regional comparative study of four countries in the region: Sri Lanka, Nepal, the Philippines, and Bangladesh. The work succeeds in identifying important “critical factors” in the institutionalisation of evaluation “at the national level” (ibid.). It builds on the systematic analytical framework of the Evaluation Globe project and successfully applies it to empirical cases. A particular merit of the study is the level of detail with which it engages in case analysis, providing substantial insights into institutionalisation in countries that have received little attention in previous research. Another recently published study provides the first regional mapping of institutionalisation of evaluation in the region. In their report on the “Status of National Evaluation Policies and Systems in Asia Pacific”

8

N. ZIERKE ET AL.

Diwakar and colleagues (2021) undertake a comparative review, including 14 countries, namely Afghanistan, Bangladesh, Bhutan, India, Indonesia, Japan, Mongolia, Myanmar, Nepal, New Zealand, Pakistan, the Philippines, South Korea, and Sri Lanka. Thus, the study achieves a relatively high degree of coverage; in particular, the Southeast Asia region is also well covered, although important countries such as Australia and China are missing. One of the key dimensions of the study is the “enabling conditions for evaluation” this includes the entrenchment of evaluation in the constitution (in 1 of 14 countries), the recommendation of a NEP (2 of 14 countries), the adoption of evaluation legislation (2 of 14 countries), the existence of a parliamentary forum on evaluation (3 of 14 countries). In addition, the report examines “institutional” and “individual” capacities for evaluation. While the former concerns aspects of the organisational structures of evaluation, the latter encompasses the issue of the skills and competencies needed for evaluations. Taking stock of this, the report presents various recommendations for action. As in the Evaluation Globe project, the report’s methodology involves establishing a conceptual framework and definitions in advance and examining various dimensions of evaluation systems and their institutionalisation. The main difference to the approach followed here is the theoretical grounding of the analytical model as well as a consideration of different systems in which evaluation is institutionalised—both are essential for the Evaluation Globe project. While the APEA paper focuses on evaluation in the national political systems, the Globe approach considers multiple dimensions and theoretically derived categories, also emphasising the role of civil society (see Sect. 3; for the advantage of this, see Meyer et al., 2022, p. 7). Finally, there is also a methodological difference in that the Globe approach takes greater account of qualitative expert assessments, whereas the report by Diwakar and colleagues uses a survey with a supplementary desk study to directly generate categorical data.1 Apart from these more recent studies, a large number of relevant early case studies and discussion papers on the topic can be found (Ahmed &

1 In 2022 Diwakar and colleagues updated their report with a new focus and country

selection (Diwakar et al., 2022). This time they included Afghanistan, Australia, Azerbaijan, Bangladesh, Bhutan, Cambodia, Fiji, India, Indonesia, Japan, Nepal, Pakistan, Philippines, Sri Lanka, and Tajikistan (n = 15) and the research focused on four dimensions, namely the ‘enabling environment’, ‘institutional capacities’, ‘standardising capacities’ and ‘VOPE capacities’.

1

INTRODUCTION

9

Bamberger, 1989; Bamberger, 1989, 1991; Hay, 2010, 2014; Khan, 1989, 1990, 1998; Khan et al., 1993; Malik & Roth, 1999; Mehrotra, 2013; Roy & Wijayasuriya, 1992), focusing on specific perspectives of institutionalisation and evaluation capacity building in the Asia-Pacific region, subregions or selected countries. In addition, there is a steadily growing field of case studies on various Asian Pacific countries published in international peer-reviewed journals (e.g., Dhakal, 2014 for Nepal). All of these studies differ from the present research in either the “interand intraregional comparative approach”, the “multidimensional systems perspective” or the “systematic as well as theoretically grounded empirical research approach”; but the systematic comparative studies tie in with these earlier contributions that started to open up and grasp the research field. Without the merits of the preceding contributions for about 30 years—to which organisations such as the World Bank, different specialised UN agencies like UNDP or UNICEF and many others as well as to an increasing extent various regional universities and public institutions have contributed significantly—international as well as intra-regional comparative studies would not be possible, which is why their contribution deserves special recognition. On the one hand, shouldered on the great achievements of the previous studies, the Evaluation Globe with its comparative view can try to see a little further; on the other hand, shoulder to shoulder with the community of evaluation practitioners and researchers on evaluation, it explores the field and by doing so wants to encourage further dialogue. 2.2

Theoretical Background

In sociology and political science, institutions are a research subject with a long tradition, the beginnings of which can be traced back to the paradigmatic works of Émile Durkheim and Max Weber, and whose paths of development—in the course of decades of fruitful debate in various schools of thought—led to their entry into the basic vocabulary of many social science disciplines. Nowadays, the term has long become part of the basic vocabulary of various social science disciplines (see Hall & Taylor, 1996 on three central social science theories of institutions). They traditionally have been viewed as bridges between individuals and society. They thus mediate between the so-called social micro and social macro level (Stachura et al., 2009; Traugott, 2013).

10

N. ZIERKE ET AL.

Institutions play a pronounced role in the structural functionalism of the 1960s, in emphasising their social stabilising function and addressing them in the context of sociology of professions (Brante, 1988; Parsons, 1939). As Talcott Parson stated: “[the] study of the institutional framework within which professional activities are carried on should help considerably to understand the nature and functions of some of these social ‘constants’” (Parsons, 1939, p. 457). In a later publication, he takes another look at these social constants and explains how they are stabilised by social expectations linked to roles—for example, those directed at a particular profession: [T]he essential aspect of social structure lies in a system of patterned expectations defining the proper behavior of persons playing certain roles, enforced both by the incumbents’ own positive motives for conformity and by the sanctions of others. Such systems of patterned expectations, seen in the perspective of their place in a total social system and sufficiently thoroughly established in action to be taken for granted as legitimate, are conveniently called ‘institutions’. The fundamental, structurally stable element of social systems then, which, according to the present argument, must play a crucial role in their theoretical analysis, is their structure of institutional patterns defining the roles of their constituent actors. (Parsons, 1954, p. 231)

One of the main criticisms of structural functionalism—its overestimation of the stability of societal systems2 —was addressed by the emerging theory of modernisation. As a macro-theory—that is, concerned with the large, overarching structures of the social world—it seeks to explain social change by means of macro-structural concepts. To this end, it uses social differentiation, a central concept of sociological systems theory. Through horizontal differentiation, functionally specialised societal subsystems such as politics, administration, health or science are formed in a historical process in modernity, which are then brought into relation with institutions defined by the division of labour (Luhmann, 1983, 1988). For societal systems, it is assumed, it is necessary to continuously differentiate and specialise its function due to ecological changes such as climate 2 The term societal system is used here when speaking of various functionally differentiated subsystems such as law, politics, religion, health, etc. It is essential to distinguish this from one of the specific societal subsystems under investigation, which is called “social system” in the study (see Sect. 3.2).

1

INTRODUCTION

11

change or system-immanent changes such as the introduction of a new scientific measurement method in the science system. This puts pressure on subsystems and they often respond by adapting their institutions (e.g., Alexander, 2001). Different subsystems have been analysed by modernisation theory, with a particular focus on the development of its institutions (for instance Giddens, 1990; Inglehart, 1998; Lerner, 1968; Parsons, 1971; Zapf, 1991). The 1990s witnessed a renaissance of modernisation theory (Kollmorgen, 2014, p. 77), after other theoretical paradigms had dominated for some time. The theory of social differentiation (Schimank, 1996), for example, understands social change as a process of social transformation in which the complexity of societal subsystems and the adaptability of their institutions increase. A generalisable component of modernisation theory applied since then can be identified, which consists of examining the interrelationships between central institutions in different functionally differentiated subsystems, with a special focus on those between politics and the economy (Pollack, 2016). In addition, inspired by classical social science considerations, the question of how the development of the economic system affects the general social differentiation of other societal subsystems and how the interactions between these subsystems are is a central study subject (Schimank, 2015, 2018; Schimank & Volkmann, 2012), often subsumed under the term “economisation”. Institutions in science have also been studied in terms of modernisation theory (see Krohn, 2000). For example, Robert K. Merton’s studies on the normative structure of modern science (Merton, 1942), which built on the systems theory of his teacher Talcott Parsons, exemplify a “brilliant elaboration of the modernization-theoretical paradigm” (Krohn, 2000, p. 315) in relation to a specific functional societal subsystem. Given the diversity of theoretical traditions in which institutions are studied, a single definition that takes account of all theoretical specificities is hardly possible, but for general orientation the term can be specified in a way that allows for different theoretical connections. In this sense, and in accordance with the definitions presented in the previous two volumes of the Globe project (Meyer et al., 2020, p. 11), institutions can be defined as a set of more or less formalised rules, norms, and regulations in specific societal subsystems, that influence social action of (corporate) actors within various social situations. In this way they mediate between different analytical levels of society in a rule-based manner (cf. Marche & Olsen, 1989, p. 160; Peters, 2019, p. 158).

12

N. ZIERKE ET AL.

This helps to understand why institutions are important for macro theories as the previously described structural functionalism, mezzo theories like economic organisation theory and institutional economics (e.g., Coase, 1937, 1991; Meyer, 2017; North, 1986, 1987, 1991; Robins, 1987) and micro theories like rational choice or other theories of social action (e.g., Esser, 2002; Kiser & Ostrom, 1982; Kroneberg, 2014, p. 119; Ostrom, 2009, 2011). For example, if looking at the sociological model of explanation presented by James Coleman in his famous work Foundations of Social Theory (Coleman, 1990), which is devoted to methodological individualism, institutions can be integrated in modelling individual as well as collective action as important external factors—as such they can be seen as important parameters in a theory of social action. In this context, Coleman (1986) distinguishes two possible causal mechanisms: first, macro-micro links and second, micro-macro links. In both cases, institutions play a mediating role between the two levels of analysis. In the 1980s, among others March and Olsen (1983) gave rise to the paradigm of “neo-institutionalism” and set the tone for a theory that stood in opposition to the purely economic rational choice and macrosociological grand theories that were prevalent at the time by making institutions the central theorem and object of study. As outlined above, it should later prove valuable in bringing different styles of thinking into dialogue which opened up various new avenues in the study of social and political life. From the beginning, their work has also been particularly important in terms of political institutions. For instance, March and Olsen (1983) emphasise the relative autonomy of political institutions, which provides a frame of reference for social action and share a common concept of governance within the subsystem. In this way, they emphasise that it is not the economy, from a materialistic-epiphenomenal point of view, or the functional imperatives of societal systems, from a classical systems-theoretical point of view, that determine the design of institutions, but that—from a more organisational-theoretical point of view—political and legal organisations have their own logic and can act at least partially autonomously, and in this respect, the system in which they operate can be described as “relatively autonomous”. Here, institutions are relevant when considering the possibilities for shaping social development and asking how organisations can direct their actions within a given societal subsystem. System boundaries regulate exchanges between societal subsystems, that’s also true when looking at governance systems and their relations

1

INTRODUCTION

13

to their environment (Kapitanova, 2013, p. 257). This is implemented by specialised institutions, which ultimately ensure system integration and support transactions between subsystems (Armingeon, 2016). It is a major topic in the study of institutions, for which there is now a rich set of concepts with a long research history (Peters, 2019). Transactions between systems are theorised, for example, through institutions that form around the “heteronomous pole” of a societal subsystem; that is, the area of a system that is structured in such a way that it allows for some degree of “connection for the next communication” (in German: Anschlusskommunikation, cf. Luhmann, 1981, 1992) for other systems. Here, actors such as organisations can align their actions with these institutionalised structures, thus enabling transfer between systems (cf. Schimank, 1988, 2006). Such interdependencies between subsystems—that are often mediated by special institutions—and the concrete pattern of their “structural coupling” (cf. Luhmann, 1991) are key to understanding modern societies, their conflicts, governance processes, and social dynamics (Schimank, 2001). The latter aspect already suggests that, in contrast to Luhmann’s relatively static basic model with its focus on autopoiesis as systemic self-reproduction (cf. Luhmann, 1995), other—often more recent— sociological approaches to systems theory model the interrelationships between systems dynamically (e.g., Burns, 2006). In this context, institutions have another conceptual advantage in helping to explain such social change: they are understood as something processual, something that is followed by path dependencies and routines of social action and, in turn, by its own institutional dynamics that can be studied over time, which ultimately allows to trace negotiation processes of new social contracts between societal subsystems and thus contribute to the understanding of social change as a whole (see e.g., Beckert, 2010a, 2010b on how institutions as well as organisations are key to understanding social dynamics). Based on the above, the following key aspects can be derived for the study of institutionalisation of evaluation (Meyer et al., 2020, p. 12): • “Rules, norms and regulations on evaluation, implemented in the already existing social subsystems. […] • Evaluation processes, procedures and routines, implemented within a broad set of organisations or networks at least as a possible way of practice within a certain policy field. […]

14

N. ZIERKE ET AL.

• Finally, institutionalisation is a process with certain steps and it is probably a long way toward building a complete and deep-seated institutional framework. […]” • Furthermore, institutionalisation helps a system to adapt to environmental change as well as resulting new environmental requirements and to structure the exchange with the environment and to make it reliable in terms of expectations. Evaluation institutions can thus optimise various aspects of a governance system (innovation potential, effectiveness, efficiency, sustainability, participation, etc.) by stimulating new routines of action and thus contributing to incremental change in politics and society.

2.3

Particularities of the Asian Pacific Region

Many substantive turns have occurred in the study of institutions, whether focusing on purely formal aspects or later turning to material practice; it opens up a wealth of research with one of the latest shifts, the spatial turn, concerning spatial or regional approaches to inquiry (Drori & Preminger, 2018), to which the Evaluation Globe would like to contribute. The volume on the Americas already drew attention to regional specifics that are relevant for contextualising the general institutional theory used. In particular, specific institutional frameworks for North and South America were identified and compared with those of Europe (Meyer et al., 2022: 10f.). For the Asia-Pacific region, too, historically determined peculiarities in the institutional framework can be identified. However, the politicalstate context alone should not be considered here, especially since the Evaluation Globe takes into account various societal subsystems. This also addresses the problem that focusing on the political dimension can “marginalize issues and actors that cannot be narrated in state terms” (Reid, 2015: Preface). Therefore, it should be emphasised that the institutional framework in Asia-Pacific countries—like in other world regions—is also shaped by historical and cultural specificities in civil society and academia. Given that evaluation can be seen largely as a phenomenon of “recent contemporary history” (cf. Schwarz, 2003), especially when looking at global diffusion processes, it makes sense to focus on the post-1990 era, without ignoring the impact of longer-term developments that preceded this period.

1

INTRODUCTION

15

First, it is important to emphasise that the Asia-Pacific region consists of several subregions, each of which has experienced its own long-term historical development, but which have also been closely intertwined at the same time. This is reflected in the literary canon in comprehensive historical compendia with a subregional or nation-state perspective (e.g., Bose & Jalal, 2023; Church, 2017; Holcombe, 2017; Peel & Twomey, 2017; Reid, 2015), while overall general anthologies or monographies of “Asia Pacific history” are, viewed comparatively, still rather rare and focus on the modern period (e.g., Borthwick, 1992; Borthwick & Latz, 2022). If considering journal publications, numerous historical contributions can be found, e.g., on economic history or in the context of world polity or world history research (Brook & McGrew, 2013; McDougall, 2016). Based on this body of research and because it fits the general theoretical framework chosen, an assessment based on world society or world politics research is presented below as an example to understand “regional-specific institutional logics”. If looking at recent contemporary history from the perspective of modernisation theory, “different development paths” become apparent in the subregions and their individual countries, as Wolfgang Zapf (1996) puts it. Also, for Asia-Pacific the countries take different pathways to modernity. World society shows a dynamic differentiation resulting in a specific internal structure. While in some countries it develops “in dense international interdependence and under competing influence” (p. 72), others choose alternatives to western modernity or a “mixture” (p. 73) of western modernisation with specific local context prevails. This results in numerous varying institutional frameworks, which are also of great importance for the respective (institutionalised) practices of evaluation. Especially, looking at local cultural specifics (local bureaucratic practices, bodies of knowledge, traditions) can be seen as helpful to contextualise and finally better understand processes of worldwide diffusion of institutions. Thus, institutionalisation can be understood as a process that does not diffuse homogeneously in the world, but that is always also adapted locally—to varying degrees. In this context, Markus Holzinger (2014) speaks of “specific local cultural reception filters” (p. 283) to be considered in studies of world political contexts, criticising a widespread assumption of sociological “world society theory”. But also, representatives of precisely this world society theory (e.g., Stichweh, 1995, 2008)—which sees in the global level a social structure of its own relating to the differentiation of a world society as an own societal system

16

N. ZIERKE ET AL.

which suggests a certain uniformity—see in the internal structure— i.e., among others in the individual nation states—“leeway”, different temporalities, a “diachrony”, and thus a diversity is reflected theoretically (Holzer, 2006; Kirschsieper & Brose, 2014; Stichweh, 2006). The general theoretical tenor that can be derived from this is consequently, that there is no singular path to modernity, but there are multiple routes to choose from and sometimes even completely alternative approaches to modernity are adopted; nevertheless, the development is characterised by global simultaneity of certain structures and constants of socio-political reality. As far as evaluation is concerned, it is striking that in the AsiaPacific region, at least in the last 20 years, it has developed to some extent in sync with other world regions, apart from the pioneer countries.3 This variety despite the synchroneity of certain aspects of societal systems has been termed “simultaneous diversity” (Kirschsieper & Brose, 2014). For researching these theoretical assumptions, studies that compare different world regions suit well.4 Empirical work already underlined special institutionalised logics when it comes to the adaption of general new models of public sector reform (e.g., Tobin, 2003 for the South Korean case) or found out local variations in organisations’ adaption to global diffusing organisational and management routines (Drori et al., 2014). In terms of globally diffusing evaluation, it is possible to see how specific local institutions and cultural contexts shape its concrete reception, adaptation, and possible innovation, as exemplified by the institutionalisation and practice of M¯aori evaluation in New Zealand (Cram, 2015). How exactly this unfolds is an empirical question and in the present study must be taken into account when interpreting the results. There are various activities in the field of development and international cooperation, including bilateral as well as multilateral action programmes or transnational organisations with branches in different countries of the region. In Asia-Pacific, there are also significant specific regional supranational entities, most notably ASEAN and APEC (Beeson, 2008). These organisations cannot be understood as a common “federation of states” with strong political relevance in the form of supranational 3 The same can be said when looking at the world’s leading countries in institutionalising evaluation, but with a different time frame. 4 The general added value of such comparative studies has been pointed out not least by the Comparative Area Studies (CAS) research programme (Ahram, 2011; Basedau & Köllner, 2007; Berg-Schlosser, 2012; Mainwaring & Pérez-Liñán, 2007).

1

INTRODUCTION

17

decision processes. Thus, there is no transnational institution with a comparable dominant role like the EU in Europe (for the central role of the EU for evaluation institutions and practices see Blomeyer & Eussner, 2020; Schoenefeld & Jordan, 2020; Stern, 2009; Stockmann et al., 2020; Van Voorst et al., 2022). The selected transnational institutions—Independent Evaluation Group of the Asian Development Bank, the World Bank, UNICEF as well as UNDP—are important actors in the field of evaluation both in financing development projects as well as initiatives for improving evaluation in the region. This becomes especially visible when looking at the enormous number of capacity building initiatives for many decades. These organisations can be seen as very crucial for institutionalising an evaluation knowledge transfer. One questions that needs to be answered in this context is whether and to which extent these transnational institutions are successful in supporting an evaluation culture in the different subregions of Asia-Pacific. These specific frameworks are likely to have an impact on the institutionalisation of evaluation, which should be considered in this volume. Three aspects seem to be especially important: a. The cohesion policy of the European Union is a central driver in the institutionalisation of evaluation in Europe, because its high project budgets and implemented evaluation obligations. The European volume accordingly underlines how important the impactful transnational institution of the EU is and historically was for the diffusion of evaluation in various states like Spain, Portugal and countries of Central Eastern Europe (Stockmann et al., 2020). Since in Asia-Pacific there is no comparable central transnational institution like the EU it is interesting to find out whether other transnational development organisations play a comparable role in Asia-Pacific. b. The described historical and socio-political variances in the region are an interesting starting point to ask if there is something like “Asia-Pacific specific evaluation cultures” or if institutionalisation is rather a homogenous diffusion process. c. Since the Globe project takes three different societal systems—the political, the social and the professions system—into account, the different institutional frameworks briefly described here might have varying effects on these subsystems.

18

N. ZIERKE ET AL.

3 Institutionalisation of Evaluation---A Systemic Perspective Monitoring and evaluation can be interpreted as an essential aspect of institutions in modern societies. As a governance instrument it is becoming increasingly important for social and political development, e.g., with regard to steering processes. This is true for all stages of the multi-level governance system (supranational, national, regional, transnational) as well as on organisational level for different forms of corporate governance. The significance also extends across many fields of action, i.e., it cannot be limited to a specific subsystem: health, education, economy, administration, science and research, etc. These subjects are researched in governance studies (Bevir, 2012; Pierre & Peters, 2021), among others in the political economy of governance (Barker, 2010; Bertelli, 2012; Ebener, 2008; Payne & Phillips, 2014; Schofield & Caballero, 2015). In the Evaluation Globe, special consideration is given to the national governance level and its institutions, accordingly the first object is to research the role of evaluation in the political subsystem. 3.1

Institutionalisation of Evaluation in the Political System

Especially three aspects are central to the study of institutions of the political system: institutional rules and frameworks, organisations and their structures, and the actual practice, i.e., what of the defined rules is implemented and how. Thus, ideas of what (policy) should be implemented and how (politics) should be implemented must first be adopted and established, which is strongly influenced by legal-institutional frameworks. As became already evident in the general theory section, when studying institutions of the political system, it can by no means be limited to legal structures, but organisations in particular must also be taken into account. The implementation of evaluation, and in particular the application of evaluation rules, always takes place in the context of organisations, which is why, on the one hand, the question of organisational structure and that of concrete organisational forms are relevant in the analysis. Together with the institutional rules and frameworks they are an essential aspect of what is called the polity, the political structures of a political system. Accordingly, it is not only legal frameworks that are significant as an institutional context, but also the question of how organisational scopes of action are structured, which substantially affects policy (the content of

1

INTRODUCTION

19

political measures) and politics (the political processes). Finally, the focus must be on whether and how the rules are applied by the organisations, which is why the evaluation practice and the use of evaluation findings in the political system must be considered. These three sub-dimensions are presented below, as they ultimately guide the derived analytical categories. Institutional Rules An important aspect of modern political systems are legal frameworks, which include laws, acts, decrees, and other kind of regulations to which the various constitutional entities must adhere. The legislative decided on the concrete formulation of the formal framework, which in most countries is implemented by elected parliaments. In order to propose legislation, it needs parliamentary majorities and an exchange with other parties, as well as public communication about it. This allows “to make adjustments in the light of discussion and experience”, to use a formulation of Karl Popper (2011, p. 340). Legal frameworks are usually quite accessible and thus comparatively “easy” to research, because normally their contents like laws or regulations are published. Additionally, information about the legislation process is also often documented like the discussions in the parliament or in specialised parliamentary committees. Through the increasing digitalisation of the public sector, the degree of transparency of such documents and process documentations continues to increase. However, the study at hand faced two challenges in this regard: 1. The term “evaluation” and how it is used varies depending on the specific context (context specificity). For this purpose, the analysis will follow the glossary of basic terms that was provided to the author teams at the beginning of the drafting process in order to reach a common understanding of the terminology and thus allow for better comparability (see Appendix 1). 2. In many cases, regulations are adopted by administrations with shared responsibilities (Lowe & Potter, 2018), which can complicate their study. The types of data examined are limited to legislation at the national level that has some general applicability in the country under study; however, specific norms or strategic policies are not examined unless they are deemed to be of more general significance.

20

N. ZIERKE ET AL.

Practices On the one hand, formalised legal frameworks are a central prerequisite for administrative action, because executive bodies gear their activities to the specifications of the legislature. On the other hand, the legislature is also dependent on the executive, since only through it does the adoption of institutional frameworks and rules make sense. For this to happen, the administration must accept the adopted rules and apply them in its work routines (Goodnow, 2003). Such institutionalised rules and settings are necessary but not sufficient when it comes to the practical relevance of evaluation. To become a routinised “administrative practice” evaluation has to be anchored in regular administrative processes (Kettl, 2018, 2019). Complex governance systems—i.e., such that include multiple actors from the private as well as civil society sector—special variants of public administration are necessary (Mayntz, 2017). For example, public-private partnerships need their own institutional frameworks to enable rational governance, sometimes implementing “customer-oriented” forms of public administration as presented in New Public Management concepts (e.g., Ansell & Torfing, 2016, 2021a, 2021b; Christensen & Lægreid, 2007, 2022; Kettl, 2000, 2015). One of the basic ideas behind that is that principles of the private sector like the service and quality orientation are transferred to the public sector in order to strengthen various aspects of the governance structure, especially its efficiency and effectiveness (Stockmann, 2008, p. 57ff.). In Asia-Pacific, too, this multi-layered structural transformation of the public sector has been studied, resulting in a whole range of different and, often locally specific governance systems and mechanisms (Cheung, 2005, 2016; Cheung & Scott, 2003; Mok & Forrest, 2008; Samaratunge et al., 2008; Tobin, 2003). This complexity of governance processes and structures and their change over time, which is illustrated by the example of the NPM but is also reinforced by other—often locally specific—public sector reforms, is a methodological challenge in their systematic comparative research. In the Evaluation Globe, this is reflected in the fact that evaluation-relevant governance processes and structures are to be covered as comprehensively as possible. Accordingly, case study authors are encouraged to create a descriptive inventory of how evaluation is institutionalised in different governance mechanisms and in different policy fields.

1

INTRODUCTION

21

Organisations and Organisational Structure Typically, social practices such as administrative practice are embedded in an organisational context which requires appropriate theoretical and conceptual consideration. Thus—as emphasised in the general theory section and also as the governance term used in the previous paragraph suggests in the literature—the Globe project, drawing on organisational theory (see e.g., Kieser & Ebers, 2019), also asks what are the concrete organisational forms in which evaluation is embedded and practised (cf. Borrmann & Stockmann, 2009; Stockmann, 2008). Organisational perspectives have been widely applied in the research literature on the role of evaluation in the political system (e.g., Pattyn & Brans, 2015; Speer et al., 2015), which provides good starting points for actor- and group-related studies on organisations or political structures and their modes of organisation (e.g., Bundi, 2018; Varone et al., 2020). The focus in the Globe project is on the one hand on concrete organisational types such as parliaments and audit offices as central places of control and oversight functions in which evaluation can play a systemically important role. On the other hand, the organisational structure as a whole must be taken into account, where the main question is how precisely evaluation is anchored in the political system and its organisations and whether and how evaluation is anchored and used in government agencies (e.g., whether evaluation institutions are set up as line or staff units). For this purpose, three assessments are central to the case studies. First, governance processes and mechanisms will be examined with respect to evaluation. In one case, evaluation is comprehensively regulated in the form of a national evaluation policy (NEP); in another, there may be very different sector-specific regulations or even non-formalised evaluation routines. Second, in addition to institutions, the extent of evaluation practice must also be assessed, with a policy area-specific view being appropriate. Third, organisations and organisational structures must be considered as a major social context for evaluation practices. All in all, an empirical diversity of institutions and practices is possible, be it with regard to the degree of organisational anchoring, the importance of parliament, audit offices or other control bodies, the involvement of governmental as well as non-governmental actors, or the instruments and variants of evaluation applied. This is the central open research question of the individual case studies, which is to be answered by detailed analyses and descriptions of the institutional and practical characteristics of the governance set up.

22

N. ZIERKE ET AL.

In summary, four key aspects can be identified for the study of the political system in the Evaluation Globe research project (see also Table 1, Column 1): • Are there national acts, laws, decrees, rules and all kind of regulations that include evaluation as a task? Besides that, are there national evaluation policies or routines that are regularly implemented in governance processes? (row 1 “National laws, regulations and policies”) • How is evaluation organisationally embedded in parliamentarian or other control structures like audit offices? (row 2 “Parliamentarian and national audit structures”) • What are the evaluation units as core elements of political and administrative organisations? (row 3 “Organisational structures”) • How evaluation results are being used for decision-making or other purposes in the political process? (row 4 “Evaluation practice”).

3.2

Institutionalisation of Evaluation in the Social System

There are various purposes for evaluation use. Only one aspect is to use evaluation results for political decision-making. Besides that, evaluation can be applied in order to “control” project, programmes or processes. Then it follows similar intentions like controlling and accountability instruments (cf. practices of performance accountability e.g., Han, 2019; Shah, 2007). Additionally, the application of evaluation and the use of its results can merely take place in the public administration and thus may be limited to the political system, but also in other social contexts evaluation uptake is possible. Especially in political cultures that are based on a sound participatory governance structure and have a long tradition of citizen involvement, this is plausible. Participation is a central subject in the analysis of political processes (‘politics’) (cf. Lauth et al., 2014, pp. 245 ff.). For example, in North America, participation of citizens within various institutions is an essential aspect of the democratic model (cf. Edwards, 2020, pp. 17ff.; Tocqueville, 2003). Looking at Asia-Pacific, for example in Australia there is also a long tradition of civil society participation, dating back to reforms in the 1960s (Aulich, 2009). Moreover, in Taiwan and South

1

INTRODUCTION

23

Korea, forms of participatory governance emerged in the course of the democratisation processes and since then have been soundly established (He, 2010; Jeong & Oh, 2010; Kassomeh, 2021; Kim, 2011; Lee & Park, 2009; Weng, 2010). All these are important historical origins for the political systems of open societies (Cohen & Arato, 1994). In general, civil society can be seen as a mediating institution between the social and the political system: on the one hand, with its help, sociopolitical interests can be communicated from the social to the political system; on the other hand, responsibilities and decision competencies can be allocated from the political to the social system. Meyer et al. (2022) differentiate two main functions of civil society organisations: First, to control and ensure that state activities are in the interest and for the need of citizens, what the authors call the “watchdog function”, which also includes critical observations by the media. In this context, evaluation results can be used to fulfil this task, both when it comes to evaluations commissioned by state authorities and self-initiated ones by civil society for its own purposes. Accordingly, by practising evaluations within civil society, citizens and CSOs can possibly hold their governments accountable. Second, to take over responsibilities for providing social services in addition to or on behalf of state authorities (Howell & Pearce, 2001; Ojo & Mellouli, 2018; Kalm & Uhlin, 2015; Laville et al., 2015; Lovan et al., 2003; Rosenbaum, 2006). That’s what Meyer and colleagues call the “service function”. If state entrusts private non-profit organisations to overtake state duties, there is some need to hold these organisations accountable and evaluation may become an appropriate instrument for this task (Carman, 2009; Cutt & Murray, 2000). These two aspects can be extended to include those that are primarily located within the social system itself and not directly at the interface between society and politics, as in the case of the two aspects mentioned above. Third, CSOs mediate interests, which is important before the control function can be fulfilled (Forbrig, 2002). In particular, concrete issues are discussed (deliberation), which can then be introduced directly or indirectly into the political decision-making process. Interests are represented and mediation is made possible in the form of communicative arenas. Accordingly, one can speak here of an “interest mediation” function. In this context, evaluations can be an important means of stimulating deliberative communication within civil society organisations, as well as in the more general discursive context, and of shaping it in an

24

N. ZIERKE ET AL.

evidence-based way according to transparent and socially agreed criteria (cf. Hanberger, 2001). Fourth, to allow mediation between interests, citizens or social groups within civil society need to articulate these. That’s another general purpose of civil society organisations, as they enable the articulation of interests in an organised manner. This could be defined as the “voice function” of evaluation. Evaluation is an appropriate tool to give voice to civil society groups and citizens which has a positive impact on empowerment (cf. Fetterman, 2023). For example, in an evaluation, a CSO can involve various local communities and by doing so give them a voice, providing an organised opportunity to articulate their needs and political interests, which ultimately can be seen as a foundation for interest’s mediation and political control by civil society. Unlike the political system, the social system is not characterised by clear and formalised structures, which poses some methodological challenges in studying the role of evaluation in civil societies. The concept encompasses a wide variety of institutions and practices that are sometimes complementary, sometimes contradictory, and sometimes highly variable over time—in other words, particularly subject to historical developments, social change, and internal dynamics. This was also evident in the context of this study: civil societies in the Asia-Pacific region are heterogeneous and have very different institutional frameworks. For this reason, the concept of “institutionalisation of evaluation in the social systems” should be interpreted parsimoniously in this study and, accordingly, it should not be widely generalised. In addition, it is important to note that, as with the political system and the systems of professions, the lenses of a practice perspective and organisational theory are crucial in analysing the issue, especially when it comes to analysing CSOs. Finally, the case studies focused as much as possible on general institutions or regular or socially widespread practices and not only on possible singular examples of best practice or CSOs that are pioneers in evaluation. The following four aspects can be summarised as central to the analysis of the institutionalisation of evaluation in the social system (see Table 1, column 2): • Is evaluation an institutionalised or at least widely used tool for improving CSOs? (row 1 “use of evaluation by civil society”)

1

INTRODUCTION

25

• Are evaluation findings often included in public debates? How is evaluation perceived and understood by the general public? (row 2 “public discourse and knowledge”) • How are actors of civil society like CSOs or citizens involved in public evaluations? (row 3 “participation of civil society”) • How do civil societies request evaluations from political institutions? (row 4 “demand for evaluation”).

3.3

Institutionalisation of Evaluation in the System of Professions

Finally, some theoretical foundations of the last analysed societal subsystem shall be presented: the system of professions. In the field of evaluation capacity development, a distinction has long been made between the demand side and the supply side (e.g., Mackay, 1999). While the two societal subsystems discussed above were primarily concerned with demand, the supply side is conceptualised below in terms of “professionalisation”. At first, it may seem surprising that this is treated as a separate societal subsystem, since in principle it could also be subsumed as a special subset under the social system discussed earlier. Although it is a specific area within the social system, at the same time it has special links with politics and civil society that distinguish it from the rest of the social system and therefore justify treating it as a separate subsystem. Following Andrew Abbott’s (1988) sociology of professions, a separate system can be derived, whereby professions are characterised by a dual structure. First, it provides education and training and offers accredited degrees, for example through universities and polytechnics that teach and certify skills and knowledge. This serves a labour market in which certain skills are in demand. However, this does not usually cover a “profession per se”—i.e., there is no one hundred percent fit between market demand and supply side—because academic degree programmes are usually strongly oriented towards scientific knowledge production and are often embedded in basic research. Second, academic programmes are introduced within an autonomous scientific system with an academic purpose and not solely with the motivation to meet external demand. Of course, there are empirical differences in the relationship between “research and practice orientation” in both respects. Accordingly, different curricula can be expected at the “classic” “research university” than, for example, at the “entrepreneurial university” (Clark, 2001; Etzkowitz, 2003, 2004, 2013;

26

N. ZIERKE ET AL.

Guerrero & Urbano, 2012) or the “development university” (Akpan et al., 2012; Coleman, 1986; McCowan, 2019) and other variants of “social impact universities” (e.g., Oliver-Lumerman & Drori, 2021).5 Meyer and colleagues (2020, 2022) highlight different elements of professions, that were important in research on professions (MacDonald, 1995): First, “knowledge transfer” can be seen as a central dimension of every profession (Freidson, 2001). Moreover, the social exchange and communication flow between communities of practice contribute to the creation of an own professional identity (Colbeck, 2008; Trede et al., 2012). By doing so, knowledge and experiences are shared within social groups. Professional communication is usually anchored in the academic system. That’s why a crucial task for an evolving profession is to establish itself in the communication structures of academia (Chan & Fisher, 2008; Schiele et al., 2012), especially in peer-reviewed academic journals (Larivière et al., 2015). Besides this, there are other ways of communicating within a profession, that often are organised outside the academic realm. For example, this is the case when professional organisations establish own communication systems in order to link practice and academia (Friedman & Phillips, 2004; Greenwood et al., 2017). Lastly, each profession has a different degree of independence, which is basically about the ability to define own norms and standards for the professional practice (Rüschemeyer, 1983). As a social process, professionalisation is associated with social closure, which means that access to a social group is limited to varying degrees and based on different social mechanisms. For example, in one country the right to conduct evaluations (“service delivery” of the supply side) may be restricted to those who have a special certificate and, in another country, it may be rather linked to, e.g., practical experiences. A profession can use such mechanisms to control access to its own ranks and by doing so to define the criteria of the supply side of the market (MacDonald, 1985; Richardson, 1997). A profession that is successful in setting such standards has competitive advantages in the service market. Interestingly, many professions focus on establishing “inward oriented” standards (Ingvarson, 1998). For example, if a VOPE develops an ethical code of 5 For a general discussion of how changing economic and political circumstances—especially in the light of the globally evolving ‘knowledge-based economies’—affect the way science and the universities are organised and knowledge is produced (see Etzkowitz & Leydesdorff, 1998, 2000; Weingart 1997.

1

INTRODUCTION

27

conduct in order to strengthen evaluation quality. Such standards may not primarily focus on boundary work at the market, but rather identity work within the profession. Both aspects are important for a profession. Moreover, internally coordinated standards also affect the market in that the demand side is often oriented towards standards of established professions. The latter can be seen, for example, if one takes a look on various terms of references, where one can clearly observe that such standards are important reference points. In contrast to the social system, the empirical analysis of the categories mentioned above comes without major challenges. The indicators are mostly easy to capture as information on academic courses, trainings programmes, journals, other publication and exchange formats, professional organisations as well as ethical standards are usually published. Difficulties only may arise when evaluation is mainly part of other professions. This may also lead to a different understanding of “evaluation as a profession”. This can be summarised with the following key questions which will guide the analysis (see also Table 1, column 3): • Do academic study, trainings and qualification programmes for evaluation exist? (row 1 “Academic education and training practices”) • Do academic journals or other media and fora for evaluation exists? (row 2 “Journals and communication platforms”) • Do professional organisations, networks or associations that manage the exchange about evaluation and promote the development of this discipline exist? (row 3 “Professional organisations”) • Are there generally binding standards or rules for evaluation? (row 4 “Existence of and compliance to standards”)

3.4

Summary

In the Evaluation Globe project, the institutionalisation of evaluation is studied in three central societal subsystems, corresponding to different forms of implementation. In the political sphere, evaluation is used as an overarching governance instrument—not restricted to specific policy fields. Laws, acts, decrees and other regulations determine how evaluation is applied in administrative processes, while the ultimate evaluation practice evolves within this legal framework, with the possibility of many

28

N. ZIERKE ET AL.

different approaches, methods and instruments, with possible policyspecific adaptations. Overall, in the political system, evaluation results are used in policy-making processes to improve the quality of public services and government activities as a whole. Since public services are also provided by civil society actors, the social system comes into focus. In this system, CSOs and other actors of civil society can use evaluation for their own purposes, which may not coincide with the evaluation agenda of state actors. Evaluation can be used in the context of different objectives: to provide services, to monitor government activities in order to optimise them in terms of their own interests, to articulate and mediate different interests within civil society, and to give voice to different parts of civil society. In addition, civil society can commission its own evaluations or incorporate the results of evaluations by public actors into its own actions. Civil society can also be involved in government evaluations through participatory institutions and can demand such involvement. Finally, civil society is an important demanding force for government evaluations, as is the media, which also plays an important role in communicating knowledge about evaluation. The situation is different for the last societal subsystem examined: the system of professions. This includes, for example, academic study programmes or courses on evaluation at universities, provides exchange formats such as, in particular, publication opportunities for evaluation research in academic journals, but also other communication mechanisms that are important for the exchange of a community of practice in applied science, such as newsletters, blog posts, and social media. Voluntary Organisations of Professional Evaluation (VOPEs) can play an important role here, for example by providing such communication opportunities and institutionalising the exchange. As a result of these professional community exchanges, quality standards and ethical guidelines for “good evaluation” can sometimes be agreed upon. The following Table 1 summarises the issues mentioned above:

1

Table 1

INTRODUCTION

29

Dimensions of the institutionalisation of evaluation

Institutionalisation of evaluation in different societal subsystems Political system: Institutional Structures and Processes Institutional structures and processes National laws, regulations and policies Parliamentarian and national audit structures Organisational structure Evaluation practice

Social System: Dissemination and Acceptance of Evaluation in Society Dissemination and acceptance of evaluation in society Use of evaluations by civil society Public discourse and knowledge Participation of civil society Demand for evaluations

System of professions: Evaluation as a Discipline

Evaluation as a discipline

Academic education and training practices Journals and communication platforms Professional organisations Existence of and compliance to standards

Source Stockmann and Meyer (2016), Meyer et al. (2020, p. 25, 2022). Authors’ own work and illustration

4 4.1

Methodological Remarks Comparison in Research on Evaluation

The research approach implemented in the Evaluation Globe is comparative in nature and aims to assess the institutionalisation of evaluation in different cases against common overarching criteria. There is a wide range of comparative methods that have been used in research on evaluation for decades. Each specific design has its pros and cons, including cross-national comparative studies 6 like the present one. Some of these are explained below, followed by a rationale for why the chosen design was most appropriate for the research question at hand. Right from the start, the question arises to what extent “comparisons” can be seen as a social science method of its own right. This question is also raised by Arend Lijphart (1971) in a classical article in which he explains how the comparative per se is inherently part of the scientific method. He distinguishes four basic methods in the social sciences, which he divides into experimental and non-experimental methods. The 6 For an overview of this design please refer to the following literature: Elder (1976), Andreß et al. (2019), Livingstone (2003, 2013), Harkness (2012), Goerres et al. (2019), Ragin (2006, 2007, 2014), Hantrais (2008), and Wong (2014).

30

N. ZIERKE ET AL.

latter include case studies, statistical and comparative methods. Starting from this typology he elaborates on the methodological independence of comparisons; while pointing out overlaps with other methods. Although comparisons also play a role in experiments or statistics, they differ from the “method of comparison” itself in the sense that only the latter faces the typical “many variables, small N” problem of “handling many variables, more difficult to solve” (p. 685). The advantage of such a method is the possibility to derive adequate hypotheses, which can be analysed in more detail in follow-up research, e.g., by statistical analysis, provided that an increase in sample size is possible.7 In this design, macro-hypotheses can only be weakened in the face of divergent cases; they cannot contribute to a statistically significant falsification. That’s why Lijphart presents four ways to deal with weaknesses. First of all, the sample size can be increased. Second, certain categories, indicators or “variables” can be grouped together, which expands the possibilities of cross-tabular analysis. Third, it is possible to focus on cases that are as similar as possible, i.e., cases that are similar on as many variables as possible, but this usually reduces the number of cases. This follows or rather approximates John Stuart Mill’s famous method of agreement (Mill, 1843; Ragin, 2014, pp. 36–39). In the social sciences, the area approach can be seen as a method by which intra-unit comparisons can be implemented in certain regions, which “offers the possibility of establishing crucial controls” (p. 689). The Evaluation Globe capitalises on this advantage by taking a region-specific approach across the volumes, allowing more options for comparison. “The advantage of intra-unit comparison is that inter-unit differences can be held constant”, Lijphart explains concisely. Careful consideration must be given to how regions can be meaningfully defined; only then will they represent a possible control. Fourth, following the principle of parsimony the researcher can

7 Since the number of possible cases in so-called “cross-national studies” is limited, Lijphart suggests changing the unit of analysis under certain circumstances. Then, for example, “multinational but cross-individual” research can be conducted (cf. for this conceptual distinction Hopkins & Wallerstein, 1967). With regard to the study of evaluation systems, such a design can be used to study organisations or individuals (evaluation experts, parliamentarians, government officials, representatives of civil society, etc.). This allows for more meaningful statistical analyses and hypothesis testing, since it is possible to move to probabilistic large-N samples, in which many minimum requirements of inductive statistics are first met. The nation-state context would then be included as an additional explanatory variable.

1

INTRODUCTION

31

focus on central categories or variables. This means that a theoretically derived selection of relevant characteristics and a reasonable level of simplification, allows “discovering controlled relationships” (p. 690). This research strategy is also applied in the Evaluation Globe. As already mentioned, one challenge of the comparative design implemented here is the overall limited number of cases in the unit of analysis of individual countries, which may lead to anticipating decisions on hypotheses (Lijphart, 1975). At the same time, the selected cases are very heterogeneous in certain categories, which protects against this and encourages caution. This empirical diversity of the cases in some aspects is quite advantageous from a qualitative research point of view, as it allows for contrasting comparisons. Seen in this light, an adequate representation of the empirical situation is supported by a varied selection of cases.8 The comparative method used here is based on case studies written by experts and takes advantage of the methodological advantages of this data collection tool. Overall, then, what emerges here is not so much a variable as a case-oriented research strategy (cf. Della Porta, 2008, p. 198ff.; Ragin, 1991), which captures key categories across different cases.9 This allows for detailed analysis and special contextualisation per case; ultimately, prepositions important for theory development or specification of hypotheses can also be identified. Especially in an international research project, it is important to work with local experts to understand specific contexts. The disadvantage of individual case studies is their low degree of generalisation (cf. Lijphart, 1971, p. 691), although, through the systematic comparison of the individual cases in a cross-country study design, this can, however, at least partially be made possible.

8 This and other strengths of small-N cross-national comparisons are discussed by Ebbinghaus (2005). 9 Overall, the Evaluation Globe research project combines intra- and inter-regional comparative research strategies. The selection of cases has to be balanced between the number of cases and the heterogeneity of the case conditions. While intra-regional comparisons tend to be homogeneous in order to allow for a most similar systems comparison, inter-regional designs tend to be heterogeneous in order to allow for a most different systems comparison and exploratory research. The sample in the Evaluation Globe allows for both: the case selection consists of relatively homogeneous cases with individual contrasting case studies; with respect to the comparison between the volumes, i.e., the world regions, the diversity of the systems studied and their basic conditions then increases by leaps and bounds, i.e., approaches a most different case design (cf. Basedau & Köllner, 2007).

32

N. ZIERKE ET AL.

Due to the overall case-oriented approach, the way of explaining possible relationships is best understood as “combinatoric”, which means “stressing specific configurations of causal conditions” (Ragin & Sonnett, 2005, p. 180). Thus, the comparison allows us to identify more general empirical patterns of institutionalisation of evaluation, without aiming at “net effects of causal conditions” (ibid.). However, due to the goal of systematic comparability based on central categories, this is accompanied by some reduction in the contextual information required for synthesis. That’s the methodological trade-off of the research strategy implemented. Nonetheless, the synthesis qualitatively interpretively uses the case studies to point out important contexts in the overview. Comprehensive contextual analyses and more extensive interpretations of individual cases in line with the “qualitative paradigm of social research” would have to be implemented with other research approaches, such as comprehensive single case study designs or comparative designs of fewer cases. The general research imperative in individual case studies is to pay more attention to case descriptiveness than to comparability, which is sometimes not possible, for example, when local specificities simply cannot be captured in terms of the categories of analysis. With regard to the methodology of the individual case studies, it is of pragmatic advantage—especially for the purpose of comparison—to provide the case study authors with an analytical guideline containing the central categories of investigation. On the other hand, too much standardisation can make specific contextualisation more difficult. In view of this, the typologised systematic comparative research approach of the Evaluation Globe takes a middle course, so to speak. The cases all follow the research grid, but allow for important contextualisation due to the basically open method of the expert-based case study; however, as described, this is not as extensive as a fully open-ended qualitative case study. Future studies, however, can take the synthesis findings of the intraregional comparison as a starting point for complementary sociological, political science, and historiographical studies, for example, by contrasting them with those. Specific problems of the comparative approach of the Evaluation Globe have already been discussed in the Americas volume (Meyer et al., 2022) and will only be briefly recapitulated here. First, there is the challenge of construct equivalence. The concept of evaluation is used in different senses, and its precise use varies according to the sector, country context or dimension of the study under consideration. This is most evident when

1

INTRODUCTION

33

the concept of evaluation is not clearly understood in relation to other instruments such as auditing, controlling or monitoring. The glossary provided to the authors with key definitions of terms does not change this. Constructive equivalence is therefore not guaranteed. In China, for example, the term is used for three different evaluation procedures, one of which is auditing in the strict sense. It is therefore the task of the case study authors to point out these differences from the globally shared understanding of the term evaluation as it is used in the glossary. This was supported by the review process, for example, by distinguishing in some case reports between the use of the term in the narrow sense and in the broad sense. In the context of media discourse, for example, the often very broad understanding of the term became apparent in the course of the analysis, which can be seen as an important result in this respect. Ultimately, how the term is interpreted in relation to the individual categories is always a researcher’s perspective. The second specific problem concerns the research methods used in each of the case studies. These can be easily synchronised in some cases like when it comes to VOPEs using other data sources. The documents are accessible and can easily be incorporated into the case study without introducing a particular research tool. In other cases, for example, evaluation practices vary widely across policy areas and countries, each requiring its own methodological response. There can therefore be no “one size fits all” approach to the methods used in case studies, although a combination of systematic document analysis and expert interviews has proved to be a very useful method in general, especially for sub-questions that are still quite unexplored. The case study authors used their own approaches and methods to answer the specific research questions and, where appropriate, noted existing limitations in the evidence base. Some case studies refer to existing research evidence where appropriate. At other points, the case studies give greater weight to descriptiveness than to comparability in terms of research approach—which is then discussed in the synthesis. 4.2

Case Selection and Research Process

The process of selecting suitable cases and then suitable case study authors is another challenge and central to the overall research concern. The selection of cases was guided by a multi-stage selection process. This included recommendations from well-known experts in the field as well as the results of repeated systematic literature reviews. An iterative process was

34

N. ZIERKE ET AL.

used to select the cases, both for the eleven countries and for the three international organisations. In some cases, network contacts with proven evaluation expertise in the respective countries were also involved in the project, mostly with reference to the favorable possibility of co-authoring a paper with other country experts. This was gratefully accepted in each of these cases, which is certainly a methodological advantage in terms of peer review. In selecting the cases, the feasibility of conducting a case study according to the analysis grid (see Appendix 2) had to be considered; here, too, literature and expert feedback were used. In a survey, on the other hand, cases with very few characteristics can easily be included; in the case study comparison design implemented here, however, this would lead to problems with regard to the narrative density for the individual categories. In the selection of cases, it was gradually decided to limit the study to the Asia-Pacific region rather than the continent of Asia as originally considered. This was also accompanied by considerations of how best to define a region to allow for contrasting comparisons, i.e., controls, when viewed inter-regionally (as described in Sect. 4.1). At the end of the iterative process, 14 country case studies (Australia, Bangladesh, Bhutan, China, India, Indonesia, Japan, South Korea, Nepal, New Zealand, Pakistan, the Philippines, Sri Lanka, and Taiwan) and 3 international organisation case studies (IED of the ADB, UNDP and UNICEF) were identified. Unfortunately, the case studies on Bhutan, Indonesia, and New Zealand could not be included in the volume because important categories of inquiry were not or could not be covered, or the underlying approach and analytical framework were not followed. Accordingly, the section on country studies ultimately comprised 11 case studies. In total, this corresponds to 37 participating authors, who dedicated themselves with great academic curiosity to the challenging and in many cases little researched topic. At the beginning of the research process, the authors were asked to familiarise themselves with all materials, such as the concept note, previous studies of the compendium, the analytical guideline, the publisher’s publishing guide, and the glossary of key terms. The central research tool is the analytical guideline: The analytical framework had been developed to be a lean and feasible instrument, on the one hand to keep the efforts for the experts within an acceptable frame, on the other hand to guarantee that the case studies are focused on the most relevant aspects and still keep an acceptable length. (Meyer et al., 2020, p. 22)

1

INTRODUCTION

35

For the second volume, minor adaptions were made to the category system, while maintaining comparability (see Meyer et al., 2022). For the Asia-Pacific volume, adjustments were made to the main set of instruments, the two analytical guidelines; both the one for international organisations and the one for country case studies. In terms of content, this concerned the specification of some questions to specific categories, as well as the summary of certain questions or generally a more easily understandable wording; in addition, individual questions that had not been comparatively examined since the first volume were excluded. Prior to the start of the writing process, authors were encouraged to form co-authoring teams, as there are many advantages to doing so. First, a team of authors usually strengthens the peer review process; and it also leads to a division of labour in specialising in different aspects of the analytical guideline, both have a positive impact on quality. In addition, authors can also be selected according to their technical specialisation (e.g., if there is particular expertise on a country’s civil society or its political system and administration). The preparation of the papers was guided by methodological recommendations. Authors were asked to conduct a detailed literature and document review and to adequately address all aspects of the analytical guideline. They were given the freedom to choose which further research methods to refer to. Guided expert interviews were communicated as a recommended method, but surveys, quantitative analysis of secondary data or other types of interviews are also conceivable to address certain aspects of the analytical guideline, especially if corresponding aspects cannot be covered by the literature. In principle, a pure desk study would be possible if there were studies covering the topic in question—but this is usually not the case (Fig. 1). The research project spanned approximately 2 years, with phases of extensive preliminary research, detailed planning, and ultimate implementation—i.e., an extended phase of article writing and subsequent review—and final editing. At the end of 2020, the first research and expert discussions began. In early 2021, the first potential authors were contacted, and in April/May 2021, the authors were finally selected and the starting signal was given for the creation of the contributions. Initially, six months were planned for this, which was then extended again due to the corona pandemic. In the early phase, existing questions were discussed with the authors to ensure a common understanding. Throughout the process, authors could contact the editors at any time to

36

N. ZIERKE ET AL.

UNICEF ROSA Nepal

Japan China South Korea

India

Taiwan

Pakistan

IED ADB, HQ Bangladesh The Philippines

Sri Lanka

UNDP Regional Hub UNICEF EAPRO

Australia

UNDP Regional Hub

Fig. 1 Selected countries and international organisations for the case studies (Note UNDP Pacific Office in Suva, Fiji; UNDP Regional Hub in Bangkok, Thailand; Headquarter of the Regional Bureau for Asia and the Pacific of UNDP in New York, USA; Headquarter of the Asian Development Bank in Manila, Philippines; UNICEF East Asia and the Pacific Regional Office [EAPRO] in Bangkok, Thailand; UNICEF South Asia Regional Office [ROSA] in Kathmandu, Nepal; Headquarter of UNICEF, in New York, USA with additional special regional offices in Seoul, South Korea and Tokyo, Japan. Source Author’s own work)

1

INTRODUCTION

37

discuss methodological or content issues or to coordinate the process. Finally, in early 2022, many papers were available and a multi-stage peer review process began. Some papers were received a little later due to delays caused by Corona. The subsequent team’s internal review process was also delayed by Covid-related constraints. By the end of 2022, the reviewed manuscripts were available and their detailed content analysis began, resulting in the synthesis chapters and the introduction. In the course of this, a few additional questions about some texts arose again and accordingly, further review and optimisation loops were implemented, which ultimately led to the finalisation of the contributions and submission of the manuscript to the publisher in March 2023. The review was essentially concerned with quality assurance of the papers or draft papers with respect to several criteria. First, it was necessary to ensure that all categories of analysis were covered. Then, the quality of the individual aspects had to be assessed and it had to be noted where further information or content additions or changes were deemed necessary. Issues of comprehension were also often discussed. In further review loops for some articles, additional questions were asked to ensure adequate assessment in the synthesis. Some articles were submitted almost ready for publication and required only minor comments. Others, particularly the country case studies, required multiple queries and detailed discussions to accurately reflect the findings for the synthesis according to the case reports. In some cases, the internal review by the editors, based on the principle of four or six eyes, included the use of secondary sources to validate the content of information (e.g., previous studies). As a final note, a methodological feature should be mentioned that mainly concerns the category-oriented data analysis of the case studies for the synthesis of the contributions, but was also important in the context of the review: the work with the category system. The categories are more or less specific based on individual criteria. Thus, in some cases, the indicator is based on essential “countable” information, where the criterion is intuitively understandable (e.g., evaluation reports are published), while in others, summative indicators, i.e., indicators that cover several aspects of content, and more qualitatively weighted indicators were formed (such as how is evaluation used and practised in civil society). The considerations presented in the following two paragraphs relate primarily to institutionalisation in the social system; as in the other two systems, the individual criteria involve a more fine-grained and easily quantifiable categorisation.

38

N. ZIERKE ET AL.

On the basis of the previous volumes, as well as in the course of the content-analytical examination of the texts, a comprehensible analysis and evaluation mechanism gradually emerges in the work with the empirical material in the case of the more open qualitative categories, which is based on the specific criteria of the previous volumes. While for most indicators the evaluation of the criteria is intuitive or easily accessible on the basis of essential information in the text, for the latter categories and their criteria there is a greater qualitative openness, which is also conducive to the consideration of case-oriented or regional specificities. In such cases, there is a pragmatic element in the fine-tuning of the analysis mechanism, which makes it easier to assess the criteria. This happens iteratively in the process of working with the material, which is why multiple passes through the material are helpful, especially when contrasting qualitative comparisons are made between cases, as well as additional built-in feedback processes. All this contributes to the best possible assessment of the criterion. The question, then, is how exactly, for example, “participation” or “use of evaluation in civil society” can be accurately assessed on the basis of the evidence analysed in the case studies. Nevertheless, a natural limitation of the method used, namely the empirical basis provided by the case studies for the criteria, becomes apparent here; this is particularly striking in the case of individual categories that have hardly been researched in some countries so far, such as the use of evaluation in civil society. These knowledge gaps were addressed in the review, but could not be fully filled in all cases, as many individual studies—not yet conducted—would relate to them (e.g., when it comes to an exact quantitative determination of the public impact of evaluation via the media). Since the qualitative interpretation is more important here, it was important to carry it out in conjunction with the two previous volumes of the compendium in order to maintain comparability. This contributed to a comparable assessment that is as realistic and qualitative as possible. In case of doubt given the available data in the case studies, a conservative assessment has been made. Once the criteria and qualitative features in the cases have been identified and sufficiently comparatively analysed, the next analytical step follows the comparative evaluation of the evidence in each case. A fundamental methodological challenge here is how to translate qualitative categories into numerical values in a comprehensible and meaningful way. This concerns the implementation of conversion mixed methods design in the analysis strategy discussed in the literature: qualitative information is converted into quantitative information (cf. Schoonenboom &

1

INTRODUCTION

39

Johnson, 2017). Quantifying qualitative data is challenging in some cases given the specifics in the case studies. For example, the question arises of how many categories the numerical category system should have in order to represent the qualitative exhaustively but also in a meaningful way. Complexity reduction must be comprehensible. If the category system is too broad, the concrete assignment could be arbitrary; if there are too few categories, too much complexity might be reduced. In the present analysis, the distinction between “low spread” and “medium spread” involves extensive consideration in some cases, especially when only subtle differences are found between similar cases. Fortunately, however, the cases in which a clear assignment is possible with a relatively high degree of certainty predominate in the sample, e.g., when it is clearly documented in the case study that there is little or no demand for evaluation. Three categories are thus justifiable in view of the tendency towards the lowest category in this dimension, but perhaps at least a little more variability could still be covered with two medium categories. On the other hand, this would again make the clear identification of criteria and their precise grading a challenge, especially since “low, medium, high” is in principle a generally well-understood abstraction. And because an average score is ultimately formed from ratings on multiple theoretically based categories, its content validity is likely to be stronger than if only a single criterion were considered. Different qualitative and quantitative types of data each go hand in hand with their own approaches to empirical reality and therefore allow for different insights. Mixed methods research is dedicated to their integrative analysis (Bazeley, 2017). While the evaluation of quantified qualitative data cannot replace detailed qualitative analysis, it has a complementary relationship to it (Fakis et al., 2014, p. 156). Quantification is therefore referred to as a “complementary method” (ibid.). The integration of both methodological perspectives offers advantages, as the results expressed in numerical values complement the qualitative research mode (Maxwell, 2010). In particular, quantitative analysis “helps identify patterns that might not be apparent through qualitative analysis because more complex patterns are often difficult to access through qualitative analysis” (Vogl, 2017, p. 309). In the Evaluation Globe’s analysis, qualitative and quantified results are treated in an integrative way, so that “quantification does not become an end in itself” (ibid.). Transforming the data in this way is not a trivial step and requires that decision rules be found for each category, e.g., what

40

N. ZIERKE ET AL.

is counted and how. As described above, this is easier for some subcategories than for others. Since not all information can be transformed to achieve meaningful complexity reduction, it is helpful to mention what was not transformed. Challenges include “balancing numerical precision with narrative complexity” (Sandelowski et al., 2009, p. 208). There are always trade-offs involved, and in the end, each research entails a specific way of dealing with the data forms and results and the weight given to them. In summary, the Globe project quantifies to enable the intended systematic comparison beyond qualitative data. Here, the transformed information offers a crucial contribution to obtain a comparative overview across several world regions and content dimensions. The particular added value of the research project is the clearer identification of patterns in the findings. Limitations of the implemented transformation lie in the sometimes rather ambiguous assignment of information under a specific criterion, making the decision of assigning a specific numerical value more difficult. In most cases, especially with respect to the political system and professional system criteria, a clear “intuitive” assignment is possible because the indicators refer to concrete countable or measurable objects—be it an evaluation policy, an organisational unit, or a VOPE. 4.3

Epistemology and Reflection

To conclude, some epistemological remarks will be made. Regarding the deductive derived theoretical categories, while their cross-case relevance is theoretical assumed for this study, it seems important to note that it is not the only possible theory for studying the subject matter. This idea is well expressed in a delightfully pragmatistic quote from Karl Popper (1972) as follows: “Whenever a theory appears to you as the only possible one, take this as a signal, that you have neither understood the theory nor the problem, which it was intended to solve” (p. 266). Moreover, modifications and case sensitive adaptions can be made in further research, e.g., when focusing on specific cases or sub-regions or on different state structures like the municipal or federal level—i.e., not having international comparisons in mind. Besides that, the concept of analysis allows to cover a variety of locally specific evaluation practices and forms of institutions. Thus, it considers itself as a descriptive investigation tool appropriate for dealing with a diverse set of cases. Ultimately, it is open to inductive category extensions by the case study authors, which contributes to a

1

INTRODUCTION

41

higher content saturation of the individual case studies. For inter- and intra-regional as well as international comparative research, some kind of agreement on the important contents of an analytical framework is supportive. A proposed understanding of a framework—open to revisions and iterative learning processes—is useful for enabling a dialogue and achieving common progress in knowledge. The Evaluation Globe project and the results presented in the following are committed to this purpose of international comparative research.

Appendix 1: Glossary for the Evaluation Globe---Compendium on the Institutionalisation of Evaluation 1. Evaluation: “Evaluation is the systematic investigation of the merit or worth of an object (program) for the purpose of reducing uncertainty in decision making” (Mertens, 1998) 2. Monitoring: “The systematic documentation of aspects of project performance (process, output, outcomes, impacts, results) that are indicative of whether the project is functioning as intended or according to some appropriate standard” (Rossi/Freeman/ Lipsey 1999). 3. Performance Audit: Objective and systematic examination of effectivity (goal-achievement) and efficiency (economic use of resources), based on external criteria. 4. Meta-Evaluation/Analyses: Meta-Evaluation/Analyses reanalyse evaluation (reports) with regard to specific analytical questions. This could be the content of the different evaluations as well as the methodological approaches or quality aspects. Another goal of meta-evaluations/analyses could be to aggregate results. 5. Comment: We assume that different countries or even sectors have different ways in dealing with this issue. Due to the fact that we would not like to exclude any form of meta-analysis we would also subsume evaluation synthesis reports, meta-evaluations or other forms under this term. 6. Internal & External Evaluation: External Evaluation describes evaluations carried out by persons (experts) who are not part of the implementing or funding organisation. Internal Evaluation describes evaluations carried out by the same organisation that is

42

N. ZIERKE ET AL.

implementing the programme. Self-evaluation is a specific form of an internal evaluation, in which case the programme implementing persons are the same as the programme evaluators (cf. Stockmann & Meyer, 2014) 7. Impact vs. Process Evaluation: Process Evaluations look at the stage of planning and implementing activities, basically their inputs and outputs whereas impact evaluations focus on the actual results and impact of programme measures. Impact refers to changes and reasons for change, thereby investigating expected as well as unexpected impacts. 8. Functions of Evaluation: According to Reinhard Stockmann and Wolfgang Meyer evaluation has three societal functions: i. Planning and Steering: The first one encompasses use of evaluation for management purposes. The instrument of evaluation is used to improve effectivity, efficiency and impact of measures and interventions. ii. Accountability and Legitimacy: In the understanding of the second function, evaluation serves for the purpose of good governance. The implementation of politically intended strategies is controlled and public accountability is provided through use of evaluations. iii. Enlightenment: For the purpose of enlightenment, evaluations can provide necessary facts to enable public dialogue. Relevant questions for this dialogue are, whether administration (where implementation and steering of programmes and measures takes place) and government & parliament (where political steering takes place) follow society’s values and goals, or not. 9. Institutionalisation: Institutionalisation indicates emergence and permanent anchoring of formal and informal rules, structures and processes that guide and stabilise behavior of individuals as well as groups, thereby regulating living together in society. 10. Evaluation Culture: Due to the fact that the term of culture is rather vague we kindly ask every author to work with the concept of “institutionalisation” as defined above and furthermore described within our research proposal. 11. Evaluation Regulations

1

INTRODUCTION

43

I. National and sectoral laws and regulations: Regulation in our case refers to legal norms that are established by state’s executive instead of legislative. They are issued for implementing or complementing existing laws and are generally binding (cf. Schubert & Klein, 2016). II. National and sectoral strategies or policies: The term “policy” can be defined as coherent strategy for action focusing on a specific thematic area. The extent of the structure can be varying from the description of concrete next steps to a very low level of structure. All policies (strategies) have in common that they postulate desirable goals (cf. Bussmann et al., 1997, p. 66f., 83). III. Administrative regulations (instructions, guidelines): This term encompasses regulations addressed at administrative authorities. 12. Independent evaluation institute: Institute, which is supposed to examine politics either of one specific sector or all national politics. Thereby the institute can decide completely independent on which thematic area it wishes to conduct an evaluation. A state’s Federal Audit Office can be such an institute, if it has an evaluation function. 13. Independent evaluation units: Ministerial departments that are provided with financial and personal resources and have got the freedom to carry out evaluations independent of the operational units. 14. Quality of Evaluations/Evaluators: It can be differentiated between mechanisms that are supposed to foster the quality of evaluations, such as standards, and those that foster the quality of evaluators, such as Codes of Ethics, Gentlemen’s Agreements or Guiding Principles for Evaluators. 15. Civil Society: The concept of civil society encompasses an area of self-organisation and self-administration by citizens, namely in form of associations organizations. For the book’s purpose we understand civil society as organized in form of civil society organisations as well as non-governmental organisations, which we do not differentiate any further.

44

N. ZIERKE ET AL.

16. Sector: Sector in this case refers to different policy-fields, such as the health sector, the education sector, the sector of development cooperation. From a system theoretical perspective, they can be seen as subsystems of the political or social system.

Appendix 2: Analytical Framwork: Compendium on the Institutionalisation of Evaluation I. Political System: Institutional Structures and Processes I.1 Evaluation regulations I.2 Evaluation practice I.3 Use of evaluations • Are there national laws • With regard to the whole • Which sectors are “good or regulations about country: How would you performers” regarding evaluation or use of describe the scope of use of evaluation and evaluation? If yes, which? conducted evaluations? Is evaluation findings? it possible to speak of a Please describe up to 3 • Are there sectoral laws or frequent rhythm of sectors that can be regulations about evaluations, for instance considered as leading in evaluation or use of for every new legislation the field of evaluation’s evaluation (e.g. a law or for every national use about school evaluation programme? Or is it or evaluation in the • Which sectors are “bad rather non-specific? Does higher education system performers” regarding evaluation take place in as example for laws in use of evaluation and all sectors/policy fields of the educational sector)? evaluation findings? a country (instead of If yes, which? Please describe up to 3 only in the field of sectors that are lagging • Are there policies or development cooperation behind in the field of strategies about for example)? And within evaluation’s use evaluation or use of one sector, is evaluation • On which aspect do evaluation, either national applied for measures or sectoral? If yes, which? most evaluations focus funded in different ways in these sectors (e.g. • Are there administrative or maybe only the ones Planning and Steering, regulations about that received funding by Accountability and evaluation or use of the European Union? Legitimacy, evaluation in different Enlightenment)? policy fields (instructions, • With regard to the whole country: How would you guidelines, etc.)? If yes, – In the case of describe the relation which? different findings in between internal and different sectors: external evaluations? What might be Which form is carried possible reasons for out more often and for these differences? what purposes? (continued)

1

INTRODUCTION

45

(continued) • What is the content of these laws/regulations/ policies/strategies or administrative regulations regarding independence of evaluation, quality, impact orientation and available budget? – Is use of evaluation specified? If yes, how? – How binding are specifications regarding use of evaluation? – What are aspired functions of evaluation (e.g. Planning and Steering, Accountability and Legitimacy, Enlightenment)? • Is evaluation and use of evaluation findings embedded in parliamentary structures? If yes, how? – Do parliamentarians in your country deal with evaluation findings for their own political work? If yes, to what extent (how often/ how detailed do they use evaluation findings)? – Do parliamentarians in your country demand evaluations for their own political work? If yes, to what extent? (How often? Do they commission evaluations? Do they publicly demand evaluations)?

• What are possible reasons • Which professional for this (e.g. groups use evaluation determination in laws, and evaluation findings policies or regulations)? regularly (e.g. political decision makers, • Is this relation differing programme or project with regard to sector or manager, administrative state level? staff)? • With regard to the whole – For what reasons are country: How would you evaluations and/or describe the relation evaluation findings between process and used by these groups? impact/outcome evaluations? Which form – How is the use of is used more often and evaluation findings for what purposes? guaranteed (for instance: management – What are possible response mechanisms, reasons for this (e.g. implementation of determination in laws, monitoring for policies or evaluation results, regulations)? others)? – Is this relation – Are there differences differing with regard with regard to to sector or state level? different sectors? • Does an independent evaluation institute exist • How is the quality of evaluations guaranteed in your country? (e.g. regular conduction – With a national of meta-evaluations responsibility? analyses, competence – With a responsibility requirements for for a specific sector or evaluators, quality policy field? requirements for • Do independent internal evaluations)? departments exist, in ministries or elsewhere?

(continued)

46

N. ZIERKE ET AL.

(continued) II Social System: Dissemination and Acceptance of Evaluation in Society II.1 Institutionalised use II.2 Public perception and II.3 Civil societies of evaluations by civil discussion of evaluation demand evaluations society and evaluation findings • Is it usual practice in • How well-known is the • Do individual citizens, your country that instrument of evaluation civil society evaluations are used to in society? organisations, private provide knowledge for enterprises or other • Are evaluation reports referenda or political actors in your country (full version) made decision making on a demand evaluations, e.g. publicly available? communal basis? from political • Is the general use of decision-makers? – If yes, how regularly evaluation publicly does this happen? If – If yes, how often does discussed in media not, what might be this happen and under (benefits of evaluation, possible hindering which circumstances/ quality of evaluations, factors? for what reasons? If and professionalisation of not, why not? What evaluation)? • Are evaluations and might be possible evaluation findings used – If yes, to what extent? hindering factors? by individual citizens/ If not, what might be civil society organisations possible hindering and or private enterprises factors? or other actors? • Are findings of actual – If yes, for what reasons evaluations publicly (e.g. enforcement of discussed (surprising their interests, findings, different knowledge or proof possibilities of dealing for work related issues, with these findings)? knowledge or proof – If yes, to what extent? for voluntary activities If not, what might be etc.) and how possible hindering regularly? If not, what factors? might be possible hindering factors? • Is it usual practice in your country that citizens or civil society organisations (NGOs, CSOs, churches etc.) are participating in evaluations (as stakeholder)? (continued)

1

INTRODUCTION

47

(continued) – If yes, how regularly does this happen? What are different forms of participation (e.g. as interview partners, as clients, as users of evaluation findings etc.)? If not, what might be possible hindering factors? III. System of Professions: III.1 Academic study courses, further training et cetera • Do programmes of higher university education for evaluators (Diploma, Master) exist in your country? If yes, how many and where? • In which other scientific disciplines is evaluation instructed as scientific subject? Please give as many examples as possible. • Do other forms of academic or non-academic training exist? (e.g. e-learning, training by consultancies, else)?

Evaluation as a Discipline III.2 Profession/discipline III. 3 Compliance to standards and quality obligations • Which professional • Do professional journals, newsletters or organisations ask their other ways/media of members to follow communication (e.g. standards or guiding e-Mail or discussion lists) principles? If yes, how exist? obligatory is this? • Which professional • Do clients demand a journals from other certain evaluation scientific disciplines deal quality and/or with evaluation regularly? compliance to standards? How does this demand • Does a professional look like (is it organisation obligatory)? (VOPE—Volunteer Organisations for • To what extent do Professional Evaluation) evaluators (and clients) exist in your country? follow these standards and/or quality – How is it organised obligations? (closed associations or open network)? – What is their actual number of members? • Do standards, guiding principles for evaluators or s.th. similar exist in your country? – Developed by the VOPE? – Adopted from another VOPE? (continued)

48

N. ZIERKE ET AL.

(continued) • Would you say that the evaluation market in your country is mostly dominated by freelancers (people calling themselves evaluators), consulting firms or scientific research institutes? • Does a certification system for evaluators exist in your country? • Does an authority, which might be asked to conciliate in case of a dispute, arbitration board exist in your country, like an arbitration board or ombudsman? • Does a professorship for evaluation exist in your country?

References Abbott, A. (1988). The system of professions: An essay on the division of expert labour. Chicago University Press. Ahmed, V., & Bamberger, M. (1989). Monitoring and evaluating development projects: The South Asian experience, seminar report series. World Bank, Economic Development Institute. Ahram, A. I. (2011). The theory and method of comparative area studies. Qualitative Research, 11(1), 69–90. Akpan, W., Minkley, G., & Thakrar, J. (2012). In search of a developmental university: Community engagement in theory and practice. South African Review of Sociology, 43(2), 1–4. Alexander, J. C. (2001). Soziale Differenzierung und kultureller Wandel: Essays zur neofunktionalistischen Gesellschaftstheorie. Campus-Verlag. Andreß, H. J., Fetchenhauer, D., & Meulemann, H. (2019). Cross-national comparative research—Analytical strategies, results, and explanations. KZfSS Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 71(1), 1–28. Ansell, C., & Torfing, J. (Eds.). (2016). Handbook on theories of governance. Edward Elgar Publishing.

1

INTRODUCTION

49

Ansell, C., & Torfing, J. (2021a). Co-creation: The new kid on the block in public governance. Policy & Politics, 49(2), 211–230. Ansell, C., & Torfing, J. (2021b). Public governance as co-creation: A strategy for revitalizing the public sector and rejuvenating democracy. Cambridge University Press. Armingeon, K. (2016). Political institutions. In H. Keman, & J. J. Woldendorp (Eds.), Handbook of research methods and applications in political science (pp. 234–247). Edward Elgar. Aulich, C. (2009). From citizen participation to participatory governance in Australian local government. Commonwealth Journal of Local Governance, 2, 44–60. Bamberger, M. (1989). The monitoring and evaluation of public sector programs in Asia: Why are development programs not evaluated? Evaluation Review, 13(1), 223–242. Bamberger, M. (1991). The politics of evaluation in developing countries. Evaluation and Program Planning, 14(4), 325–339. Bank, V., & Lames, M. (2000). Über Evaluation. bajOsch-Hein. Barker, R. M. (2010). Corporate governance, competition, and political parties: Explaining corporate governance change in Europe. Oxford University Press. Basedau, M., & Köllner, P. (2007). Area studies, comparative area studies, and the study of politics: Context, substance, and methodological challenges. Zeitschrift Für Vergleichende Politikwissenschaft, 1(1), 105–124. Bazeley, P. (2017). Integrating analyses in mixed methods research. Sage. Beckert, J. (2010a). How do fields change? The interrelations of institutions, networks, and cognition in the dynamics of markets. Organization Studies, 31(5), 605–627. Beckert, J. (2010b). Institutional isomorphism revisited: Convergence and divergence in institutional change. Sociological Theory, 28(2), 150–166. Beeson, M. (2008). Institutions of the Asia-Pacific: ASEAN, APEC and beyond. Routledge. Berg-Schlosser, D. (2012). Comparative area studies—Goldener Mittelweg zwischen Regionalstudien und universalistischen Ansätzen? Zeitschrift Für Vergleichende Politikwissenschaft, 1(6), 1–16. Bertelli, A. M. (2012). The political economy of public sector governance. Cambridge University Press. Bevir, M. (2012). Governance: A very short introduction. Oxford University Press. Blomeyer, R., & Eussner, A. (2020). European Union. In R. Stockmann, W. Meyer, & L. Taube (Eds.), The institutionalisation of evaluation in Europe (pp. 463–479). Palgrave Macmillan. Borrmann, A., & Stockmann, R. (2009). Evaluation in der deutschen Entwicklungszusammenarbeit. Band 1: Systemanalyse. Band 2: Fallstudien. Sozialwissenschaftliche Evaluationsforschung, Band 8. Waxmann.

50

N. ZIERKE ET AL.

Borthwick, M. (2018 [1992]). Pacific century: The emergence of modern Pacific Asia. Routledge. Borthwick, M., & Latz, G. (2022). The Pacific century study guide. Routledge. Bose, S., & Jalal, A. (2023). Modern South Asia: History, culture, political economy. Routledge. Brante, T. (1988). Sociological approaches to the professions. Acta Sociologica, 31(2), 119–142. Brook, C., & McGrew, A. (2013). Asia-Pacific in the new world order. Routledge. Bundi, P. (2018). Parliamentarians’ strategies for policy evaluations. Evaluation and Program Planning, 69, 130–138. Burns, T. R. (2006). The sociology of complex systems: An overview of actorsystem-dynamics theory. World Futures, 62(6), 411–440. Bussmann, W., Klöti, U., & Knoepfel, P. (1997). Einführung in die Politikevaluation. Helbig & Lichtenhahn. Carman, J. C. (2009). Nonprofits, funders, and evaluation: Accountability in action. The American Review of Public Administration, 39(4), 374–390. Chan, A. S., & Fisher, D. (Eds.). (2008). The exchange university: Corporatisation of academic culture. UBS-Press. Cheung, A. B. (2005). The politics of administrative reforms in Asia: Paradigms and legacies, paths and diversities. Governance, 18(2), 257–282. Cheung, A. B. (2016). NPM in Asian countries. In T. Christensen, & P. Lægreid (Eds.), The Ashgate research companion to new public management (pp. 147– 160). Routledge. Cheung, A. B., & Scott, I. (2003). Governance and public sector reforms in Asia: Paradigms, paradoxes and dilemmas. In A. Cheung, & I. Scott (Eds.), Governance and public sector reform in Asia (pp. 12–35). Routledge. Christensen, T., & Lægreid, P. (Eds.). (2007). Transcending new public management: The transformation of public sector reforms. Ashgate. Christensen, T., & Lægreid, P. (2022). Taking stock: New public management (NPM) and post-NPM reforms-trends and challenges. In A. Ladner, & F. Sager (Eds.), Handbook on the politics of public administration (pp. 38–49). Edgar Elgar. Church, P. (2017). A short history of South-East Asia. Wiley. Clark, B. (2001). The entrepreneurial university: New foundations for collegiality, autonomy, and achievement. Higher Education Management, 13(2), 9–24. Coase, R. H. (1937). The nature of the firm. Economica, 4(16), 386–405. Coase, R. H. (1991). The institutional structure of production. Journal des économistes et des études humaines, 2(4), 431–440. Cohen, J. L., & Arato, A. (1994). Civil society and political theory. MIT Press. Colbeck, C. L. (2008). Professional identity development theory and doctoral education. New Directions for Teaching and Learning, 113, 9–16.

1

INTRODUCTION

51

Coleman, J. S. (1986). The idea of the developmental university. Minerva, 24(4), 476–494. Coleman, J. S. (1990). Foundations of social theory. Harvard University Press. Cram, F. (2015). Lessons on decolonizing evaluation from Kaupapa M¯aori evaluation. Canadian Journal of Program Evaluation, 30(3), 296–312. Cutt, J., & Murray, V. (2000). Accountability and effectiveness evaluation in non-profit organisations. Routledge. Della Porta, D. (2008). 11 Comparative analysis: Case-oriented versus variableoriented research. In D. Della Porta, & M. Keating (Eds.), Approaches and methodologies in the social sciences (pp. 198–222). Cambridge University Press. Dhakal, T. R. (2014). Institutionalization and use of evaluations in the public sector in Nepal. Journal of MultiDisciplinary Evaluation, 10(23), 51–58. Diwakar, Y., De Mel, R. L., & Samarsinghe, C. H. (2021). A study on the status of national evaluation policies and systems in Asia Pacific region. Asia Pacific Evaluation Association. Diwakar, Y., De Mel, R., Samarsinghe, H., Liyanagamage, M., Mahmodi, S., & Kalugampitiya, A. (2022). A study on the status of national evaluation policies and systems in the Asia Pacific Region—2022. APEA. https://asiapacificeval. org/programs/regional-evaluation-strategy/promoting-national-evaluationupdates/. Accessed on 13 March 2023. Drori, G. S., Höllerer, M. A., & Walgenbach, P. (Eds.). (2014). Global themes and local variations in organization and management. Routlege. Drori, G. S., & Preminger, B. (2018). Between institutional theory and Lefebvre: Sensemaking, logics and enactment of, and in, space. In K. Dale, S. F. Kingma, & V. Wasserman (Eds.), Organizational space and beyond (pp. 104–130). Routledge. Ebbinghaus, B. (2005). When less is more: Selection problems in large-N and small-N cross-national comparisons. International Sociology, 20(2), 133–152. Ebener, A. (2008). Institutional evolution and the political economy of governance. In A. Ebener, & N. Beck (Eds.), The institutions of the market: Organisations, social systems, and governance (pp. 287–308). Oxford University Press. Edwards, M. (2020). Civil society (4th ed). Polity Press. Elder, J. W. (1976). Comparative cross-national methodology. Annual Review of Sociology, 2(1), 209–230. Esser, H. (2002). Soziologie: Institutionen (Vol. 5). Campus Verlag. Etzkowitz, H. (2003). Research groups as ‘quasi-firms’: The invention of the entrepreneurial university. Research Policy, 32(1), 109–121. Etzkowitz, H. (2004). The evolution of the entrepreneurial university. International Journal of Technology and Globalisation, 1(1), 64–77. Etzkowitz, H. (2013). Anatomy of the entrepreneurial university. Social Science Information, 52(3), 486–511.

52

N. ZIERKE ET AL.

Etzkowitz, H., & Leydesdorff, L. (1998). The endless transition: A “Triple Helix” of university-industry-government relations: Introduction. Minerva, 36(3), 203–208. Etzkowitz, H., & Leydesdorff, L. (2000). The dynamics of innovation: From national systems and “Mode 2” to a Triple Helix of university–industry– government relations. Research Policy, 29(2), 109–123. Fakis, A., Hilliam, R., Stoneley, H., & Townend, M. (2014). Quantitative analysis of qualitative information from interviews: A systematic literature review. Journal of Mixed Methods Research, 8(2), 139–161. Fetterman, D. M. (2023). Empowerment evaluation and social justice. Routledge. Forbrig, J. (2002). The nexus between civil society and democracy suggesting a critical approach. In W. Reichel (Ed.), Political priorities between East and West: Europe’s rediscovered wealth—What the accession-candidates in Eastern and Central Europe have to offer (pp. 79–103). Institute for the Danube Region and Central Europe. Freidson, E. (2001). Professionalism: The third logic. Polity Press. Friedman, A., & Phillips, M. (2004). Balancing strategy and accountability: A model for the governance of professional associations. Nonprofit Management and Leadership, 15(2), 187–204. Furubo, J. E., Rist, R. C., & Sandahl, R. (2002). International atlas of evaluation. Transaction Publishers. Giddens, A. (1990). Consequences of modernity. Polity Press. Goerres, A., Siewert, M. B., & Wagemann, C. (2019). Internationally comparative research designs in the social sciences: Fundamental issues, case selection logics, and research limitations. KZfSS Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 71(1), 75–97. Goodnow, F. J. (2003). Politics and administration: A study in government. Routledge. Greenwood, R., Suddaby, R., & Hinings, C. R. (2017). Theorizing change: The role of professional associations in the transformation of institutionalised fields. Academy of Management Journal, 45(1), 58–80. Guerrero, M., & Urbano, D. (2012). The development of an entrepreneurial university. The Journal of Technology Transfer, 37 , 43–74. Hall, P. A., & Taylor, R. C. R. (1996). Political science and the three new institutionalisms. Political Studies, 44(5), 936–957. Han, Y. (2019). The impact of accountability deficit on agency performance: Performance-accountability regime. Public Management Review, 22(6), 927– 948. Hanberger, A. (2001). Policy and program evaluation, civil society, and democracy. American Journal of Evaluation, 22(2), 211–228. Hantrais, L. (2008). International comparative research: Theory, methods and practice. Palgrave Macmillan.

1

INTRODUCTION

53

Harkness, J. A. (2012). Comparative survey research: Goals and challenges. In E. D. de Leeuw, J. Hox, & D. Dillman (Eds.), International handbook of survey methodology (pp. 56–77). Routledge. Hay, K. (2010). Evaluation field building in South Asia: Reflections, anecdotes, and questions. American Journal of Evaluation, 31(2), 222–231. Hay, K. E. (2014). Building the field of evaluation in South Asia: A framework and ideas. In K. E. Hay, & S. Kumar-Range (Eds.), Making evaluation matter: Writings from South Asia (p. 39). Sage. He, L. (2010). Social movement tradition and the role of civil society in Japan and South Korea. East Asia, 27 (3), 267–287. Holcombe, C. (2017). A history of East Asia. Cambridge University Press. Holzer, B. (2006). Spielräume der Weltgesellschaft: Formale Strukturen und Zonen der Informalität. In T. Schwinn (Ed.), Die Vielfalt und Einheit der Moderne: Kultur- und Strukturvergleichende Analysen (pp. 259–279). Springer Link. Holzinger, M. (2014). Fehlschlüsse über die‘Weltgesellschaft’. Kölner Zeitschrift Für Soziologie & Sozialpsychologie, 66(2), 267–289. Hopkins, T. K., & Wallerstein, I. (1967). The comparative study of national societies. Social Science Information, 6(5), 25–58. Howell, J., & Pearce, J. (2001). Civil society and development: A critical exploration. Lynne Rienner. Inglehart, R. (1998). Modernisierung und Postmodernisierung: Kultureller, wirtschaftlicher und politischer Wandel in 43 Gesellschaften. Campus-Verlag. Ingvarson, L. (1998). Professional development as the pursuit of professional standards: The standards-based professional development system. Teaching and Teacher Education, 14(1), 127–140. Jacob, S., Speer, S., & Furubo, J. E. (2015). The institutionalisation of evaluation matters: Updating the international atlas of evaluation 10 years later. Evaluation, 21(1), 6–31. Jeong, M. G., & Oh, S. G. (2010). Searching for participatory governance in Korea. Journal of Contemporary Asia, 40(2), 275–290. Kalm, S., & Uhlin, A. (2015). Civil society and the governance of development: Opposing Global Institutions. Palgrave Macmillan. Kalugampititya, A. (2021). Critical factors for institutionalizing evaluation at national level: Study on four countries in Asia- Sri Lanka, Nepal, Philippines and Bangladesh (PhD Thesis). Department of Sociology, Saarland University, Saarbücken, Germany. Kapitanova, J. (2013). Regeln in Sozialen Systemen. Springer VS. Kassomeh, N. (2021). Revisiting the democratic transition in South Korea and Taiwan. Asian Political Science Review, 5(1), 1–12.

54

N. ZIERKE ET AL.

Kettl, D. F. (2000). The transformation of governance: Globalization, devolution, and the role of government. Public Administration Review, 60(6), 488–497. Kettl, D. F. (2015). The transformation of governance: Public administration for the twenty-first century. JHU Press. Kettl, D. F. (2018). Politics of the administrative process. Sage/CQ Press. Kettl, D. F. (2019). From policy to practice: From ideas to results, from results to trust. Public Administration Review, 79(5), 763–767. Khan, M. A. (1989). A South Asian regional study on current thoughts and practices in monitoring and evaluation. Economic Development Institute of the World Bank. Khan, M. A. (1990). Monitoring and evaluation of development projects in South East Asia, the Experience of Indonesia, Malaysia, the Philippines and Thailand. Economic Development Institute of the World Bank. Khan, M. A. (1998). Evaluation capacity building: An overview of current status, issues and options. Evaluation, 4(3), 310–328. Khan, M. A., de Guzman, G. G., Trinidad, E. A., & Selda, G. G. (1993). Study of government monitoring and evaluation systems: The case of the Philippines. Central Evaluation Office, United Nations Development Program. Kieser, A., & Ebers, M. (Eds.). (2019). Organisationstheorien. Kohlhammer Verlag. Kim, P. S. (2011). Civic engagement, politics and policy in South Korea: Significant developments but a considerable way to go. Public Administration and Development, 31(2), 83–90. Kirschsieper, D., & Brose, H. G. (2014). Un-/Gleichzeitigkeit und Synchronisation. Zum Verhältnis von Diachronie und Synchronie in der Theorie Sozialer Systeme. Zeitschrift für Theoretische Soziologie, 3(2), 172–219. Kiser, L., & Ostrom, E. (1982). The three worlds of action: A metatheoretical synthesis of institutional approaches in strategies of political inquiry. In E. Ostrom (Ed.), Strategies of political inquiry (pp. 172–222). Sage. Kollmorgen, R. (2014). Modernisierungstheoretische Ansätze. In R. Kollmorgen, W. Merkel, & H. J. Wagener (Eds.), Handbuch Transformationsforschung. Springer-Verlag. Krohn, W. (2000). Wissenschaftssoziologie: zwischen Modernisierungstheorie und Sozialkonstruktivismus auf schwankendem epistemischem Boden. Soziologische Revue, 23(Supplement), 314–325. Kroneberg, C. (2014). Frames, scripts, and variable rationality: An integrative theory of action. In G. Manzo (Ed.), Analytical sociology. Actions and Networks (pp. 95–123). Wiley. Larivière, V., Haustein, S., & Mongeon, P. (2015). The oligopoly of academic publishers in the digital era. PLoS ONE, 10(6), 1–15.

1

INTRODUCTION

55

Lauth, H. J., Pickel, G., & Pickel, S. (2014). Vergleich politischer Systeme. Eine Einführung. utb. Laville, J. L., Young, D. R., & Eynaud, P. (Eds.). (2015). Civil society, the third sector and social enterprise: Governance and democracy. Routledge. Lee, Y., & Park, T. (2009). Civil participation in the making of a new regulatory state in Korea: 1998–2008. Korea Observer, 40(3), 461. Lerner, D. (1968). Modernisation: Social aspects. In L. David (Ed.), International encyclopedia of the social sciences (pp. 386–402). Macmillan. Lijphart, A. (1971). Comparative politics and the comparative method. American Political Science Review, 65(3), 682–693. Lijphart, A. (1975). II. The comparable-cases strategy in comparative research. Comparative Political Studies, 8(2), 158–177. Livingstone, S. (2003). On the challenges of cross-national comparative media research. European Journal of Communication, 18(4), 477–500. Livingstone, S. (2013). Challenges to comparative research in a globalizing media landscape. In F. Esser, & T. Hanitzsch (Eds.), The handbook of comparative communication research (pp. 437–451). Routledge. Lovan, W. R., Murray, M., & Shaffer, R. (Eds.). (2003). Participatory governance: Planning, conflict mediation and public decision making in civil society. Routledge. Lowe, D., & Potter, C. (2018). Understanding legislation: A practical guide to statutory interpretation. Bloomsbury Publishing. Luhmann, N. (1981). The improbability of communication. International Social Science Journal, 33(1), 122–132. Luhmann, N. (1983). Legitimation durch Verfahren. Suhrkamp. Luhmann, N. (1988). Soziale Systeme: Grundriß einer allgemeinen Theorie. Suhrkamp. Luhmann, N. (1991). Operational closure and structural coupling: The differentiation of the legal system. Cardozo Law Review, 13, 1419. Luhmann, N. (1992). What is communication? Communication Theory, 2(3), 251–259. Luhmann, N. (1995). Social systems. Stanford University Press. MacDonald, K. M. (1985). Social closure and occupational registration. Sociology, 19(4), 541–556. MacDonald, K. M. (1995). The sociology of the professions. Sage. Mackay, K. (1999). Evaluation capacity development: A diagnostic guide and action framework. World Bank, Operations Evaluation Department. Mainwaring, S., & Pérez-Liñán, A. (2007). Why regions of the world are important: Regional specificities and region-wide diffusion of democracy. In G. Munk (Ed.), Regimes and democracy in Latin America: Theories and methods (pp. 199–229). Oxford University Press.

56

N. ZIERKE ET AL.

Malik, K., & Roth, C. (1999). Evaluation capacity development in Asia. UNDP, NCSTE and The World Bank. March, J. G., & Olsen, J. P. (1983). The new institutionalism: Organizational factors in political life. American Political Science Review, 78(3), 734–749. Marche, J. C., & Olsen, J. P. (1989). Rediscovering institutions. The Organizational Basis of Politics. The Free Press. Maxwell, J. A. (2010). Using numbers in qualitative research. Qualitative Inquiry, 16(6), 475–482. Mayntz, R. (2017). From government to governance: Political steering in modern societies. In D. Scheer, & R. F. Rubik (Eds.), Governance of integrated product policy (pp. 18–25). Routledge. McCowan, T. (2019). The developmental university. In T. McCowan (Ed.), Higher Education for and beyond the sustainable development goals (pp. 91– 114). SpringerLink. McDougall, D. (2016). Asia Pacific in world politics (2nd ed.). Lynne Rienner Publishers. Mehrotra, S. (2013). The government monitoring and evaluation system in India: A work in progress (ECD Working Paper Series, 28). Independent Evaluation Group, The World Bank. https://ieg.worldbankgroup.org/sites/ default/files/Data/reports/ecd_wp28_india_me_0.pdf. Accessed on 23 June 2021. Mertens, D. M. (1998). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches. Sage. Merton, R. K. (1942). The normative structure of science. In R. K. Merton (Ed.), The Sociology of Science: Theoretical and Empirical Investigations (pp. 267–278). University of Chicago Press. Meyer, J. W. (2017). Reflections on institutional theories of organizations. In R. Greenwood, C. Oliver, T. B. Lawrence, & R. E. Meyer (Eds.), The Sage handbook of organizational institutionalism (pp. 831–852). Sage. Meyer, W., Stockmann, R., & Szentmarjay, L. (2022). The institutionalisation of evaluation: Theoretical background, analytical concept and methods. In R. Stockman, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 3–37). Springer International Publishing. Meyer, W., Stockmann, R., & Taube, L. (2020). The institutionalisation of evaluation theoretical background, analytical concept and methods. In R. Stockmann, W. Meyer, & L. Taube (Eds.), The Institutionalisation of evaluation in Europe (pp. 3–34). Palgrave Macmillan. Mill, J. S. (1843). A system of logic (2 Vols). John W. Parker Mok, K. H., & Forrest, R. (Eds.). (2008). Changing governance and public policy in East Asia. Routledge.

1

INTRODUCTION

57

North, D. C. (1986). The new institutional economics. Journal of Institutional and Theoretical Economics (jite)/Zeitschrift für die gesamte Staatswissenschaft, 142(1), 230–237. North, D. C. (1987). Institutions, transaction costs and economic growth. Economic Inquiry, 25(3), 419–428. North, D. C. (1991). Instiutions. Journal of Economic Perspectives, 5(1), 97–112. Ojo, A., & Mellouli, S. (2018). Deploying governance networks for societal challenges. Government Information Quarterly, 35(4), 106–112. Oliver-Lumerman, A., & Drori, G. S. (2021). From Ivory tower to academic commitment and leadership: The changing public mission of universities. Edward Elgar Publishing. Ostrom, E. (2009). The institutional analysis and development framework and the commons. Cornell Law Review, 95, 807. Ostrom, E. (2011). Background on the institutional analysis and development framework. Policy Studies Journal, 39(1), 7–27. Parsons, T. (1939). The professions and social structure. Social Force, 17 (4), 457–467. Parsons, T. (1954). The present position and prospects of systematic theory in sociology. In T. Parsons (Ed.), Essays in sociological theory (pp. 212–237). Free Press. Parsons, T. (1971). Evolutionäre Universalien in der Gesellschaft. In W. Zapf (Eds.), Theorien des sozialen Wandels (pp. 55–74). Kiepenheuer & Witsch. Pattyn, V. R., & Brans, M. (2015). Organisational analytical capacity: Policy evaluation in Belgium. Policy and Society, 34(3–4), 183–196. Payne, A., & Phillips, N. (Eds.). (2014). Handbook of the international political economy of governance. Edward Elgar Publishing. Peel, M., & Twomey, C. (2017). A history of Australia. Bloomsbury Publishing. Peters, B. G. (2019). Institutional theory in political science: The new institutionalism. Bloomsbury Publishing. Pierre, J., & Peters, B. G. (2021). Advanced introduction to governance. Edward Elgar Publishing. Pollack, D. (2016). Modernisierungstheorie – Revised: Entwurf einer Theorie moderner Gesellschaften. Zeitschrift Für Soziologie, 45(4), 219–240. Popper, K. R. (1972). Objective knowledge. Clarendon press. Popper, K. R. (2011 [1945]). The Open Society and its Enemies (First published in two volumes in 1945, 5th Edition 1966). Published in 2011 in Routledge Classics. Routledge. Ragin, C. C. (1991). Introduction: The problem of balancing discourse on cases and variables in comparative social science. In C. C. Ragin (Ed.), Issues and alternatives in comparative social research (pp. 1–8). Brill.

58

N. ZIERKE ET AL.

Ragin, C. C. (2006). How to lure analytic social science out of the doldrums: Some lessons from comparative research. International Sociology, 21(5), 633– 646. Ragin, C. C. (2007). Comparative methods. In W. Outhwaite, & S. P. Turner (Eds.), The Sage handbook of social science methodology (pp. 67–80). Sage. Ragin, C. C. (2014). The comparative method: Moving beyond qualitative and quantitative strategies. University of California Press. Ragin, C. C., & Sonnett, J. (2005). Between complexity and parsimony: Limited diversity, counterfactual cases, and comparative analysis. In S. Kropp, & M. Minkenberg (Eds.), Vergleichen in der Politikwissenschaft (pp. 180–197). VS Verlag für Sozialwissenschaften. Reid, A. (2015). A history of Southeast Asia: Critical crossroads. Wiley. Richardson, A. J. (1997). Social closure in dynamic markets: The incomplete professional project in accountancy. Critical Perspectives on Accounting, 8(6), 635–653. Robins, J. A. (1987). Organizational economics: Notes on the use of transactioncost theory in the study of organizations. Administrative Science Quarterly, 32(1), 68–86. Rosenbaum, A. (2006). Cooperative service delivery: The dynamics of public sector-private sector-civil society collaboration. International Review of Administrative Sciences, 72(1), 43–56. Rosenstein, B. (2013). Mapping the status of national evaluation policies . Parliamentarians Forum on Development Evaluation in South Asia and EvalPartners. http://www.pfde.net/index.php/publications-resources/ global-mapping-report-2015. Accessed on 14 October 2020. Rosenstein, B. (2015). Status of national evaluation policies: Global mapping report (2nd ed.). Parliamentarians Forum on Development Evaluation in South Asia jointly with EvalPartners. https://globalparliamentarianforum. files.wordpress.com/2016/02/the-status-of-evaluation-policies.pdf. Accessed on 7 January 2020. Rosenstein, B., & Kalugampitiya, A. (2021). Global mapping of the status of national evaluation policies (3rd ed.). Global Evaluation Initiative. Roy, R. F., & Wijayasuriya, P. M. W. (1992). A study of government monitoring and evaluation systems: The case of Sri Lanka. Central Evaluation Office, United Nations Development Program. Rüschemeyer, D. (1983). Professional autonomy and the social control of expertise. In R. Dingwall, & P. Lewis (Eds.), The sociology of the professions: Lawyers, doctors and others (pp. 38–58). Macmillan St. Martin’s Press. Samaratunge, R., Alam, Q., & Teicher, J. (2008). The new public management reforms in Asia: A comparison of South and Southeast Asian countries. International Review of Administrative Sciences, 74(1), 25–46.

1

INTRODUCTION

59

Sandelowski, M., Voils, C. I., & Knafl, G. (2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208–222. Schiele, B., Claessens, M., & Shi, S. (Eds.). (2012). Science communication in the world: Practices, theories and trends. Springer Science & Business Media. Schimank, U. (1988). Gesellschaftliche Teilsysteme als Akteurfiktionen. Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 40, 619–639. Schimank, U. (1996). Theorien gesellschaftlicher Differenzierung. Springer-Verlag. Schimank, U. (2001). Teilsysteminterdependenzen und Inklusionsverhältnisse. Ein differenzierungstheoretisches Forschungsprogramm zur System- und Sozialintegration moderner Gesellschaft. In E. Barlösius, H. Müller, & S. Sigmund (Eds.), Gesellschaftsbilder im Umbruch. Soziologische Perspektiven in Deutschland (pp. 109–130). Springer-Verlag. Schimank, U. (2006). Teilsystemische Autonomie und politische Gesellschaftssteuerung – Beiträge zur akteurzentrierten Differenzierungstheorie. 2. Band. VS Verlag. Schimank, U. (2015). Modernity as a functionally differentiated capitalist society: A general theoretical model. European Journal of Social Theory, 18(4), 413– 430. Schimank, U. (2018). Capitalist society: A view from the theory of functional differentiation: Capitalism and labor: Towards critical perspectives. In K. Dörre et al. (Eds.), Capitalism and labor: Towards critical perspectives. International Labour Studies (Vol. 16, pp. 171–184). Campus Verlag. Schimank, U., & Volkmann, U. (2012). Economizing and marketization in a functionally differentiated capitalist society—A theoretical conceptualization. In U. Schimank & U. Volkmann (Eds.), The marketization of society: Economizing the non-economic. Welfare Societies Conference Paper (pp. 37–63). Research Cluster ‘Welfare Societies’. Schoenefeld, J. J., & Jordan, A. J. (2020). Environmental policy evaluation in the EU: Between learning, accountability, and political opportunities. In A. Zito, C. Burns, & A. Lenschow (Eds.), The future of European Union environmental politics and policy (pp. 179–198). Routledge. Schofield, N., & Caballero, G. (Eds.). (2015). The political economy of governance: Institutions, political performance and elections. Springer. Schoonenboom, J., & Johnson, R. B. (2017). How to construct a mixed methods research design. Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 69(Suppl 2), 107–103. Schubert, K., & Klein, M. (2016). Das Politiklexikon. Begriffe, Fakten, Zusammenhänge. Bundeszentrale für politische Bildung. Schwarz, H. P. (2003). Die neueste Zeitgeschichte. Vierteljahresheft für Zeitgeschichte, 51(1), 5–28. Shah, A. (Ed.). (2007). Performance accountability and combating corruption. World Bank Publications.

60

N. ZIERKE ET AL.

Speer, S., Pattyn, V., & De Peuter, B. (2015). The growing role of evaluation in parliaments: Holding governments accountable? International Review of Administrative Sciences, 81(1), 37–57. Stachura, M., Bienfait, A., Albert, G., & Sigmund, S. (Eds.). (2009). Der Sinn der Institutionen: Mehr-Ebenen-und Mehr-Seiten-Analyse. Springer-Verlag. Stern, E. (2009). Evaluation policy in the European Union and its institutions. New Directions for Evaluation, 123, 67–85. Stichweh, R. (1995). Zur Theorie der Weltgesellschaft. Soziale Systeme, 1(1), 29–45. Stichweh, R. (2006). Strukturbildung in der Weltgesellschaft—Die Eigenstrukturen der Weltgesellschaft und die Regionalkulturen der Welt. In T. Schwinn (Ed.), Die Vielfalt und Einheit der Moderne: Kultur- und strukturvergleichende Analysen (pp. 239–257). SpringerLink. Stichweh, R. (2008). Das Konzept der Weltgesellschaft: Genese und Strukturbildung eines globalen Gesellschaftssystems. Rechtstheorie, 39(2–3), 329–355. Stockmann, R. (2008). Evaluation and quality development: Principles of impactbased quality management. Peter Lang. Stockmann, R. (2013). Evaluation in der Entwicklungszusammenarbeit. In J. Wilhelm, & H. Ihne (Eds.), Einführung in die Entwicklungspolitik (pp. 541– 562, 3rd ed.). LIT Verlag. Stockmann, R. (2022). Einführung in die Evaluation. In R. Stockmann (Ed.), Handbuch zur Evaluation. Eine praktische Handlungsanleitung. Sozialwissenschaftliche Evaluationsforschung, Band 16. (2nd ed.) Waxmann. Stockmann, R. (2024). An introduction to evaluation. In R. Stockmann (Ed.), A practitioner handbook on evaluation. (2nd ed.) Edward Elgar. Stockmann, R., & Meyer, W. (2014). Evaluation. Eine Einführung (2nd edition). Waxmann. Stockmann, R., & Meyer, W. (Eds.). (2016). The future of evaluation: Global trends, new challenges, shared perspectives. Palgrave Macmillan. Stockmann, R., Meyer, W., & Szentmarjay, L. (Eds.). (2022). The institutionalisation of evaluation in the Americas. Springer Nature. Stockmann, R., Meyer, W., & Taube, L. (Eds.). (2020). Institutionalisation of evaluation in Europe. Palgrave Macmillan. Tobin, I. (2003). Bureaucratic power and the NPM reforms in Korea. International Review of Public Administration, 8(1), 89–102. Tocqueville, A. (2003). Democracy in America and two essays on America. Penguin. Traugott, M. (2013). Emile Durkheim on institutional analysis. Chicago University Press. Trede, F., Macklin, R., & Bridges, D. (2012). Professional identity development: A review of the higher education literature. Studies in Higher Education, 37 (3), 365–384.

1

INTRODUCTION

61

UNDP (2015). Towards a baseline study: Insights on national evaluation capacities in 43 countries. Independent Evaluation Office and the International Policy Centre for Inclusive Growth. United Nations Development Programme. Van Voorst, S., Zwaan, P., & Schoenefeld, J. J. (2022). Policy evaluation: An evolving and expanding feature of EU governance. In P. R. Graziano, & J. Tosun (Eds.), Elgar encyclopedia of European Union Public Policy (pp. 595– 603). Edward Elgar Publishing. Varone, F., Bundi, P., & Gava, R. (2020). Policy evaluation in parliament: Interest groups as catalysts. International Review of Administrative Sciences, 86(1), 98–114. Vedung, E. (2010). Four waves of evaluation diffusion. Evaluation, 16(3), 263– 277. Vogl, S. (2017). Quantifizierung. KZfSS Kölner Zeitschrift Für Soziologie Und Sozialpsychologie, 69(Suppl 2), 287–312. Weingart, P. (1997). From “Finalization” to “Mode 2”: Old wine in new bottles? Social Science Information, 36(4), 591–613. Weng, B. S. (2010). A short history of Taiwan’s democracy movement. In B. Bridges, & L. Sang Ho (Eds.), Public governance in Asia and the limits of electoral democracy (pp. 115–146). Edward Elgar Publishing. Wollmann, H. (2017). Policy evaluation and evaluation research. In F. Fischer, & G. J. Miller (Eds.), Handbook of public policy analysis (pp. 419–428). Routledge. Wong, J. (2014). Comparing beyond Europe and North America. In I. Engeli, & A. C. Rothmayr (Eds.), Comparative policy studies: Conceptual and methodological challenges (pp. 163–184). Palgrave Macmillan. Zapf, W. (Ed.). (1991). Die Modernisierung moderner Gesellschaften: Verhandlungen des 25. Deutschen Soziologentages in Frankfurt am Main 1990 (Vol. 25). Campus Verlag. Zapf, W. (1996). Die Modernisierungstheorie und unterschiedliche Pfade der gesellschaftlichen Entwicklung. Leviathan, 24(1), 63–77.

PART II

National Developments

CHAPTER 2

Evaluation in Australia Brad Astbury and Scott Bayley

While evaluation is a global phenomenon, the way it is defined, conducted and used varies considerably within and across countries. In line with other chapters in this volume, we examine the institutionalisation of evaluation in Australia from a systems perspective, focusing on how political, social, and professional systems have shaped evaluation practice. Our analysis begins with some scene settings. We provide a general country overview and a brief history of evaluation in Australia. Next, we consider institutional structures and processes (political system), including national legislative and regulatory requirements for

The original version of the chapter has been revised: Errors have been corrected. A correction to this chapter can be found at https://doi.org/10.1007/978-3-031-36918-6_17 B. Astbury (B) Evaluation Melbourne School of Population and Global Health, University of Melbourne, Melbourne, VIC, Australia e-mail: [email protected] S. Bayley Scott Bayley Evauation Services, Canberra, ACT, Australia © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023, corrected publication 2024 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_2

65

66

B. ASTBURY AND S. BAYLEY

evaluation and the influence of these on embedding programme evaluation across the Australian Public Service (APS). We then turn to the social system and explore the use of evaluations by civil society as well as public perceptions and engagement with evaluation. Finally, we discuss the system of evaluation professionalisation and offer some concluding remarks on the future outlook for evaluation in the ‘Land Down Under’.

1

General Country Overview

Australia lies in the southern hemisphere and is the world’s sixth-largest country by area. It is also the driest inhabited continent on earth with a population of approximately 26 million citizens who live mainly in urbanised cities on the eastern seaboard. Economically, Australia is a highly developed country that ranks well on almost all global indicators relating to life expectancy, per capita income, health, education, political freedom and liveability (Department of Foreign Affairs and Trade, 2021). However, there are significant individual, family, and community differences in quality of life resulting from deeply entrenched social, cultural, and economic inequalities. In particular, Indigenous Australians experience higher rates of poor health, poverty, inadequate housing, employment rates and other social and health problems relative to non-Indigenous Australians (Mitrou et al., 2014). Australia is a representative democracy and constitutional monarchy that has inherited strong elements of the British Westminster political system, including the separation of powers between three main branches of government. The legislative branch is a bicameral parliament responsible for making laws and comprises a House of Representatives and a federal Senate. The executive branch of the Prime Minister, Cabinet, Ministers, government departments, statutory authorities and public servants are responsible for the administration and enforcement of the law. Finally, the judicial branch includes the High Court of Australia and other federal courts that interpret and apply the law. This division of power provides a system of checks and balances and is a cornerstone of Australian liberal democracy (Davis et al., 1993). The Commonwealth of Australia is a federation comprising three tiers of government, including the federal parliament, six state and two territory parliaments, and local councils. The federal parliament is responsible for national issues that affect all Australians. State and territory jurisdictions function under the Australian constitution with the authority to legislate on matters such as education, criminal and civil law, public administration, policing, health, and transport. Local councils are responsible for rubbish collection, parks, recreational facilities and social and

2

EVALUATION IN AUSTRALIA

67

community services. There are distinctive evaluation functions and practices across these levels of government, as well as diverse theoretical influences that have shaped evaluation culture (McTaggart et al., 1991; Rogers & Davidson, 2013). A detailed analysis of these is beyond the scope of this chapter. Accordingly, the focus lies on describing the state of evaluation at the national-federal level. 1.1

Brief History of Evaluation

There is a long-standing tradition of evaluation in Australia, dating back to at least the mid-1970s. A watershed moment came in 1976, following the Royal Commission on Australian Government Administration, referred to as the Coombs Report. This sowed the seed for change, highlighting the need for more responsible administration of public monies through performance audit and evaluation (Sharp, 2003). Three years later, another public inquiry was released. The 1979 Baume Report entitled ‘Through a glass darkly’ noted the lack of formal evaluation of social programmes and further paved the way for greater commitment to evaluating government programmes. During the 1980s, a series of changes to the operation of public service was introduced with the goal of improving the appropriateness, efficiency, and effectiveness of government activity. A key influence on formalisation of evaluation was the shift towards New Public Management (NPM), which introduced programme planning and budgeting systems that integrated evaluation within the policy cycle. Under the Financial Management Improvement Program (FMIP) and reforms collectively labelled as ‘Managing for Results (MFR)’ a Commonwealth Government evaluation strategy was established in 1988. The main elements of the strategy included approval of agency evaluation plans, a requirement that every programme be evaluated every three to five years, provisions for evaluation plans to be included in all new policy proposals, and publishing of evaluation reports (Graves et al., 2021; Keating, 1990; Mackay, 1992). To further strengthen evaluation across departments, a range of capacity building activities were delivered by a central evaluation unit within the Department of Finance. This included: the provision of guidance materials and training; advice and support on individual evaluations; a register of published evaluation reports; and coordination of an informal network of evaluators. Mackay and Robertson (1994) report that the evaluation strategy led to an increased impact of evaluation findings on

68

B. ASTBURY AND S. BAYLEY

government budgets for new policy proposals and informed Cabinet decision-making. This golden age of evaluation institutionalisation in Australia did not last. The focus on programme evaluation within the public service began to decline in 1997. The new Howard Liberal/National Government abolished many of the reforms of the previous decade, repealing mandatory requirements for evaluation in the policy planning process and significantly reducing evaluation budgets. This resulted in fewer evaluations being conducted and de-skilling of staff in terms of evaluation capability (Graves, 2020; Gray & Bray, 2019; Mackay, 2011). Since 2010, a renewed commitment to performance reporting and evaluation has occurred, especially following enactment of the Public Governance, Performance and Accountability (PGPA) Act 2013 and introduction of the Commonwealth Performance Framework on the 1st of July 2015. The PGPA Act and performance framework have triggered reforms to evaluation systems and processes that are taking shape across government (Morton & Cook, 2018). The next section discusses contemporary institutional structures and national evaluation regulations in more detail and examines their influence within the Australian Public Service (APS) and policy fields where evaluations are conducted and used most frequently.

2

Institutional Structures and Processes 2.1

Evaluation Regulations

The enhanced Commonwealth Performance Framework and PGPA Act seek to enable the Australian federal parliament and public to understand the use of public resources, whether Commonwealth agencies are achieving their purposes, and who is benefitting from Commonwealth activity. Under the framework, all Commonwealth agencies must report on how performance will be measured and assessed, considering the operating environment, organisational capability and risks (Department of Finance, 2021). The framework requires a shift away from over-reliance on input and output-oriented performance measures. Organisational performance planning and reporting should establish explicit links between activities and outcomes achieved and provide meaningful information with a clear line of sight between planned and actual performance. This allows parliament

2

EVALUATION IN AUSTRALIA

69

and the general public to assess the extent to which intended results were achieved and the factors that affected performance. There is a strong role for evaluators in helping Commonwealth agencies and departments make this crucial adjustment to an outcome-based reporting system (Morton & Cook, 2018). It is important to note that the framework is not focused on enabling performance-based budgeting or results-based management. Rather, the intent of the framework is to provide robust information as one of several inputs used to inform the allocation of public resources by parliament. In this regard, the enhanced framework has similar objectives to the United States Government Performance and Results Act 1993. Furthermore, the framework aims to be ‘more than a system of rules and centrally mandated reporting processes’ (Morton & Cook, 2018, p. 147). It is meant to generate evidence that drives meaningful dialogue between Commonwealth agencies and their stakeholders (for example, the Parliament and the public). This represents a significant change in how performance information was considered before the performance framework and PGPA Act commenced. Previously, dissemination of performance information was largely ad hoc, via budget statements and annual reports. Scrutiny of information in parliament was sporadic and concentrated on specific programmes or activity areas, rather than what was being achieved overall by a government department or agency. According to Morton and Cook (2018) there are some indications that the federal parliament is beginning to interact with the performance framework and value what it aims to achieve. For example, the Joint Committee of Public Accounts and Audit (JCPAA, 2017) have broadly endorsed the direction being taken to institutionalise performance reporting requirements across the government. Reflecting on the experience of introducing annual performance statements, the JCPAA encouraged public officials to strongly engage with agency performance statements and emphasised the need for “greater use of case studies and narratives to build on quantitative reporting of outputs, as a means of demonstrating outcomes” (Morton & Cook, 2018, p. 148). However, implementation of the performance framework and PGPA Act has not been without challenges. In September 2019, the government commissioned an independent review of the federal public sector, referred to as the Thodey Review (Alexander & Thodey, 2018). This wide-ranging report found that additional funding was required for

70

B. ASTBURY AND S. BAYLEY

service-wide capability investments to generate better outcomes for Australians. If the APS is to be a trusted and valued source of strategic policy advice to the Government, the long-term decline in research and evaluation expertise needed to be reversed. Overall spending on learning and development was low and dedicated funding is required for a concerted effort to build research, policy and evaluation skills of public service employees. The review also found that the public service needed to optimise use of data in decision-making, improve and tailor services, facilitate risk-based regulation, and conduct targeted evaluations. This will allow the APS to improve the efficiency and effectiveness of policies and service delivery. Research commissioned for the review also noted that deficiencies in the approach to evaluation are not solely due to a lack of skills, but ‘also a product of cultural practices that have evolved within the APS, and of the environment in which the APS operates’ (Bray et al., 2019, p. 6). In relation to evaluation, the Thodey review recommended that there was a need to: . embed a culture of evaluation and learning from experience to underpin evidence-based policy and delivery, . develop an APS-wide approach to build evaluation capability and ensure systematic evaluation of programmes and policies, . establish a central enabling evaluation function to support APS evaluation practices and expertise, . develop in-house evaluation functions and annual plans and publish evaluations, unless exempt by the Cabinet, and . amend budget requirements to establish a systematic approach for formal evaluations (pp. 34–37). In its response to the Thodey review, the Australian Government partly agreed with these recommendations and undertook for the Department of Finance to establish a small team to help build evaluation expertise and practices. This team has been tasked with leveraging evaluation knowledge that already exists in many agencies, with other agencies to review and boost their own evaluation capabilities. In December 2021, the Department of Finance published an evaluation policy and toolkit on its website to further support implementation of the Commonwealth Performance Framework, requirements of the

2

EVALUATION IN AUSTRALIA

71

PGPA Act and recommendations from the Thodey review. The policy and toolkit include guidance materials on topics such as: defining the nature and purposes of evaluation, steps in planning and undertaking an evaluation, choosing evaluation approaches, principles and standards, building an evaluation culture, and embedding evaluative thinking into everyday practice.1 From 2022, an introductory evaluation module on the toolkit and framework is being offered through the newly established APS Academy, to support learning and development, with a focus on leadership, integrity, and core public service skills. 2.2

Parliamentarian and National Audit Structures

Australia’s parliamentary system has several mechanisms for undertaking reviews of policies and programmes.2 The Senate (the Upper House) delegates a range of tasks to its committees as does the House of Representatives (the Lower House). Some committees are limited only to members of the Senate or to members of the House of representatives while other committees have joint membership. These committees comprise a mixture of both ongoing and temporary assemblies that examine a wide range of issues. Examples include: public accounts and audit; the scrutiny of legislation; trade and investment; rural and regional affairs; through to matters such as education and employment; Covid-19; autism; aquaculture; and social media and safety. Parliamentary committees have the flexibility to meet in numerous locations, hear evidence directly as well as receive written submissions. There is scope to conduct many inquiries simultaneously allowing different issues to be investigated with findings informing government activity on a regular basis. A key strength of parliamentary committees is that they are open and transparent, enabling individuals and organisations to express their views publicly, and ensure these are considered as part of the evidence base that informs policy and decision-making process. As of 1 Available at https://www.finance.gov.au/government/managing-commonwealth-res ources/planning-and-reporting/commonwealth-performance-framework/evaluation-com monwealth-rmg-130. Accessed on 27 January 2022. 2 While there is a research office in Parliament to support to Minister’s requests for evidence there is not an independent evaluation unit in addition to Parliamentary committees.

72

B. ASTBURY AND S. BAYLEY

October 2022, there were 76 public inquiries underway across the Senate and House of Representative committees. In Australia, there is also the option of undertaking a type of public inquiry known as a ‘Royal Commission’. This is the highest form of inquiry on matters of public importance that are only established in exceptional circumstances. A Royal Commission has broad powers to gather information to assist with its inquiry, including summoning witnesses and requesting individuals or organisations to provide documents as evidence. Royal Commissions have terms of reference that set out the key areas of investigation as well as the timeline by which the inquiry must be completed. Australia also has a national Auditor-General who is an independent officer of the Parliament with responsibility under the Auditor-General Act 1997 for auditing Commonwealth entities and reporting to the Australian Parliament. The Auditor-General is supported by the Australian National Audit Office (ANAO). In the 2021–2022 financial year, the ANAO published 40 performance audits as well as completed financial audits of the whole of government and specific government sectors. The Auditor-General’s functions include: . auditing the financial statements of Commonwealth entities, Commonwealth companies and their subsidiaries, . auditing annual performance statements of Commonwealth entities, . conducting performance audits of an entity’s operations and administrative support systems, assurance reviews, . auditing the performance measures of Commonwealth entities, Commonwealth companies and their subsidiaries, and . reporting directly to the Parliament on any matter or to a minister on any important matter. Since 2016, there have been some discussions in the evaluation community about the need for an ‘Evaluator-General’ in Australia (Gruen, 2016, 2018). The Evaluator-General would be an independent statutory agency with investigative and reporting powers like the AuditorGeneral, though in the area of monitoring and evaluation rather than audit. Its existence would also help to promote the profession of evaluation which has a lower professional status and visibility than accounting, economics, and performance audit.

2

EVALUATION IN AUSTRALIA

73

The Evaluator-General’s role would involve more than sitting above government monitoring and evaluation systems. It would include the authority to direct and provide resources for evaluation within Commonwealth agencies and comprise a team of staff that collaborate closely with agencies and service providers to design and deliver evaluations. To maintain independence, the Evaluator-General would have ultimate responsibility for the conduct of an evaluation. A key function would be ensuring reports were available to service providers to support performance improvement, as well as disseminating findings to the public along with commentary and analysis. While many in the evaluation community see merit in debating this proposal, at the time of writing the issue has received little attention from the Australian government. Instead, an internal government unit called the Australian Centre for Evaluation was created and became operational on the 1st July, 2023. It is expected that the new central unit will conduct a small number of evaluations each year, with a focus on randomised trials, and partner with government agencies to lift evaluation capability and conduct impact evaluations on agreed priorities. 2.3

Organisational Structure

Australia has a number of specialised public agencies that are funded to carry out evaluations. For example, the Productivity Commission is an advisory agency of the Australian Government, located within the Treasury portfolio. The Commission does not administer government programmes or exercise executive power. The core function is to conduct public inquiries on behalf of the Australian Government on key policy issues bearing on Australia’s economic performance and community wellbeing. The Commission has had a significant impact on policy and programme reforms over a number of years. While there are a range of government institutes that carry out evaluation-related functions, two in particular are worth mentioning given their long history. The Australian Institute of Criminology (AIC) is a national research centre on crime and justice that assembles trend data and disseminates research, evaluations, and policy advice. The AIC has been operating since 1973. The AIC informs criminal justice policy and practice by funding and carrying out policy research of national significance and building an evidence-based about what works and why in relation to policing, courts, corrections and other arms of the justice system. The AIC works with national and international stakeholders to produce new insights, drawing on qualitative, quantitative, and mixed methods data.

74

B. ASTBURY AND S. BAYLEY

The AIC draws upon information provided by the Commonwealth and state and territory governments to undertake a range of its research and evaluation activities. The Australian Institute of Health and Welfare (AIHW) is an independent statutory Australian Government agency with considerable experience working with health and welfare data since its establishment in 1987. AIHW’s role is to provide information and statistics for the benefit of the Australian people by obtaining, retrieving, and analysing data on a range of public health and general welfare issues and producing information and reports that can be used to improve decision-making around policy development and programme service delivery. In addition to these specialised agencies, many Commonwealth government departments also carry out evaluations. For example, the departments responsible for health, education and employment, industry, science and energy, international development, and Indigenous affairs all have internal evaluation units. These units contract out evaluations to private sector consulting firms, while also undertaking some evaluations in-house. In contrast to the positive assessment of progress by Morton and Cook (2018),3 Gray and Bray (2019) raise a number of concerns about the state of evaluation practice in Australia, including the capacity of agencies to undertake evaluations, the limited number of long-term evaluations, and the lack of an evidence base to inform key areas of policy. Based on their consultations with current and former senior public servants, Gray and Bray (2019), identify specific issues regarding the quality of outcome evaluation, the extent of evaluation, and whether evaluation effort is being appropriately targeted. Indeed, across most of their discussions, a highly negative assessment of evaluation institutionalisation within the APS emerged. There was a general view that evaluations were often not valuable due to their narrow focus, weak methodology, and limited data. Respondents also reported a lack of independence with a general “bias towards showing program ‘success’ for fear of causing embarrassment to government” (Gray & Bray, 2019, p. 13). Although these current and former senior public servants strongly supported the conduct of impact and economic evaluations, the

3 The authors were employed at the time by the Department of Finance, which leads implementation of the Commonwealth Performance Framework. This may partly explain the general positivity expressed about the framework and its influence on the public service.

2

EVALUATION IN AUSTRALIA

75

need for ongoing programme monitoring and timely formative evaluation findings to support continuous improvement of programmes was also highlighted. To address these limitations, there was some consensus that greater centralisation within Commonwealth agencies and across government was required. The Department of Finance was mentioned most often as the agency best suited to lead the whole of government coordination. Even so, it was emphasised that buy-in from agencies was critical to foster an authentic commitment to evaluation, as otherwise there was a risk that central administration of evaluation could have perverse effects, such as promoting ‘box ticking’ ritualistic compliance. 2.4

Evaluation Practice and Use

The frequency and extent of evaluation activity across different policy sectors in Australia ebbs and flows, largely in line with government investment in reform activities. This is because increases in expenditure relating to new policies often include mandatory budget allocations for evaluation. Traditionally, areas of stronger focus for evaluation include health, education, justice, employment, and international development. As discussed below, the civil society sector has been slower to consider ways in which evaluation can be facilitated and embedded. There are no dominant types of evaluation that prevail in Australia. Evaluation commissioners generally refer to basic distinctions between process, outcome and economic evaluation and advocate mixed method data collection approaches. It is unusual for published evaluations to discuss the role of evaluation theories or models, but tools associated with theory-based evaluation such as logic models are prolific and have become highly routinised in most areas of evaluation practice. Methodological appropriateness holds sway as a guiding principle, although there has been some debate in recent years about the need to enhance the general quality and methodological rigour of evaluation studies. For some, this means increasing the application of scientific research designs, such as randomised controlled trials or strong quasiexperiments using statistically matched comparison groups. For others this means tailoring evaluations to suit purpose, context, budget, time, data constraints, and decision-making needs of key intended users. Quality assurance of evaluation, for example through commissioning of independent meta-evaluations is not common practice.

76

B. ASTBURY AND S. BAYLEY

Australian evaluators lament the lack of use or ‘knowledge translation’ of evaluation reports, echoing the meta-narrative around use that has been occurring since the profession emerged in the 1960s. In one of the few empirical studies undertaken in the Australian context, Maloney (2017) investigated Australian Evaluation Society (AES) members’ perceptions of the levels of use of evaluation and the factors associated with use, as well as how evaluators overcome barriers to use. AES members identified non-use of evaluation as a substantial worry, consistent with the broader evaluation literature. However, they also reported examples of evaluations that had resulted in both direct, instrumental use of findings as well as indirect, conceptual use through engagement in the evaluation process. Use of evaluations also varied based on the type and level of stakeholders in an organisation. For example, frontline staff may apply findings to improve programme operations while the political context prevents senior leadership from officially endorsing and applying lessons from evaluations in their management response. AES members identified a diverse range of factors as influential in determining evaluation use. Maloney (2017) found that these largely mirror those reported through empirical studies and synthesises of research on evaluation utilisation from the United States and Europe. Also consistent with prior literature, factors could be grouped accorded to demand and supply-side issues. In relation to the former, leadership commitment and receptiveness to evaluation was a significant enabler. In terms of the supply side, agreeing on the purpose and communicating findings effectively were seen to be particularly important in the Australian context. AES interviewees described a range of strategies that they employ to promote evaluation use. While only half said that these actions are informed by theory, all take steps that are generally aligned with Patton’s (2013) Utilisation-Focused Evaluation checklist. Many interviewees reported applying the same broad approach to facilitating use across different contexts, although how this looked in practice varied because their approach involved tailoring plans in line with stakeholders’ needs. Common tactics to support use included: . assisting in the programme design phase and working with intended users to agree on the purpose of the evaluation upfront,

2

EVALUATION IN AUSTRALIA

77

. engaging stakeholders to ensure the evaluation addresses needs and fits the context, as well as building ownership of the evaluation, . communicating findings and sharing emerging results so these can be used as the programme is being implemented or so that stakeholders are better prepared to act on the findings presented in reports, . tailoring delivery approaches to the organisation and ensuring recommendations are feasible and appropriate for the context, and . engaging in follow-up activities to promote the use and support development of an evaluation culture or mindset. As Maloney (2017) argues, given that the broad range of factors affecting evaluation use are now well documented, what evaluators need are context-based models of evaluation influence that unpack interactions leading to higher levels of use at individual, interpersonal and collective levels. Evaluators would also benefit from guidance on how to position their work for adoption in the argumentation process that is central to policy-making in a deliberative democracy, such as Australia.

3

Societal Dissemination and Acceptance of Evaluation in Civil Society

In Australia, the term civil society or ‘third sector’ refers broadly to a diverse range of social and community services delivered by not-for-profit agencies (NFP), non-government organisations (NGOs), and charities. Civil society organisations vary greatly in size, purpose, governance and operations, and can include a focus on health, education, sports, cultural and religious activities, unions, environment, social services, recreation, research, think tanks, neighbourhood associations and housing, to name but a few areas of activity. 3.1

Use of Evaluations by Civil Society

Demand for services provided by civil society organisations continues to grow, as do accountability requirements that have intensified as a result of recent reforms to the regulatory framework for the Australian NFP sector. For example, the Productivity Commission (2010) inquiry into the Contribution of the Not-for Profit Sector and introduction of legislative reporting requirements under the Australian Charities and Not-for-profits

78

B. ASTBURY AND S. BAYLEY

Commission Act 2012 have placed more pressure on the sector to measure performance and demonstrate effectiveness. The Australian Charities and Not-for-Profits Commission (ACNC), the principal regulator of charities at the Commonwealth level, reports that approximately 600,000 NFPs are operating in Australia. Of these, around 51,000 are registered. Data provided by the ACNC (2021) on registered charities indicates that in the 2019 reporting year, revenue grew by 6.8%, with total revenue of $166 billion. About half of the NFPs and charities in Australia use volunteers and unpaid staff, although are also significant employers, constituting approximately 11% of total employment in Australia. Given the size and importance of the civil society sector in Australia, it is surprising that so few studies exist on the state of evaluation practice and use in NGOs and NFPs. The available evidence suggests that the level and quality of evaluation varies markedly from limited attention in the majority of small to medium-size Australian NGOs, to a high level of evaluation activity in large domestic and internationally focused NGOs (Neesham et al., 2017; Rogers et al., 2019). Responding to government requirements for evidence of effectiveness and value for money remains challenging for Australian civil society organisations. This is due to the diversity and complexity of interventions delivered, short funding cycles, resource limitations and multiple funders with different expectations and reporting requirements. Additional difficulties arise in measuring and attributing longer-term outcomes targeted by NGOs, who often deliver services in collaboration with multiple agencies to promote collective impact (Kelly, 2021). Empirical studies of evaluation processes in Australian NGOs have found that many agencies have a strong interest in evaluation capacity building and the application of social impact measurement techniques to strengthen evaluative knowledge and skills, support organisational learning and demonstrate achievements. However, external accountability pressures have often emphasised compliance to standardised measurement frameworks, rather than comprehensive, tailored evaluations. As a result, efforts to embed evaluation have met with varying levels of success, and in some organisations evaluation is viewed as a symbolic exercise that is more a burden than a benefit to programme and organisational improvement (Barraket & Yousefpour, 2013; Doherty et al., 2015; McCoy et al., 2013).

2

EVALUATION IN AUSTRALIA

79

Little is known about the extent to which citizens or community service organisations use evaluations in Australia for public debate and shaping public opinion. A recent qualitative study by Williamson and Kingston (2021) found that a common motivation for evaluation among leaders of public philanthropies is the importance of fostering donor and public confidence in the organisation. This suggests that one important function of evaluation for civil society organisations is to demonstrate credibility and influence public perceptions of effectiveness in order to garner support within a competitive funding landscape. In summary, most civil society agencies in Australia struggle to carry out and use evaluations effectively due to resource, time, and expertise constraints. However, the larger and more well-resourced organisations regularly conduct and publish evaluations to demonstrate legitimacy and win government contracts in an increasingly competitive environment. These ‘super-sized welfare businesses’ have performance monitoring systems that align with metrics required by external agencies, internal evaluation teams, and funding to commission external evaluations that address the evidence needs of government sponsors and corporate donors (Neesham et al., 2017; Onyx et al., 2016). 3.2

Public Perception and Discussion of Evaluation

In contrast with other fields, such as law, accounting, and even market research, the concept of evaluation is not well understood among the general public in Australia. It is common, as in other parts of the world, to hear evaluators in Australia report challenges in describing their work to laypersons (Mason & Hunt, 2019). Evaluators are frequently confused with auditors, social researchers, and management consultants and ‘lack the status, prestige, and autonomy of a profession’ (Picciotto, 2019, p. 94). While there is a general lack of recognition and public understanding of evaluation as a profession with a unique skillset and knowledge base, external evaluation reports or summaries are sometimes made publicly available via government websites and occasionally discussed in newspapers and social media. Politicians also regularly espouse the evidencebased policy mantra, but usually only react to negative evaluation findings or promote positive results when it serves their interests. This typically occurs when evaluations are released on the impact of government interventions relating to controversial topics in the media spotlight.

80

B. ASTBURY AND S. BAYLEY

Several well-known institutions, such as the Auditor-General, Productivity Commission and Grattan Institute have repeatedly called for better evaluation of Australian government programmes, including a focus on services and initiatives to improve outcomes for Indigenous people and communities. As noted earlier, there has also been some discussion over the past few years about the need to transform policy-making by creating an ‘Office of the Evaluator-General’ that would be tasked with conducting rigorous, independent evaluations to ensure that taxpayer funding is only used on initiatives that work (Siminski & Cobb-Clark, 2019). These requests have failed to attract public interest or garner support from the Australian government.4 However, the election of a new government in May 2022 and establishment of an internal unit in Treasury called the Australian Centre for Evaluation may help to raise public awareness of evaluation.5 More generally, it has been observed that public discourse on policy effectiveness in Australia is also regressing in line with a global trend towards populism and sloganeering in a ‘post-truth era’. This term implies that the search for truth is less relevant in determining the value of policy interventions than appeals to emotion and personal belief. While evaluation findings continue to play a role in shaping policy, the influence of facts and rational analysis has been diminished in contemporary Australian politics by reliance on ‘massaging perceptions’ and managing the media with ‘spin doctors [that] craft explanations, plausible or otherwise, for every failed policy’ (Jones, 2020, p. 56).6

4 The reality versus rhetoric of government support for evaluation was illustrated by the unexpected decision to abolish the well-regarded Office for Development Effectiveness and Independent Evaluation Committee that led evaluations on the effectiveness of Australianfunded overseas aid projects (Howes, 2020). 5 The new Labour government made a pre-election commitment to establishing an Evaluator General. See also the October 2022 Independent Review into Australia’s Response to Covid-19 which recommended the establishment of an independent Office of the Evaluator General. The lead author of this report, Peter Shergold, was formerly the most senior official in the APS and his views remain influential in government (Shergold et al., 2022). 6 The threat that ‘post-truth politics’ poses for the evaluation community has been raised elsewhere by Patton (2018), Picciotto (2019) and Schwandt (2019) who defines post-truth politics as ‘a political culture in which debate is framed largely by appeals to emotion disconnected from the details of policy and by the repeated assertion of talking points to which factual rebuttals are ignored’ (p. 320).

2

3.3

EVALUATION IN AUSTRALIA

81

Participation of Civil Society and Demand for Evaluation

It is common for evaluators in Australia to seek the involvement of a wide range of stakeholders, including programme beneficiaries to ensure that evaluations are relevant, ethical, and inclusive. This is partly due to the democratic and egalitarian culture of Australian society as well as influence of participatory and empowerment approaches on evaluation practitioners. There is also growing interest in concepts such as ‘co-design’ and involvement of consumers with ‘lived experience’ in the design and implementation of evaluation processes. While there is variation across Australian government departments and agencies most also espouse the virtues of citizen participation in evaluation. Citizen involvement is considered important to determine the responsive of programmes to end user needs and improve government accountability. Increasingly, evaluation commissioners in areas such as mental health care are stipulating in requests for tender a formal requirement for the evaluation team to include one or more members with lived experience of mental health as a consumer and/or carer. Other agencies have established quality criteria and standards for evaluation that emphasise inclusiveness. For example, the National Indigenous Australians Agency (NIAA) has published an evaluation framework that identifies respectful inclusion of diverse Indigenous perspectives in evaluation as a core principle to ensure findings are meaningful and useful to Indigenous communities (Department of Prime Minister & Cabinet, 2018). Many Australian health authorities, such as Primary Health Networks are also required as part of their constitution to establish a Community Advisory Committee. Members of this committee regularly participate in commissioning, co-designing, and evaluating new service delivery systems to meet the needs of people in their region and address gaps in health care. Finally, while it is rare for individual citizens or organisations to demand evaluations, there is a long civic tradition in Australia of using public inquiries to address issues that attract public attention. For example, gaps in services for people living with a disability or lack of support options for the homeless. Public inquiries include formal participation processes to capture views about the adequacy of government policies from individuals, civil society organisations, community groups, unions, academic experts, private enterprise and various other actors. Examples of different types of public inquiries include Judicial Inquiries,

82

B. ASTBURY AND S. BAYLEY

Ombudsman and other ‘Watchdog’ Inquiries, Royal Commissions, Parliamentary Inquiries, and inquiries carried out by statutory advisory bodies such as the Law Reform Commission and Productivity Commission. Public inquiries can be considered a ‘type’ of evaluation to the extent that a range of evidence is collected through public hearings and submissions, then synthesised to make judgements, with recommendations provided for improving policies and programmes. Topic areas of public inquiries are wide-ranging, covering the full spectrum of public policy issues and at times even focusing on the quality of evaluation in particular sectors.7 Possible factors hindering public demand for evaluation include low awareness of evaluation, mistrust of government and limited efforts to stimulate public interest in evaluation as an important tool for improving the quality of judgments about the value of policies and programmes.

4

Professionalisation of Evaluation

In Australia, evaluation is not an officially accredited profession. There are ethical guidelines and a competency framework, but no formally recognised national standards for determining the quality of evaluations or evaluators. Nevertheless, evaluation practice is relatively mature and includes established higher education and training programmes, a range of professional development courses, a journal, a large marketplace for evaluation consultancies and a professional society with regional networks and special interest groups. 4.1

Academic Study Courses and Training Practices

There is an extensive range of evaluation-related subjects that form part of broader academic courses taught within Australian universities. Undergraduate and honours students are often able to enrol in specific

7 Note: For civil society organisations and individuals it is common to regularly ask

questions about the effectiveness of government interventions (e.g., public health measures in response to COVID, efforts to reduce inflation, climate change policies), this is a general demand for ‘evidence’ that often occurs in interviews for media articles and press releases, rather than a formal request by individuals and/or groups for a specific intervention to be ‘evaluated’.

2

EVALUATION IN AUSTRALIA

83

electives that cover evaluation curriculum from the perspective of assessment in education, policy development and implementation and research design in psychology, sociology, public health, and economics. However, deeper exposure to evaluation theory and practice primarily occurs at the graduate level. The University of Melbourne has offered graduate level courses in evaluation since the early 1980s, led by academic staff at the Centre for Program Evaluation (CPE).8 Development of a specialist universitybased evaluator education programme in Australia occurred in response to the growing need for knowledge and skills among those wanting to commission, conduct and use evaluations in organisational settings (Owen, 2004). The course began in 1981 as the Graduate Diploma in Evaluation and Assessment and included three compulsory subjects and two evaluation projects. In 1997 the course was upgraded to a Postgraduate Diploma in Evaluation and a range of new elective subjects were introduced. In 2000 a Postgraduate Certificate and Masters in Assessment and Evaluation were launched, including online and blended study modes. In 2011 the course was renewed and relaunched as the Master of Evaluation. The CPE currently offers a fully online Master and Graduate Certificate in Evaluation, along with opportunities to undertake a PhD specialising in evaluation. Students completing the Master of Evaluation are required to complete three core subjects in evaluation foundations, practice, and applied research methodology and four elective subjects from a suite of eight that includes debates in evaluation, qualitative and quantitative methods, educational evaluation, mixed methods, evaluation capacity development, impact evaluation and value for money. Students also undertake an evaluation capstone that involves completing an individual project. The course remains the only one of its kind in Australasia and has attracted an increasing number of domestic and international students seeking formal qualifications in the field of evaluation.

8 The Centre for Program Evaluation (CPE), formerly the Tertiary Education Research Unit (TERU), was established in 1975 by Dr Gerald Elsworth at the Melbourne State College. The College became part of the University of Melbourne in 1989. The TERU initially provided internal research and advisory services to the College on course evaluation and assessment of student progress. It was renamed the CPE in 1982 to reflect growing emphasis on externally commissioned evaluations of educational and social programmes.

84

B. ASTBURY AND S. BAYLEY

The Australian Evaluation Society (AES) offers professional development through an annual conference and regular workshops and seminars. In 2021, a Group Mentoring Pilot was launched with initial evaluation findings showing that the programme was beneficial and should be continued. While these training and mentoring opportunities are not formally accredited, they appeal to a range of audiences and provide practical information and advice on how to apply different evaluation approaches and techniques. Over the past few years, the AES has been actively exploring options to strengthen the capacity and professionalisation of the Australian evaluation sector. This includes a greater focus on professional learning that is explicitly linked to evaluator competencies, support for sharing and learning from practices, partnerships with other associations, and public engagement and advocacy (Peersman & Rogers, 2017). In addition to university courses and AES training, a growing number of associations, institutes and consultancies provide short courses in evaluation and research methods for practitioners working in specific sectors such as international development, not-for-profit, business management and public administration. For example, the Australian Consortium for Social and Political Research Inc. (ACSPRI) runs a popular five-day short course on public policy evaluation, the Institute for Public Policy and Governance (IPPG) recently developed a 10-hour online professional development course on getting started in evaluation, the Centre for Social Impact (CSI) provides a two-day outcomes measurement workshop and Clear Horizon consulting launched an online evaluation academy in 2019 that includes courses ranging in length from four to 15 weeks. 4.2

Journals and Communication Platforms

The Evaluation Journal of Australasia (EJA) is a peer-reviewed publication distributed quarterly by the Australian Evaluation Society.9 The first edition of the EJA was issued in 1989.10 The EJA has had a chequered 9 Australian journals in policy areas such as education, criminology, health, public administration, youth studies, employment and international development also regularly publish evaluations studies and methodological papers relating to evaluation. Discussion of journals that do not specialise in evaluation is beyond the scope of the chapter. 10 The EJA was re-launched in 2001, incorporating Evaluation News and Comment, a former magazine style publication that was also produced by the AES from 1992 to

2

EVALUATION IN AUSTRALIA

85

publication history, including three years in which no editions were issued (1999, 2000, and 2004) and two years (1997 and 1998) where journal content consisted entirely of edited versions of papers presented at the annual AES conference, reflecting the difficulties of sourcing scholarly articles and submissions worthy of publication (Fraser, 2001; Wilson, 1999). During the past two decades, there has been regular turnover of editors, limiting the consistency of editorial guidance for the journal. Furthermore, the EJA has only recently been listed in ranking indices11 and has a relatively low profile compared to other evaluation-specific journals. This has affected the capacity to attract original research and articles by academics and researchers for whom citation of articles and acceptance of publication in journals with high impact factors is key for the purpose of building their professional careers (Davis, 2014). The AES Board has identified the need to review the strategic position and direction of the journal. In 2018 an arrangement was secured with SAGE Publications to host the journal on its platform. This has improved the manuscript submission and review system and raised the visibility of articles through enhanced search functionalities. Further changes to EJA governance and resourcing were phased in during 2020, with a new editorial structure and team introduced in December 2021. The AES also supports other communication platforms that provide information and networking opportunities. This includes Regional Networks that deliver a monthly programme of seminars covering contemporary topics and issues in evaluation, Special Interest Groups (SIGs) which are formal groups of members with a common interest in an aspect of evaluation,12 the AES blog and a regular electronic newsletter distributed to all members. Apart from the activities of the AES, there are numerous journals from other disciplines that regularly publish results from evaluations. For

2000. Numbering of the volumes recommenced from Volume 1. The AES also previously published The Bulletin from 1987 to 1989, which was replaced by Evaluation News and Comment. 11 This is because journals published infrequently or less than quarterly do not meet the criteria for inclusion in databases that report journal impact factor (Sellitto, 2009). 12 The following SIGs are currently active: Australian Public Sector Evaluation Network, Design and Evaluation, Emerging Evaluators for Social Justice, Multicultural Evaluation and Systems Evaluation.

86

B. ASTBURY AND S. BAYLEY

example, the Australian and New Zealand Journal of Criminology, the Australian Journal of Primary Health, the Australian Journal of Education, the Australian Journal of Public Administration and Australian Social Work. 4.3

Professional Organisations

The formation of the AES in 1987 grew out of the first National Evaluation Conference in 1982, which provided a catalyst for bringing together individuals from a diverse range of fields and sectors to discuss the ‘state of the art’ in Australia. The mission of the AES, which has remained relatively unchanged over the past 35 years, is to strengthen and promote evaluation practice, theory and use in order to achieve the vision of quality evaluation that makes a difference. The AES is governed by a Board of Directors and managed by a Chief Executive Officer. The main source of revenue is the annual conference, followed by membership fees and professional development activities. Over the period from 2000 to 2021 membership numbers have grown steadily from around 600 to just over 1,000 members. Members are located in all Australian states and territories, with a small proportion of members residing in New Zealand and internationally. Membership is diverse and includes evaluation commissioners, managers and practitioners working in the public and community sector, consultants, academics and teachers and students of evaluation. The AES maintains a directory of members providing commercial evaluation services, a members’ centre with electronic access to resources for members only, and eligibility for nomination to the AES Awards for Excellence in Evaluation. The AES strategic priorities for 2019–2022 include Indigenous and non-Indigenous capacity in culturally safe evaluation, professional and career pathways, meeting diverse membership needs and collaboration and partnerships to strengthen the field of evaluation. These four priority areas are supported by volunteer advisory committees and working groups, with input from AES life members and fellows. While the AES remains the pre-eminent evaluation association in Australia, other member-based associations, and informal networks with an interest in evaluation-related topics include the Institute of Public Administration (IPA), the Social Impact Measurement Network Australia (SIMNA), the Research Society, the Implementation Science Network and Behavioural Insights Practitioners’ Network. In particular, SIMNA,

2

EVALUATION IN AUSTRALIA

87

which was established in 2012 as the Australian affiliate of Social Value International, has over 1,000 members from not-for-profits, businesses, philanthropy, social investment, consulting, corporations and academic organisations. Interest in social impact measurement in Australia seems to be following a global trend towards measuring outcomes in the context of corporate and philanthropic investment initiatives (Vo & Christie, 2018). There is no available research or accurate data on the size or composition of the evaluation industry in Australia. However, the expanding number of evaluation freelancers, consultancies and research institutes who bid for evaluation contracts suggests that government, corporate and not-for-profit expenditure on evaluation services is significant and, in some areas, growing substantially. For example, one government department, the National Indigenous Australians Agency (NIAA), has committed $40 million over four years to support evaluation (Australian National Audit Office, 2019). In addition, most of the large government departments have special panels of preferred evaluation suppliers with multimillion-dollar annual evaluation budgets. Limited information on the evaluation marketplace is not unique to Australia and has been identified as a significant theoretical and empirical gap in the profession’s knowledge base (Nielsen et al., 2018). 4.4

Existence and Compliance with Standards and Quality Obligations

Members of the AES are expected to abide by the Code of Ethical Conduct and the Guidelines for the Ethical Conduct of Evaluations. The code was developed by a working group in response to a complaint from an evaluation client and first released as an Interim Code of Ethics and circulated to AES members in 1992 (Sharp, 2003). The guidelines were released in 1997 and updated along with the code in 2013. These standards are designed to promote ethical evaluation practices, support member rights and maintain the reputation of the AES as a trusted organisation. The code outlines responsibilities that evaluators should uphold to the field of evaluation and to the public (AES, 2013a). For example, AES members should consider the interests of all stakeholders, remain competent and rigorous in their practice, respect confidentiality, be accountable for their performance, and carry out evaluations honestly, fairly and sensitively. The code also sets out a number of responsibilities to the AES and to fellow members, such as using reasonable criticism and not damaging

88

B. ASTBURY AND S. BAYLEY

the professional reputation of others, respecting the privacy and diversity of member interests, and declaring conflicts of interest when making decisions or providing advice to the AES. Where there is a demonstrated breach of the code, the AES Board has the authority to terminate membership in the society. The guidelines are broader in scope and provide a framework for discussing and resolving ethical issues that might arise during the course of an evaluation (AES, 2013b). They cover three main stages of evaluation: commissioning and preparing for an evaluation, conducting an evaluation, and reporting evaluation findings. While the primary focus is evaluators, the guidelines also address the activities of evaluation commissioners and managers, including responsibilities for ensuring that the integrity of evaluation reports is not violated. A survey of AES members found there is reasonable awareness of the code and guidelines among members. It also found that most had experienced ethical dilemmas in their work, typically pressure from commissioners and mangers to influence or control evaluation findings (Turner, 2003). The survey revealed support for establishing quality standards for evaluation. A draft set of standards were developed in 2001 for the Australian context but there has been no commitment from the AES to develop these further or endorse standards used elsewhere.13 Furthermore, there is no requirement for credentialing of evaluators or a formal certification system in Australia. Instead, the AES has opted for an Evaluators’ Professional Learning Competency Framework. The framework was published in 2013 and draws on competencies identified by evaluation associations in New Zealand, Canada and Europe. Seven domains of competence are covered, including evaluative attitude and professional practice, evaluation theory, culture, stakeholders and context, research methods and systematic inquiry, project management, interpersonal skills, and evaluation activities. There is no formal expectation that AES members will be assessed against their competencies. Rather, the framework is intended to be a flexible guide that 13 A number of Australian government agencies have developed in-house quality standards for evaluation. While not explicitly referred to as standards, the Australian government has also recently released a Commonwealth Evaluation Policy that takes effect from the 1st of December 2021. The policy includes five principles for high-quality evaluation, including: (1) fit for purpose, (2) useful, (3) robust, ethical, and culturally appropriate; (4) credible; and (5) transparent where appropriate (Department of Finance, 2021).

2

EVALUATION IN AUSTRALIA

89

encourages individual professional development, informs education and training programmes, guides recruitment or engagement of evaluators and supports the selection of team members when planning evaluations (AES, 2013a, 2013b). Another priority area for professionalisation is cultural capability. In September 2021, the AES launched a First Nations Cultural Safety Framework to strengthen the understanding of principles for culturally safe evaluation14 and encourage a wide variety of evaluation audiences to critically reflect on how they can be culturally accountable in the way that evaluations are designed and carried out. While not a prescription for practice, the framework has been endorsed by the AES as a strategic priority and reflects a strong commitment to ensuring that values relating to cultural safety are considered as an important standard for determining whether an evaluation is high-quality, respectful and ethical (Gollan & Stacey, 2021).

5

Conclusion and Future Outlook

This chapter has examined the influence of political, social and professional contexts on the framing, conduct and use of evaluation in Australia. Care should be exercised in extrapolating findings from our broad sketch of the national picture. This is because a country-wide focus obscures the diversity of evaluation practices within and across different sectors, levels of government, organisations and policy settings. Even so, there are some promising signs that revitalisation of concern towards evaluation institutionalisation is occurring following a period of stagnation during the first decade of the new millennium. On this basis, our outlook for the immediate future of evaluation in Australia is cautiously optimistic. It is anticipated that Commonwealth government agencies will slowly rebuild their evaluation capacities over the next five to 10 years in an ad hoc manner, subject to the shifting priorities of changing federal governments. Large internal evaluation units

14 The framework defines cultural safety as ‘an experience determined by First Nations peoples when they are in situations where their presence is welcome and respected, their experiences are believed and validated, their cultures are centred and valued, their knowledges and skills are recognised and supported, their advice is listened to and acted upon, and they do not experience racism in any form’ (p. 39).

90

B. ASTBURY AND S. BAYLEY

within government departments will remain rare, with most agencies contracting private consulting firms to undertake their evaluations. Published evaluations will concentrate on social, educational and health programmes, emphasising impact and economic studies of large-scale government reforms. Some areas of public expenditure such as defence and assistance to industry will continue to receive limited evaluation coverage. In addition, the Australian media will maintain its reporting on programme failures and conflicts among members of parliament and programme stakeholders. We are unlikely to see a resurgence of investigative journalism and evidence-based debates in our media. There will be growing consensus among AES members regarding the benefits of professionalisation and the need to increase the range and quality of education and training opportunities to suit flexible learning arrangements. There will also be a strong focus in 2023 and beyond on the practical application of Indigenous evaluation paradigms, culturally safe evaluation and involvement of consumers in evaluation processes. The Australian evaluation landscape will continue to emphasise multiple methods rather than any particular quantitative or qualitative methodology. Pluralism and variable quality will remain a norm throughout the next decade, although we hope to see more attention to evaluation quality assurance and meta-evaluation. Finally, there will be exponential developments in big data, including linking of administrative data systems and the use of data visualisation techniques to increase the quality and richness of real-time monitoring and evaluation reporting.

References Alexander, E., & Thodey, D. (2018). Independent review into the operation of the Public Governance, Performance and Accountability Act 2013 and rule. Department of Finance. https://www.finance.gov.au/publications/ reviews/2018-2019/independent-review-operation-public-governance-per formance-and-accountability-act-2013-and-rule. Accessed on 19 December 2022. Australian Charities and Not-for-Profits Commission. (2021). Australian charities report (7th ed.). Australian Government. https://www.acnc.gov.au/ tools/reports/australian-charities-report-7th-edition. Accessed on 23 January 2023. Australian Evaluation Society (AES). (2013a). Code of ethics. https://aes.asn. au/images/AES_Code_of_Ethics_web.pdf?type=file. Accessed on 23 January 2023.

2

EVALUATION IN AUSTRALIA

91

Australian Evaluation Society (AES). (2013b). Guidelines for the ethical conduct of evaluations. https://www.aes.asn.au/images/AES_Guidelines_web_v2.pdf? type=file. Accessed on 23 January 2023. Australian National Audit Office. (2019). Evaluating Aboriginal and Torres Strait Islander programs (Auditor-General Report No. 47 of 2018–19). https://www.anao.gov.au/work/performance-audit/evaluating-indigenousprograms. Accessed on 23 January 2023. Barraket, J., & Yousefpour, N. (2013). Evaluation and social impact measurement amongst small to medium social enterprises: Process, purpose and value. Australian Journal of Public Administration, 72(4), 447–458. Bray, J. B., Gray, M., & t’Hart, P. (2019). Evaluation and learning from failure and success: An ANZSOG research paper for the Australian Public Service Review Panel. Australia and New Zealand School of Government. https://www.anzsog.edu.au/resource-library/research/learning-fromfailure-and-success. Accessed on 23 January 2023. Davis, G. T. (2014). Why do we still have journals? Administrative Science Quarterly, 59(2), 193–201. Davis, G., Wanna, J., Warhurst, J., & Weller, P. (1993). Public policy in Australia (2nd ed). Allen & Unwin. Department of Finance. (2021). Commonwealth Evaluation Policy. Australian Government. https://www.finance.gov.au/government/managing-common wealth-resources/planning-and-reporting/commonwealth-performance-fra mework/commonwealth-evaluation-policy. Accessed on 23 January 2023. Department of Foreign Affairs and Trade. (2021). Australia in brief. Australian Government. https://www.dfat.gov.au/about-us/publications/aus tralia-in-brief. Accessed on 23 January 2023. Department of Prime Minister and Cabinet. (2018). Indigenous Advancement Strategy evaluation framework. Commonwealth of Australia. https://www. niaa.gov.au/sites/default/files/publications/ias-evaluation-framework.pdf. Accessed on 23 January 2023. Doherty, B., Eccleston, R., Hansen, E., Natalier, K., & Churchill, B. (2015). Building evaluation capacity in micro community organisations—More burden than benefit? Evaluation Journal of Australasia, 15(4), 29–37. Fraser, D. (2001). Editorial. Evaluation Journal of Australasia, 1(2), 2–3. Gollan, S., & Stacey, K. (2021). Australian Evaluation Society First Nations Cultural Safety Framework. Australian Evaluation Society. https://www.aes. asn.au/images/AES_FirstNations_Cultural_Framework_finalWEB_final.pdf. Accessed on 23 January 2023. Graves, P. (2020). Evaluation in the Australian Public Service: Formerly practised—Not yet embedded. Evaluation Journal of Australasia, 20(4), 229–243.

92

B. ASTBURY AND S. BAYLEY

Graves, P. W., Blackman, D., & O’Donnell, M. (2021). Embedding Australian Public Service management reforms: The Secretary could not make it so. Australian Journal of Public Administration, 80(2), 1–16. Gray, M., & Bray, J. R. (2019). Evaluation in the Australian Public Service: Current state of play, some issues and future directions. An ANZSOG research paper for the Australian Public Service Review Panel. Australia & New Zealand School of Government. https://www.apsreview.gov.au/sites/def ault/files/resources/appendix-b-evaluation-aps.pdf. Accessed on 23 January 2023. Gruen, N. (2016). Why Australia needs an evaluator-general. The Mandarin. Published 9 May 2016. https://www.themandarin.com.au/64566-nicholasgruen-evaluator-general-part-two/. Accessed on 23 January 2023. Gruen, N. (2018). The case for an Evaluator General: Why we need it and how we should configure it. https://www.themandarin.com.au/wp-content/uploads/ 2018/11/Nicholas-Gruen-Evaluator-General-Submission-2018.pdf. Accessed on 23 January 2023. Howes, S. (2020). Office of Development effectiveness: Praised, then abolished. Development Policy Blog. https://devpolicy.org/office-of-development-eff ectiveness-praised-then-abolished-20200925-3/. Accessed on 23 January 2023. Joint Committee of Public Accounts and Audit (JCPAA). (2017). Report 469: Commonwealth performance framework—Inquiry based on Auditor-General’s Reports 31 (2015–16), 6 and 58 (2016–17). https://www.aph.gov.au/Par liamentary_Business/Committees/Joint/Public_Accounts_and_Audit/CPF/ Report_1. Accessed on 23 January 2023. Jones, B. (2020). What is to be done: Political engagement and saving the planet. Scribe. Keating, M. (1990). Managing for results in the public interest. Australian Journal of Public Administration, 49(4), 387–397. Kelly, L. (2021). Evaluation in small development non-profits: Deadends, victories, and alternative routes. Palgrave Macmillan. Mackay, K. (1992). The use of evaluation in the budget process. Australian Journal of Public Administration, 51(4), 436–439. Mackay, K. (2011). The performance framework of the Australian Government, 1987 to 2011. OECD Journal on Budgeting, 11(3), 1–48. Mackay, K., & Robertson, I. (1994). The influence of evaluation findings in the Commonwealth budget. Evaluation News and Comment, 3(1), 15–18. Maloney, J. (2017). Evaluation: What’s the use? Evaluation Journal of Australasia, 17 (4), 25–38. Mason, S., & Hunt, A. (2019). So what do you do? Exploring evaluator descriptions of their work. American Journal of Evaluation, 40(3), 395–413.

2

EVALUATION IN AUSTRALIA

93

McCoy, A., Rose, D., & Connolly, M. (2013). Developing evaluation cultures in human service organisations. Evaluation Journal of Australasia, 13(1), 15–20. McTaggart, R., Caulley, D., & Kemmis, S. (1991). Evaluation traditions in Australia: Distillation of the old, wellspring of the new. Evaluation and Program Planning, 14(3), 123–130. Mitrou, F., Cooke, M., Lawrence, D., Povah, D., Mobilia, E., Guimond, E., & Zubrick, S. R. (2014). Gaps in Indigenous disadvantage not closing: A census cohort study of social determinants of health in Australia, Canada, and New Zealand from 1981–2006. BMC Public Health, 14(1), 1–9. Morton, D., & Cook, B. (2018). Evaluators and the enhanced commonwealth performance framework. Evaluation Journal of Australasia, 18(3), 141–164. Neesham, C., McCormick, L., & Greenwood, M. (2017). When paradigms meet: Interacting perspectives on evaluation in the non-profit sector. Financial Accountability & Management, 33(2), 192–219. Nielsen, S. B., Lemire, S., & Christie, C. A. (2018). The evaluation marketplace and its industry. New Directions for Evaluation, 160, 13–28. Onyx, J., Cham, L., & Dalton, B. (2016). Current trends in Australian nonprofit policy. Nonprofit Policy Forum, 7 (2), 171–188. Owen, J. M. (2004). Evaluation forms: Toward an inclusive framework for evaluation practice. In M. C. Alkin (Ed.), Evaluation roots: Tracing theorists’ views and influences (pp. 356–369). Sage. Patton, M. Q. (2013). Utilisation-focused evaluation checklist. Evaluation Checklist Project. https://www.wmich.edu/sites/default/files/attachments/u350/ 2014/UFE_checklist_2013.pdf. Accessed on 23 January 2023. Patton, M. Q. (2018). Evaluation science. American Journal of Evaluation, 39(2), 183–200. Peersman, G., & Rogers, P. (2017). Pathways to advance professionalisation within the context of the Australasian Evaluation Society. ANZOG/Better https://www.aes.asn.au/images/Discussion-Papers-and-Rep Evaluation. orts/Pathways_to_advance_professionalisation_within_the_context_of_the_ Australasian_Evaluation_Society_Peersman_Rogers.pdf. Accessed on 23 January 2023. Picciotto, R. (2019). Is evaluation obsolete in a post-truth world? Evaluation and Program Planning, 73, 88–96. Productivity Commission. (2010). Contribution of the not-for-profit sector. Australian Government. https://www.pc.gov.au/inquiries/completed/notfor-profit/report/not-for-profit-report.pdf. Accessed on 23 January 2023. Public Governance, Performance and Accountability Act. (2013). https://www. legislation.gov.au/Details/C2016C00414. Accessed on 23 January 2023. Rogers, A., Kelly, L. M., & McCoy, A. (2019). Evaluation literacy: Perspectives of internal evaluators in non-government organizations. Canadian Journal of Program Evaluation, 34(1), 1–20.

94

B. ASTBURY AND S. BAYLEY

Rogers, P. J., & Davidson, E. J. (2013). Australian and New Zealand evaluation theorists. In M. C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists’ views and influence (pp. 371–385). Sage. Schwandt, T. A. (2019). Post-normal evaluation? Evaluation, 25(3), 317–329. Sellitto, C. (2009). A study of journal publication attributes: Some considerations for academics in the information systems discipline. Webology, 6(1), Article 66. Sharp, C. A. (2003). Development of program evaluation in Australia and the Australasian Evaluation Society—The early decades. Evaluation Journal of Australasia, 3(2), 6–15. Shergold, P., Broadbent, J., Marshall, I., & Varghese, P. (2022). Fault lines: An independent review into Australia’s response to COVID-19. https://assets.web site-files.com/62b998c0c9af9f65bba26051/6350438b7df8c77439846e97_ FAULT-LINES-1.pdf. Accessed on 23 January 2023. Siminski, P., & Cobb-Clark, D. A. (2019). Labor’s idea of an Evaluator General could dramatically cut wasteful spending. The Conversation. https://theconversation.com/labors-idea-of-an-evaluator-general-coulddramatically-cut-wasteful-spending-115840. Accessed on 23 January 2023. Turner, D. (2003). Evaluation ethics and quality: Results of a survey of Australasian Evaluation Society members. https://www.aes.asn.au/images/ ethics_survey_summary.pdf?type=file. Accessed on 23 January 2023. Vo, A. T., & Christie, C. A. (2018). Where impact measurement meets evaluation: Tensions, challenges, and opportunities. American Journal of Evaluation, 39(3), 383–388. Williamson, A. K., & Kingston, K. L. (2021). Performance measurement, evaluation and accountability in public philanthropic foundations. Evaluation Journal of Australasia, 21(2), 101–119. Wilson, J. D. (1999). Editorial: Is there a distinctive Australasian evaluation tradition? Evaluation Journal of Australasia, 11(1), 3–4.

CHAPTER 3

Evaluation in Bangladesh Md Faisal Ibne Wali, Sharif N. As-Saber, Muhammad Wali Ullah, Muhammad Maruf Ibne Wali, and Nabila Kamal

1

General Country Overview

The People’s Republic of Bangladesh is a country in South Asia, bordering India and Myanmar. It was part of British India until the subcontinent was partitioned in 1947 into two independent sovereign countries, India and Pakistan. Bangladesh then became a province of Pakistan, known as East Pakistan (Faruque, 2009; Quibria, 2019). In 1971, after a nine-month-long liberation war, Bangladesh emerged as an independent nation separated from Pakistan (Choudhury & Basher,

M. F. I. Wali (B) RMIT University, Melbourne, VIC, Australia e-mail: [email protected] S. N. As-Saber College of Economics and Political Science, Sultan Qaboos University, Seeb, Oman e-mail: [email protected] M. W. Ullah EnviSArc Consultancy & Management, Dhaka, Bangladesh © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_3

95

96

M. F. I. WALI ET AL.

2002; Quibria, 2019). In its first few years as a sovereign nation, Bangladesh struggled with desperate poverty, overpopulation, and overwhelming damage to its institutional and physical capital (Mahmud, 2008). Given the situation, the country was once dubbed as an ‘international basket case’ (Misra, 2022). However, since then, Bangladesh has capitalised on its two most crucial endowments—land and labour and managed to come out of such a dilapidated condition (Helal & Hossain, 2013; Khondker, 2015). Bangladesh has experienced substantial economic growth since its independence. Over the past decade, Bangladesh has been among the fastest-growing economies globally (World Bank, 2021a). For Bangladesh, the gross domestic product (GDP) growth rate varied between 2–3.5% until the 1980s (Helal & Hossain, 2013). It started achieving higher economic growth by introducing a free market economy during the 1990s (Mahmood & Absar, 2015; Pricewaterhouse Coopers, 2015). From 2000 to 2010, Bangladesh experienced a steady and robust GDP growth at an average rate of more than 6% (Giménez et al., 2014). In 2019, the country reported a GDP growth rate of 8.2% (Asian Development Bank, 2021). Similar to other countries, Bangladesh’s economic progress has been hard hit by the COVID-19 pandemic. However, in the fiscal year 2020/2021, the country managed to achieve a real GDP growth of 5.2%, which is the highest in Asia (The Financial Express, 2021a). Observing the economic resilience of Bangladesh during the COVID-19 pandemic, the Asian Development Bank (2021) predicted Bangladesh’s GDP growth rate of 7.2% in 2022. Over the past 30 years, Bangladesh has navigated through significant changes. After independence, agriculture was the primary source of income for the people of Bangladesh, contributing 56% to the country’s GDP (Quibria, 2019). At present, the economy is primarily driven

M. M. I. Wali Operations Management, Reeves College, Calgary, AB, Canada N. Kamal Independent University Bangladesh, Dhaka, Bangladesh

3

EVALUATION IN BANGLADESH

97

by the export-oriented readymade garments (RMG) industry, remittances, and the agricultural sector (Islam, 2019). According to data from the Bangladesh government’s Department of Agricultural Extension and the United Nations (UN) Food and Agriculture Organization (FAO), Bangladesh is ranked among the top 10 globally for producing various agricultural products, including rice, fruits, and vegetables. In addition, Bangladesh holds leading positions in the export of RMG products (second in the world), jute (first in export, second in production), and remittance (eighth in the world) (Islam, 2021). Bangladesh is also rich in renewable energy resources, including solar energy, hydroelectricity, wind energy, biomass, and non-renewable resources, including raw coal, gas, minerals, ceramic soil, stone, and boulders (Bhuiyan et al., 2021; Reza, 2017). Such richness in natural resources promises the development of potentially relevant industries and economic advancement. Bangladesh is a democratic republic with two tiers of government, national and local, in political and administrative contexts. Local government is enshrined in the Constitution, and the main legislative texts include Acts covering Zila Parishad or District Councils (2000), Upazila Parishad or Sub-District Councils (1998, amended 2009), union Parishads or Union Councils (2009), Pourashavas or Municipal Councils (2009), City Corporations (2009) and Hill District Councils (1989) (Chattogram City Corporation, 2012; Commonwealth Local Government Forum, 2021; Laws of Bangladesh, 2019a, 2019b, 2019c, 2019d, 2019e). Monitoring and evaluation have been a crucial part of project and policy implementation in Bangladesh since the country’s emergence in 1971 (IMED, 2019a). Evaluation occurs in the country through government agencies and the private sector, including non-government organisations (NGOs) and international development agencies. In the following sections, we discuss the details of monitoring and evaluation practices in Bangladesh.

2 Institutional Structures and Processes (Political System) In this section, we discuss evaluation regulation, central government institutions of evaluation in Bangladesh, parliamentarian and national audit structures of evaluation, and evaluation practices in the country.

98

M. F. I. WALI ET AL.

2.1

Evaluation Regulations

After liberation in 1971, the number of foreign-aided projects increased in Bangladesh, and consequently, strengthening monitoring and evaluation policy and practices became critical for the country (IMED, 2019a). Later, national laws were approved that made the evaluation of all administrative practices including contractual arrangements for development activities mandatory in Bangladesh. For instance, Rules of Business 1996 section 4 (ix) makes evaluation of execution plans mandatory for government ministries in the country (Aminuzzaman, 2013; Cabinet Division, 2017). In Bangladesh, the Implementation Monitoring and Evaluation Division (IMED) is the apex body for monitoring and evaluation. IMED has developed a policy framework that highlights policies on evaluation, monitoring, and review (IMED, 2019a). The government of Bangladesh also allocates a portion of its annual budget to the development of monitoring and evaluation (Ministry of Finance, 2022). Furthermore, sectoral regulations on evaluation are also common in Bangladesh. For instance, Bangladesh has enacted the Government Educational and Training Institutions Order 1972 and President’s Order No. 71 of 1972 (an adapted version of the Government Educational and Training Institutions Ordinance 1961) to monitor and evaluate government educational and training institutions (Laws of Bangladesh, 2019f, 2019g). 2.2

Parliamentarian and National Audit Structures

The parliament of Bangladesh (the Jatiyo Sangshad),1 which is the country’s supreme legislative body, commissions evaluations and has standing committees for monitoring and evaluation (Aminuzzaman, 2013). These committees include the Committee on Estimates, the Public Accounts Committee, and the Committee on Public Undertaking. These committees evaluate financial statements and audit reports of the central government and state-owned enterprises (Wyatt, 2021). However, the role of evaluation reports in parliamentary debates in Bangladesh remains unclear.

1 Bangladesh Parliament (2021). http://www.parliament.gov.bd/. Accessed on 9 October 2021.

3

EVALUATION IN BANGLADESH

99

The People’s Republic of Bangladesh (Act No. 1972, part VIII) requires establishing an Office of the Auditor General (Legislative and Parliament Affairs Division, 2019). The Office of the Comptroller and Auditor General (OCAG) heads Bangladesh’s Supreme Audit Institution. This unit is responsible for auditing government receipts and public spending and evaluating whether expenditures have yielded value for money in government offices, public bodies, and statutory organisations. The Constitution has provided the OCAG with complete independence, implying that it has exclusive authority to access all required documents and conduct evaluations independently. The OCAG usually conducts compliance, financial and performance audits of the ministries of the Bangladesh government on an annual basis (Comptroller & Auditor General of Bangladesh, 2020). The OCAG decides on further courses of action depending on that report. 2.3

Organisational Structure

Monitoring and evaluation in Bangladesh occur at public and private levels. The government’s planning and implementation division must approve any development project before reaching its monitoring and evaluation stage. First, project planning is approved in the National Parliament and sent to the National Economic Council (NEC). The NEC further finalises and approves the plan and then sends it to the Executive Committee of the National Economic Council (ECNEC) for final approval. The Ministry of Planning then works on the project’s formulation, implementation, and evaluation. The Ministry of Planning of Bangladesh has four divisions: (1) Economic Relations Division, (2) Planning Commission (3) IMED and (4) Statistical Division and Bangladesh Institute of Development Studies under the Statistical Division (Hamiduzzaman, 2014). Government institutions have evolved through significant structural changes since the independence of Bangladesh. In 1975, considering the importance of monitoring and evaluation, the Bangladesh government created the Project Implementation Bureau under the office of the President of Bangladesh (Hamiduzzaman, 2014; IMED, 2019a). In 1977, the Project Implementation Bureau was upgraded to an individual division, the Project Monitoring Division, named IMED in the early 1980s and placed under the Ministry of Planning (Hamiduzzaman, 2014; IMED,

100

M. F. I. WALI ET AL.

2019a; Majid, 2008). IMED is now Bangladesh’s apex body for monitoring and evaluation. It monitors and evaluates the implementation of public-sector development projects included in the Annual Development Programme (ADP). It is also responsible for inspecting field projects and submitting reports to the President of Bangladesh and relevant ministers (IMED, 2019a). Moreover, IMED provides advisory and consultancy services to ministries or agencies involved in project implementation. It oversees matters related to the Central Procurement Unit’s activities and compliance with public procurement regulations (Hamiduzzaman, 2014). It is centrally responsible for monitoring and evaluating the performances of all ministries and divisions. It also oversees other development projects in the country. For effective functioning, IMED has categorised the ministries and divisions into eight sectors (see Table 1). IMED functions in two ways (IMED, 2019a): • Conducting and controlling monitoring and evaluation of the various sectors/industries/organisations in the country • Collaborating with various private (local and international) organisations to monitor and evaluate. Although IMED takes the lead role in monitoring and evaluating in Bangladesh, all ministries and divisions are responsible for evaluating and monitoring their own activities and reporting to relevant government bodies, including IMED. For instance, the Bangladesh Road Transport Corporation (BRTC), which is a semi-autonomous, state-owned transport corporation under the Ministry of Road Transport and Bridges, has an Annual Performance Agreement (APA) team involved in monitoring and evaluation as advised by IMED (BRTC, 2021). The utilisation of technologies for evaluation, including software/ applications, has also been observed across ministries and divisions of the government. For instance, the Ministry of Health and Family Welfare tracks the progress of its ADP using an online database system known as the ‘ADP Monitoring System’. All line managers of the Health, Population and Nutrition Sector Development Programme and other project directors of different development projects under the ministry regularly update the database with financial and physical data about their ongoing progress (Directorate General of Health Services, 2016).

3

Table 1

EVALUATION IN BANGLADESH

Sectoral division of ministries by IMED

Sector

Ministries and divisions covered by the sector

Sector 1

01. 02. 03. 04. 05. 06. 07. 08. 01. 02. 03. 04. 05. 06. 07. 01. 02. 03. 01. 02. 03. 04. 05. 06. 01. 02. 03. 04. 05. 06. 07. 08. 09. 01. 02. 03. 04. 05. 06. 07. 01. 02. 03. 04. 05. 06. 07. 08. 09.

Sector 2

Sector 3

Sector 4

Sector 5

Sector 6

Sector 7

101

Power Division Energy and Mineral Resources Division Ministry of Foreign Affairs Ministry of Public Administration Ministry of Liberation War Affairs Law and Justice Division Ministry of Disaster Management and Relief Bangladesh Public Service Commission Road Transport and Highways Division Ministry of Railways Bridge Division Cabinet Division Bangladesh Election Commission Secretariat Economic Relations Division Anti-Corruption Commission Local Government Division National Parliament Secretariat Legislative and Parliamentary Affairs Division Ministry of Agriculture Ministry of Water Resources Ministry of Food Ministry of Land Financial Institutions Division Internal Resources Division Health Services Division Medical Education and Family Welfare Division Public Security Division Security Services Division Ministry of Cultural Affairs Ministry of Information and Broadcasting Ministry of Expatriates’ Welfare and Overseas Employment Ministry of Labour and Employment Ministry of Industries Ministry of Primary and Mass Education Secondary and Higher Education Division Technical and Madrasha Education Division Ministry of Youth and Sports Ministry of Fisheries and Livestock Ministry of Civil Aviation and Tourism Ministry of Textiles and Jute Posts and Telecommunications Division Information and Communication Technology Division Ministry of Housing and Public Works Prime Minister’s Office Ministry of Women and Children Affairs Ministry of Commerce Ministry of Science and Technology Finance Division Ministry of Religious Affairs

(continued)

102

M. F. I. WALI ET AL.

Table 1

(continued)

Sector

Ministries and divisions covered by the sector

Sector 8

01. 02. 03. 04. 05. 06. 07. 08. 09.

Ministry of Chittagong Hill Tracts Affairs Ministry of Environment, Forest, and Climate Change Ministry of Shipping Rural Development and Co-operatives Division Ministry of Social Welfare Planning Division Statistics and Informatics Division Implementation Monitoring and Evaluation Division Ministry of Defence

Source IMED (2021)

2.4

Evaluation Practice and Use

Evaluation Practice and Use at the Government Level In Bangladesh, most decisions on evaluation practices are delegated to ministries, local governments, and public organisations. IMED has been working as the focal point for monitoring and evaluation in Bangladesh and oversees both internal and external assessment tasks (IMED, 2019a). The IMED prepares monthly, quarterly, and annual evaluation reports. Monthly financial progress reports are submitted to relevant ministries and the ECNEC. Quarterly progress reports help IMED identify early problems and submit recommendations to the ECNEC. The annual report highlights financial and physical progress and is submitted to the NEC at the end of each fiscal year (Hamiduzzaman, 2014; IMED, 2019a). Table 2 presents the impact evaluations conducted by IMED from 2009 to 2021. Data on evaluations conducted in the 2020–2021 fiscal year (Table 3) imply that these evaluations were conducted on various sectors including road transport and bridges, rural water and electricity supply development, agriculture, and education. IMED has also developed the Monitoring and Evaluation Policy Framework, which has the following core objectives (IMED, 2019a): • to ensure accountability among the critical agencies engaged in public service sectors • to assist in ensuring results from development investments

3

Table 2 Number of impact evaluations by IMED

EVALUATION IN BANGLADESH

103

Fiscal year

Number of impact evaluations conducted by IMED

2020–2021 2019–2020 2018–2019 2017–2018 2016–2017 2015–2016 2014–2015 2013–2014 2012–2013 2011–2012 2010–2011 2009–2010 Sum (2009–2021)

8 24 28 9 12 18 15 14 12 10 10 7 167

Source IMED (2022)

Table 3

Impact evaluations by IMED 2020–2021

Fiscal year

Details

2020–2021

01. Construction of Sheikh Kamal Setu, Sheikh Jamal Setu and Sheikh Russel Setu 02. South Asia Subregional Economic Cooperation (SASEC) Road Connectivity Project: Improvement of Banapole and Burimari Land Port 03. Amar Bari Amar Khamar Project 04. Bangladesh Rural Water Supply and Sanitation Project (BRWSSP) 05. Procurement of Scientific Equipment for the Development of Educational Facilities in Bangladesh Public Universities 06. Safe Crop Production Project through Integrated Pest Management (IPM) 07. Establishment of Fire Service and Civil Defence Burn Treatment Hospital 08. 1.8 million customers connection through the extension of rural electrification (Department of Power/Ministry of Power, Energy and Mineral Resource)

Source IMED (2022)

• to support in assessment and measurement of direct and indirect impacts of development interventions on the lives and livelihoods of the people

104

M. F. I. WALI ET AL.

• to improve effectiveness and result orientation in future strategies, policies, and programmes by obtaining feedback from past learning. Figure 1 shows the critical visions and objectives of monitoring and evaluating the ADP across three-time frames: short, medium, and long. IMED has taken the initiative to bring harmony and set identical investment management and monitoring objectives for the Finance Division and IMED’s monitoring and evaluation principles appear to align with long- and medium-term goals and targets, including UN Sustainable Development Goals (SDGs) (IMED, 2019a). Nonetheless, IMED is also focused on ensuring governmental evaluation quality, emphasising the evaluation of its performance. To ensure the quality of the evaluation, IMED has focused on the capacity building of its divisions through its project, ‘Strengthening Monitoring & Evaluation (M&E) Capabilities of IMED’. With this ongoing mission, IMED monitors its activities, evaluates its strengths and weaknesses, and brings structural changes to improve its operations (IMED, 2019a). Bangladesh’s government has also emphasised the monitoring and evaluation of the realisation of the UN’s SDGs. The SDGs are a global call by the UN to end poverty, protect the Earth’s environment and ensure that all people can enjoy peace and prosperity (UN Development Programme, 2021). Bangladesh is a signatory of the 2030 Agenda for Sustainable Development. The government has involved various stakeholders in implementing the SDGs with a ‘whole of society’ approach (Citizen’s Platform for SDGs, Bangladesh, 2019). The government considers public governance and monitoring as vital for successfully implementing this agenda (Morton et al., 2017). The General Economic Division (GED) under the Ministry of Planning is responsible for monitoring and evaluating SDGs in Bangladesh within the public sector. Allen et al. (2018) emphasise that an evaluation of national progress in implementing the SDGs is needed to identify existing gaps. To this end, the GED has developed a Monitoring and Evaluation Framework for the SDGs (General Economic Division, 2018). The GED has also published progress reports (in 2020, the second progress report was published), where it evaluates Bangladesh’s public and private sector progress in realising the SDGs. Another important monitoring and evaluation tool is the Annual Performance Agreement (APA), which is important to note here. The Prime Minister’s Office introduced the APA in Bangladesh in 2015.

3

EVALUATION IN BANGLADESH

105

Fig. 1 Management and monitoring of annual development programme (Note MTBF = Medium-Term Budget Framework; FYP = Fiscal Year Plan; AB = Annual Budget; KPI = Key Performance Indicator; ADP = Annual Development Programme; M & E = Monitoring & Evaluation. Source Author’s own work and illustration. General Economic Division [2018])

It is an instrument to assess a specific ‘target with value’ for understanding the progress of planned activities by relevant departments and their subordinate offices (Azad, 2020). Evaluation Practice at the Sector/Industry Level A closer look at sector-based evaluation reveals that although evaluation is conducted frequently in some sectors, evaluation remains infrequent in others. According to the IMED (2019b), the three most important sectors in Bangladesh where evaluation is conducted regularly are education, health and nutrition, family welfare, and social welfare. These

106

M. F. I. WALI ET AL.

sectors are the most critical service delivery sectors for achieving the government’s development goals. Extensive monitoring and evaluation take place in the health sector in Bangladesh. The Ministry of Health and Family Welfare is at the centre of health monitoring and policy assessment (Ministry of Health & Family Welfare, 2008). Many public and private organisations monitor and evaluate various health-related projects. For example, the Bangladesh Integrated Nutrition Project commenced in 1995 to reduce child malnutrition. This project was monitored and evaluated by multiple organisations, including Save the Children, the International Centre for Diarrhoeal Disease Research (ICDDR), Bangladesh, the Nutritional Surveillance Project of Helen Keller International, and the Operations Evaluation Department of the World Bank (White, 2005). Bangladesh also established an annual process to monitor the progress of its Health, Population, and Nutrition Sector Development Programme (Directorate General of Health Services, 2013). Moreover, the Bangladeshi education sector is also frequently evaluated and monitored. At the public level, the education system in Bangladesh is divided into three levels: primary, secondary, and higher secondary. Primarylevel education is provided under two major institutional arrangements (streams): general and madrasah, while secondary education has three major streams: general, technical-vocational, and madrasah. Higher education has three streams: general (pure and applied science, arts, business, and social science), madrasah, and technology education. The Education Ministry of Bangladesh monitors and evaluates education across all these levels (Education Ministry, 2021). Although the government has less control over the private education system, especially English medium schools, these institutions must register with and submit their evaluation reports to the government (Abbas, 2017). There is also a separate government body called the University Grants Commission (UGC), which is responsible for monitoring and evaluating the higher education institutions in the country (UGC, 2021). However, there is scepticism about the capacity and/or authority to monitor the quality of higher education in Bangladesh while most of the leading public universities rank poorly according to various international surveys (The Financial Express, 2021b). Major industries in Bangladesh vary in their approach to monitoring and evaluation. Industries are primarily responsible for monitoring and evaluating their own activities. However, various government and civil

3

EVALUATION IN BANGLADESH

107

society organisations (local and international) also monitor practices and policies at the industry level (cf. The Independent, 2020). For example, in the case of the Bangladesh RMG industry, the global initiatives of the Accord for Workplace Safety, Alliance for Bangladesh Worker Safety and National Initiative were instrumental in improving the workplace condition within the RMG sector (Moazzem et al., 2018). Moreover, RMG industry-based local associations, such as Bangladesh Garment Manufacturers and Exporters Association (BGMEA) and Bangladesh Knitwear Manufacturers and Exporters Association (BKMEA), monitor RMG industry policies and practices in collaboration with government bodies and international organisations. For instance, BGMEA, along with International Labour Organization (ILO), evaluated and acted upon the ILO/IPEC garment sector project to remove and rehabilitate child workers in the garment industry (Jeddere-Fisher & Khair, 2004). However, evaluation often lacks continuity and consistency in the informal sector in Bangladesh. Most formal jobs are associated with semi to high-skilled workers, while unskilled work is mostly associated with the informal sector (Asian Development Bank, 2012). Informal employment includes agricultural day labourers, urban street vendors, paid domestic workers, and at-home producers of clothing and other manufactured goods (Raihan, 2010). Informal jobs mostly fall outside the domain of the government’s labour market regulation. Moreover, informal workers do not enjoy legal protections concerning working hours, health, and safety or mandated benefits that would normally be a feature of formal employment opportunities in large, registered private sector firms or in the public sector (Hussmanns, 2004). Evaluation activities are mostly absent and neglected in the informal sector. Overall, both impact and process evaluations are conducted in Bangladesh, especially in the formal sector. We have listed various impact evaluations conducted by IMED in Table 2. Along with impact evaluations, process evaluations are also common in Bangladesh. For instance, in 2017, the World Bank conducted a process evaluation of Bangladesh Water Partnership for Cleaner Textiles (PacT), which is an IFC-led advisory partnership for cleaner textiles (Manchanda, 2017; PaCT, 2022).

108

M. F. I. WALI ET AL.

3 Societal Dissemination/ Acceptance (Social System) 3.1

Use of Evaluations by Civil Society

The discourse of monitoring and evaluation in development practice and theory has recently gained prominence. The purveyors of development increasingly prioritise monitoring and evaluation as a platform for learning and accountability. This growing importance arose from increasing scrutiny by civil society on good governance and their demands for efficient public administration (Kabonga, 2020). Although IMED is primarily responsible for monitoring and evaluation of governmental activities in Bangladesh, various civil society organisations (both local and international) are directly and indirectly involved in monitoring and evaluation. While evidence of the frequency of evaluation conducted by local civil society organisations in Bangladesh is low and limited, there are instances of involvement by international organisations in such evaluation process, e.g., the World Bank Group conducted the Bangladesh Country Program Evaluation to assess the development effectiveness of their engagement with Bangladesh during the 2011–2020 fiscal years (World Bank Group, 2020). Similarly, the United Nations International Children’s Emergency Fund (UNICEF) prepared evaluation reports on its own response to the Rohingya refugee crisis in Bangladesh (UNICEF et al., 2019). 3.2

Public Perception and Discourse on Evaluation and Evaluation Results

In Bangladesh, the evaluation instrument is well known, especially among government bodies, public and private organisations, and NGOs. The evaluation reports by these organisations are usually made available to the public, especially on online platforms including their official websites. These organisations also make evaluation reports available to their donors. A World Bank conference paper from 1999 suggests that submitting evaluation reports to donors has been a widespread practice in Bangladesh for organisations like the Bangladesh Rural Advancement Committee (BRAC) (Ahmed & Rafi, 1999). News about evaluation studies and their brief results by government bodies, civil society organisations and private companies are often reported in national newspapers for the public. For

3

EVALUATION IN BANGLADESH

109

instance, Dhaka Tribune, a leading newspaper in Bangladesh, reported the evaluation conducted by the Sustainable Development Solution Network on various countries’ positions in realising the SDGs (Dhaka Tribune, 2021a). 3.3

Participation of Civil Society in Evaluations

Local and international civil society organisations in Bangladesh do not follow any specific participation procedure for evaluation. These organisations either evaluate to make decisions regarding their organisation’s projects or collaborate with the Bangladesh government to conduct evaluations. National level large Civil Society Organisations (CSOs) are often engaged in evaluation activities. For instance, BRAC has a Research and Evaluation Division and conducts evaluations regularly. Although this division was initially established to evaluate the activities of BRAC, it now undertakes multidisciplinary studies on various development issues and subjects of national and global importance, including poverty alleviation, socio-economic development, agriculture, environment, disability, and gender. Ad-hoc studies are also commissioned from time to time to inform BRAC management about distinct aspects of programme implementation (BRAC, 2021a). Such evaluation provides an analytical basis for BRAC’s programme decisions and help them to better perform and make their development efforts more evidence-based, effective, and community-sensitive (Chowdhury et al., 2014). In general, local and international CSOs in Bangladesh use evaluation-based processes to make decisions on programme initiation and continuation. Moreover, such organisations use evaluative evidence when seeking funding locally as well as internationally. The Center for Development Evaluation is another significant local civil society institution actively involved in evaluation in Bangladesh. This organisation conducts development and economic surveys and field experiments to evaluate different projects and programmes and develop sustainable policies for government and NGOs. Its research-based evaluations cover poverty, health, gender, agriculture, food security, climate change, social protection and migration, financial inclusion and microfinance, and environmental policy analysis (Center for Development Evaluation, 2021). Several international civil society organisations are also actively involved in monitoring and carrying out evaluations in Bangladesh,

110

M. F. I. WALI ET AL.

including the Asian Development Bank, Mendez England and Associates Inc., the United States Agency for International Development (USAID), and various other UN organisations. Moreover, many other organisations work in partnership with USAID in Bangladesh to monitor and evaluate various sectors in Bangladesh. For instance, Measure Evaluation, funded by USAID, evaluated surveys for newly established urban and rural health service delivery projects by USAID in Bangladesh (Measure Evaluation, 2021). Similarly, Social Impact, a global development management consulting firm, provides monitoring, evaluation, strategic planning, and capacity-building services to advance development effectiveness (Social Impact, 2021). International Business and Technical Consultants, Inc. is also an international development consulting company involved in monitoring and evaluationbased research (International Business and Technical Consultants, Inc., 2021). However, scholars (e.g., Rahman, 2019) have advocated for more participatory monitoring and evaluation in Bangladesh to enhance the quality and frequency of monitoring practices. 3.4

Demand for Evaluation Within Civil Society

Apart from actively participating in monitoring and evaluation, civil society organisations in Bangladesh also demand a better monitoring and evaluation system within the relevant government and non-government bodies. International donors and financial institutions require sectorbased monitoring and evaluation reports when they donate or sanction loans for projects in Bangladesh. Being a developing country, Bangladesh has been receiving loans and funds for its development initiatives from various international development partners, including the World Bank, Japan International Cooperation Agency, and the European Union. For instance, in 2020, after the outbreak of the COVID-19 pandemic, the World Bank (2020) approved USD 1.05 billion for three projects to help Bangladesh create quality jobs, accelerate economic recovery, and build resilience to future crises. At the beginning of 2021, Bangladesh also received USD 500 million from the World Bank to help upgrade the Jashore-Jhenaidah highway and connect rural roads and markets including ‘growth centres’ (World Bank, 2021b). To approve an allocation of such loans and donations, these donor organisations first require detailed reports from the relevant government body and constant and efficient monitoring of the projects, once approved.

3

EVALUATION IN BANGLADESH

111

In addition to international donors, various international business organisations (multinational organisations) require industry-based monitoring and evaluation reports before investing in business projects in Bangladesh. As Bangladesh is now considered the ‘Next Asian Tiger’, because of its tremendous economic growth opportunities (Islam, 2021), Bangladesh has become a centre of attraction for foreign direct investment (FDI). Bangladesh’s FDI increased by USD 847 million in September 2021 quarter compared to a growth of USD 821 million in the previous quarter (Census and Economic Information Center, 2022). Despite COVID-19 and its adverse impact globally have negatively affected the FDI inflow in Bangladesh (Dhaka Tribune, 2021b), it remains one of the most lucrative countries for FDI. Nevertheless, before deciding on investing in Bangladesh, international business organisations look for industry-based monitoring and evaluation reports. Various public and private organisations provide financial monitoring and evaluation reports to these multinational business organisations. For instance, KPMG, which provides audit, tax, and investment advisory services, monitors and evaluates the investment feasibility of various industries/sectors in Bangladesh (KPMG, 2021a). It also publishes the Bangladesh Investment Guide (KPMG, 2021b). In response to the trend of increasing FDI in Bangladesh, many multinational organisations now frequently demand monitoring and evaluation reports on Bangladesh.

4 Professionalisation of Evaluation (System of Professions) 4.1

Academic Study Courses and Training Practices

In Bangladesh, the scope of university or higher educational institutionbased courses on monitoring and evaluation remains limited. However, universities in Bangladesh offer short courses on monitoring and evaluation in association with public and private think tanks. For instance, the University of Dhaka, through an affiliation with the Bangladesh Institute of Governance and Management (BIGM), offers various trainings on evaluation (BIGM, 2021a). In association with local and international organisations, private universities in Bangladesh have also started offering short courses on monitoring and evaluation. For instance, the Independent University, Bangladesh, in

112

M. F. I. WALI ET AL.

partnership with the International Centre for Climate Change and Development (ICCCAD), Bangladesh Centre for Advanced Studies (BACS), and the International Institute for Environment and Development (IIED) in London offers short courses on evaluation for community-based adaptation (ICCCAD, 2022a, 2022b). In addition to universities, public and private think tanks and training institutes also offer non-academic short courses on monitoring and evaluation. Such training courses are targeted to multiple and multi-sector stakeholders. For instance, BIGM offers a 10-week policy analysis course that has been designed specifically for public and private sector officials (BIGM, 2021b). Various other organisations also offer training on monitoring and evaluation in their fields of interest. For instance, the International Centre for Diarrhoeal Disease Research, Bangladesh (ICDDR) launched the Translating Research into Action project in 2012. They provided a nine-day workshop on monitoring and evaluation for health professionals working in Bangladesh’s population, health, and nutrition programmes (International Centre for Diarrhoeal Disease Research Bangladesh, 2012). The Bangladesh government’s National Academy for Planning and Development under the Ministry of Planning organises training programmes and provides consultancy services on evaluation and monitoring. This institution is also responsible for disseminating knowledge on planning and evaluation through publications, seminars, and workshops (National Academy for Planning and Development, 2019a). Moreover, various online courses are now available in Bangladesh that help enhance knowledge and skills in Bangladesh-specific monitoring and evaluation. For instance, the Institute of Professional Development Programs, a professional services firm, has been offering online training, consultancy, and education services on monitoring and evaluation. In addition to highlighting on the differences between traditional and participatory monitoring and evaluation systems, such services offer lessons on monitoring and evaluation basics and the process and protocol of monitoring various projects and data collection for monitoring (Institute of Professional Development Programs, 2021). Similarly, BdJobs (a leading job site and training institute in Bangladesh) provides online training on monitoring and evaluation in Bangladesh (Bdjobs eLearning, 2021).

3

4.2

EVALUATION IN BANGLADESH

113

Journals and Communication Platforms

Monitoring and evaluation of various sectors or industries of Bangladesh have also captured the attention of academics and policymakers. For example, research on the efficacy and assessment of the healthcare sector of Bangladesh and its healthcare policies has been published locally and internationally (e.g., Ahsan et al., 2017; Huda et al., 2014; Kang et al., 2021; Mahmood et al., 2011). The various governance issues including procurement and education have also been under scrutiny across the media and academic research. However, there are still some concerns about the latest or updated data availability and accessibility in some sectors including health care, housing, and education sectors. In this regard, Ahsan et al. (2017) argue that while Bangladesh is data rich, particularly from household surveys and health facility assessments, the country’s capacity for providing regular updates on major global health indicators remains low. These authors have also emphasised on the need for enhanced monitoring strategies to meet increasing data demands in health care, education, and housing sectors. Scholars including Huda et al. (2014) have also identified the scope for further improvements in the monitoring and evaluation of the healthcare development programmes in the country. There is a government initiative such as the Health, Population and Nutrition Sector Development Programme that aims to ensure that all citizens of Bangladesh enjoy good health and wellbeing by expanding access to quality and equitable health care in a healthy environment (Directorate General of Family Planning, 2021). However, Huda et al. (2014) argue that successful implementation of such programmes in the country requires significant improvements in programme monitoring and evaluation. In addition, monitoring and evaluation of the telecommunication industry in Bangladesh have also been studied and published, where scholars have evaluated and expressed their concerns about the various aspects of the telecommunication industry, including the service quality, accounting systems, and company performances (Neogy, 2014a, 2014b, 2015). Research on monitoring and evaluation across the industries/sectors of Bangladesh have been published in sector/industry-based journals, for instance: Health Research Policy & Systems, PLoS Medicine, Global Health Action, Global Disclosure of Economics and Business, American Journal of Trade and Policy and South Asian Journal. The Bangladesh government’s National Academy of Planning and Development also publishes studies on

114

M. F. I. WALI ET AL.

evaluation and monitoring in their journal, Development Review (National Academy for Planning and Development, 2019b). Given the specialised journals on monitoring and evaluation remain limited in Bangladesh, a small number of studies have been published in international journals specialised in monitoring and evaluation, e.g., Evaluation and Program Planning (Kang et al., 2021). 4.3

Professional Organisations

Professional organisations specialising in evaluation remain inadequate in Bangladesh. As previously discussed (see Sect. 3.2), local and international civil society organisations participate in monitoring and evaluation in Bangladesh. The local civil society organisations involved include NGOs (e.g., BRAC) and research institutes (e.g., The Center for Development Evaluation) (BRAC, 2021b; Center for Development Evaluation, 2021). On the other hand, the international civil society organisations involved in the evaluation are international development organisations (e.g., USAID) and international consulting firms (e.g., Social Impact) (Measure Evaluation, 2021; Social Impact, 2021). Moreover, for evaluators in Bangladesh, there are no standard guiding principles; rather, guidelines for evaluators vary across organisations (for evaluators of international, local, public, or private organisations). For the public sector, IMED is responsible for providing guidelines on evaluation and asks all ministries and departments to follow these guidelines. IMED also provides programme-specific guidelines, which it asks relevant evaluators to follow. For instance, in 2019, IMED published its Monitoring and Evaluation Guideline on Education, Health and Nutrition, Family Welfare and Social Welfare Sector Projects of the Annual Development Program (IMED, 2019b). Local and international private organisations also develop their individual evaluation guidelines. The World Bank Group has such evaluation guidelines, called the institution-wide, principles-based living evaluation policy, that applies to all parts of the group. Professional organisations such as the Institute of Engineers, Bangladesh (IEB), Bangladesh Institute of Bank Management (BIBM), Bangladesh Medical Association (BMA) and Doctors Association of Bangladesh (DAB) occasionally organise training

3

EVALUATION IN BANGLADESH

115

sessions that could enhance sector-specific awareness and understanding about evaluation and monitoring.2

5

Conclusions

In this chapter, we have highlighted the current state of monitoring and evaluation which have been crucial for project/policy implementation in Bangladesh across public and private sectors. However, while the education, health, banking, and financial industries have been subjected to greater scrutiny than others, monitoring and evaluation at large remain at their nascent stage for many industries and sectors. The government monitors and evaluates various aspects of projects/policies across both public and private sectors. However, one of the major challenges to authentic and efficient evaluation and monitoring is the lack of coordination within the executive branch of the government, where local administration faces challenges from local elites and stakeholders involved in project implementation (As-Saber & Rabbi, 2009; Waheduzzaman & As-Saber, 2015). In addition, Bangladesh has no standard certification system for evaluators or an arbitrator for evaluation and monitoring purpose. The government must take a stand to introduce a set of standards in this regard and streamline and manage the evaluation and monitoring process. The analysis also reveals that further strengthening of the monitoring and evaluation system in Bangladesh requires more active participation from the private sector. Although the private sector is improving its monitoring and evaluation practices at its own pace, further support and public–private cooperation are necessary. Civil society organisations, and community groups should also work together to enhance Bangladesh’s

2 There are several other organisations in Bangladesh which have been playing some role in the country’s performance and evaluation processes. Such role, however, is not significant enough which, in most cases remains limited to mere networking, observations and criticisms of the existing state of performance management. There are several VOPEs which have been in existence in Bangladesh as well which include, Evaluation Forum and the Bangladesh Evaluation and Monitoring Network (Holvoet et al., 2011). Bangladesh Evaluation Society (BES) is considered to be the only active VOPE in 2023, which was established in 2015 and was “largely inspired by the Dhaka Conclave for International Development by the International Initiative for Impact Evaluation (3ie)” (https://eva luationsociety.org.bd/. Accessed on 10 March 2023). The BES is an open forum for evaluation practitioners and policy makers.

116

M. F. I. WALI ET AL.

overall monitoring and evaluation system to make it inclusive, participatory, and acceptable. Moreover, public accountability is crucial. Hence, it is necessary to develop measures to ensure accountability in the monitoring and evaluation process and protocol. In doing so, a collaborative and synchronised system of evaluation and monitoring should be developed through the appropriate and efficient use of skills, resources, and technology.

References Abbas, S. M. (2017). Government to collect info on English medium schools. Dhaka Tribune. https://www.dhakatribune.com/bangladesh/education/ 2017/11/26/govt-info-english-medium-schools. Accessed on 1 September 2021. Ahmed, S., & Rafi, M. (1999). N.G.O.s and evaluation: The BRAC experience. World Bank Conference on Evaluation and Poverty Reduction. Washington DC, USA. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.511. 9824&rep=rep1&type=pdf. Accessed on 31 March 2022. Ahsan, K. Z., Tahsina, T., Iqbal, A., Ali, N. B., Chowdhury, S. K., Huda, T. M., & Arifeen, S. E. (2017). Production and use of estimates for monitoring progress in the health sector: The case of Bangladesh. Global Health Action, 10(sup1), 1298890. Allen, C., Metternicht, G., & Wiedmann, T. (2018). Initial progress in implementing the Sustainable Development Goals (S.D.G.s): A review of evidence from countries. Sustainability Science, 13(5), 1453–1467. Aminuzzaman, S. M. (2013). Public policy processes and citizen participation in Bangladesh. In M. Sabharwal, & E. M. Berman (Eds.), Public administration in South Asia: India, Bangladesh, and Pakistan (pp. 213–236). CRC Press, Taylor & Francis. Asian Development Bank (ADB). (2012). The informal sector and informal employment in Bangladesh. https://www.adb.org/publications/informal-sec tor-and-informal-employment-bangladesh. Accessed on 30 August 2022. Asian Development Bank (ADB). (2021). Economic indicators for Bangladesh. https://www.adb.org/countries/bangladesh/economy. Accessed on 31 July 2021. As-Saber, S. N., & Rabbi, M. F. (2009). Democratisation of Upazila Parishad and its impact on responsiveness and accountability: An appraisal. Journal of Administration & Governance, 4(2), 53–71. Azad, A. K. (2020, October 13). A.P.A. and efficiency of public servants. The Financial Express. https://thefinancialexpress.com.bd/views/apa-and-eff iciency-of-public-servants-1605284817. Accessed on 12 October 2021.

3

EVALUATION IN BANGLADESH

117

Bangladesh Institute of Governance and Management (BIGM). (2021a). About us. http://www.bigm.edu.bd/Bigm/Home/About/47. Accessed on 1 October 2021. Bangladesh Institute of Governance and Management (BIGM). (2021b). 10-week long policy analysis course for public and private sector officials. http:/ /www.bigm.edu.bd/Bigm/Training/ShowSubSiteHeaderContent?MenuId= 1025&SubMenuID=48&SubSiteMenuID=52. Accessed on 1 October 2021. Bangladesh Road Transport Corporation (BRTC). (2021). http://www.brtc. gov.bd/. Accessed on 9 October 2021. Bdjobs eLearning. (2021). Participatory monitoring and evaluation. https:// elearning.bdjobs.com/trainingdetails.asp?TrainingId=96553&upcoming=0. Accessed on 1 October 2021. Bhuiyan, M. R. A., Mamur, H., & Begum, J. (2021). A brief review on renewable and sustainable energy resources in Bangladesh. Cleaner Engineering & Technology, 4, 1–18. BRAC. (2021a). BRAC Research and Evaluation Division. http://www.brac. net/sites/default/files/disability/RED.pdf. Accessed on 1 October 2021. BRAC. (2021b). Who we are. http://www.brac.net. Accessed on 1 October 2021. Cabinet Division (Government of the People’s Republic of Bangladesh). (2017). Rules of business 1996 (Revised up to April 2017). https://cabinet.portal.gov. bd/sites/default/files/files/cabinet.portal.gov.bd/legislative_information/ d8e711a8_0744_47d4_a1fd_fce1dfb1a5d7/Rules%20Of%20Business-2-199 60001.pdf. Accessed on 9 April 2022. Census and Economic Information Center (CEIC). (2022). Bangladesh foreign direct investment. https://www.ceicdata.com/en/indicator/bangladesh/for eign-direct-investment. Accessed on 1 April 2022. Center for Development Evaluatio.n (2021). About C.D.E. http://cde.com.bd/ about-cde/. Accessed 1 September 2021. Chattogram City Corporation. (2012). City Corporation Act 2009. https://www. ccc.org.bd/ccc_act2009. Accessed on 9 March 2023. Choudhury, S., & Basher, S. (2002). The enduring significance of Bangladesh’s war of independence: An analysis of economic costs and consequences. The Journal of Developing Areas, 36(1), 41–55. Chowdhury, A. M. R., Jenkins, A., & Nandita, M. M. (2014). Measuring the effects of interventions in BRAC, and how this has driven ‘development.’ Journal of Development Effectiveness, 6(4), 407–424. Citizen’s Platform for SDGs, Bangladesh. (2019). Four years of SDGs in Bangladesh and the way forward: Looking through the prism of non-state actors. Citizen’s Platform for SDGs., Bangladesh and Centre for Policy Dialogue. Commonwealth Local Government Forum. (2021). Bangladesh. https://www. clgf.org.uk/regions/clgf-asia/bangladesh/. Accessed on 31 July 2021.

118

M. F. I. WALI ET AL.

Comptroller and Auditor General of Bangladesh. (2020). Brief introduction. https://www.cagbd.org/about/brief-introduction. Accessed on 5 October 2021. Dhaka Tribune. (2021a). Report: Bangladesh among 3 top performers in sustainable development. Published 14 June 2021. https://archive.dhakatribune. com/bangladesh/2021/06/14/bangladesh-ranks-109th-in-sdg-index-2021. Accessed on 3 February 2022. Dhaka Tribune. (2021b). FDI to Bangladesh fell 11% last year amid global decline. Published 21 June 2021. https://www.dhakatribune.com/business/ economy/2021/06/21/fdi-to-bangladesh-fell-11-last-year. Accessed on 1 October 2021. Directorate General of Family Planning. (2021). Health population and nutrition sector program (HPNSP). https://dgfp.portal.gov.bd/sites/default/ files/files/dgfp.portal.gov.bd/notices/69123aa7_a552_48fd_951f_a39793b 45a61/4th%20sector%20%2030-7-18.pdf. Accessed on 3 April 2022. Directorate General of Health Services. (2013). HPNSDP PIP 2011–16. https:/ /dghs.gov.bd/index.php/en/mis-docs/important-documents/item/hpnsdp2011-16?category_id=115. Accessed on 1 September 2021. Directorate General of Health Services. (2016). A.D.P. progress monitoring system. https://www.dghs.gov.bd/index.php/en/e-health/our-ehealth-eservi ces/84-english-root/ehealth-eservice/491-adp-progress-monitoring-system. Accessed on 9 October 2021. Education Ministry. (2021). Education structure: Structure of the education system. http://www.shed.gov.bd/site/page/568b71f6-2811-4463-bf2cf3d71db59cef/Education-Structure-. Accessed on 1 September 2021. Faruque, A. A. (2009). Current status and evolution of industrial relations system in Bangladesh. International Labour Organization. https://ecommons.cor nell.edu/bitstream/handle/1813/87108/6CCBDB09d01.pdf?sequence=1& isAllowed=y. Accessed on 31 July 2021. General Economic Division. (2018). Monitoring and evaluation framework of Sustainable Development Goals (S.D.G.s): Bangladesh perspective. https:// www.sdg.gov.bd/public/img/upload/resources/5d353c9911996_doc_file. pdf. Accessed on 1 October 2021. Giménez, L., Jolliffe, D., & Sharif, I. (2014). Bangladesh, a middle income country by 2021: What will it take in terms of poverty reduction? The Bangladesh Development Studies, 37 (1&2), 1–19. Hamiduzzaman, M. (2014). Planning and managing of development projects in Bangladesh: Future challenges for government and private organisations. Journal of Public Administration and Policy Research, 6(2), 16–24. Helal, M., & Hossain, M. A. (2013). Four decades of economic development of Bangladesh: An assessment. Journal of the Asiatic Society of

3

EVALUATION IN BANGLADESH

119

Bangladesh (Hum.), 58(2), 335–362. http://www.ugc.gov.bd/#. Accessed on 1 September 2021. Holvoet, N., Dewachter, S., & Gildemyn, M. (2011). Strengthening national evaluation societies: An exploratory survey. Institute for Development Policy and Management (IOB). University of Antwerpen. Huda, T., Khan, J. A., Ahsan, K. Z., Jamil, K., & Arifeen, S. E. (2014). Monitoring and evaluating progress towards universal health coverage in Bangladesh. PLoS Medicine, 11(9), e1001722. Hussmanns, R. (2004). Measuring the informal economy: From employment in the informal sector to informal employment (Working paper No. 53). Policy Integration Department, International Labour Organisation. Geneva. http:// www.brac.net/sites/default/files/disability/RED.pdf. Accessed on 27 August 2022. IMED. (2019a). M&E policy study, Implementation, Monitoring and Evaluation Division (IMED), Bangladesh. https://imed.portal.gov.bd/sites/default/ files/files/imed.portal.gov.bd/page/6e936973_ca9e_495e_a95c_5b22d4c f4bc2/Policy.pdf. Accessed on 31 July 2021. IMED. (2019b). Monitoring & evaluation (M&E) guideline on education, health & nutrition, family welfare and social welfare sector projects of Annual Development Program, Implementation, Monitoring and Evaluation Division (IMED), Bangladesh. https://imed.portal.gov.bd/sites/default/files/files/ imed.portal.gov.bd/page/6e936973_ca9e_495e_a95c_5b22d4cf4bc2/Edu cation.pdf. Accessed on 1 October 2021. IMED. (2021). Sectors of IMED, Implementation, Monitoring and Evaluation Division (IMED), Bangladesh. https://imed.gov.bd/site/page/9aa0ac58b4e8-4a02-9b76-6bb6272441d7/Name-of-the-Ministrfy/Division-underIMED-Sectors. Accessed on 9 October 2021. IMED. (2022). Impact evaluation report. https://imed.gov.bd/site/page/f11 c522e-0ddd-49ec-908b-53eee46ba659/Impact-Evaluation-Report. Accessed on 31 August 2022. Institute of Professional Development Programs. (2021). Online certificate course on ‘Participatory Monitoring and Evaluation’. http://www.ipdpbd. com/training/212/online-certificate-course-on-participatory-monitoringand-evaluation-6-days-evening. Accessed on 1 October 2021. International Business and Technical Consultants, Inc. (2021). About us. https:/ /www.ibtci.com/about. Accessed on 1 September 2021. International Centre for Climate Change and Development (ICCCAD). (2022a). Our partners. https://www.icccad.net/our-partners/. Accessed on 9 April 2022. International Centre for Climate Change and Development (ICCCAD). (2022b). 14th short course. https://www.icccad.net/short-courses/previousshort-courses/14th-short-course/. Accessed on 9 April 2022.

120

M. F. I. WALI ET AL.

International Centre for Diarrhoeal Disease Research (ICDDR) Bangladesh. (2012). Training workshop on monitoring and evaluation builds capacity of mid level development professionals. Published 27 September 2012. https:/ /www.icddrb.org/news-and-events/news?id=592&task=view&mode=send. Accessed on 1 October 2021. Islam, F. (2021). Bangladesh among world’s top 10 in 13 sectors. The Prothom Alo. Published 27 March 2021. https://en.prothomalo.com/business/bangla desh-among-worlds-top-10-in-13-sectors. Accessed on 31 July 2021. Islam, M. R. (2019). Economic growth rates and exports of Bangladesh: The Bengal tiger? South Asia Research, 39(3), 285–303. Jeddere-Fisher, K., & Khair, S. (2004). Combined evaluation of ILO/IPEC garment sector projects as part of the “Memorandum of Understanding” framework with the Bangladesh Garment Manufacturers and Exporters Association. https://www.ilo.org/wcmsp5/groups/public/---ed_mas/---eval/doc uments/publication/wcms_171909.pdf. Accessed on 31 March 2022. Kabonga, I. (2020). Reflections on the Zimbabwean crisis 2000–2008 and the survival strategies: The sustainable livelihoods framework (S.L.F.) analysis. Africa Review, 12(2), 192–212. Kang, Y., Cho, M., Rahman, M. M., Cho, Y., Han, S., & Dutta, M. L. (2021). Design of a collaborative monitoring and evaluation system for a community-based nutrition project in rural Bangladesh. Evaluation and Program Planning, 84, 101892. Khondker, H. H. (2015). Bangladesh: History, culture and global diplomacy. Asian Journal of Social Science, 43(5), 635–647. KPMG. (2021a). Overview. https://home.kpmg/bd/en/home/about/ove rview.html. Accessed on 1 October 2021. KPMG. (2021b). Bangladesh investment guide. https://home.kpmg/bd/en/ home/insights/2021/09/bangladesh_investment_guide.html. Accessed on 1 October 2021. Laws of Bangladesh. (2019a). Zila Parishad Ain, 2000 [District Councils Act, 2000]. http://bdlaws.minlaw.gov.bd/act-details-841.html?lang=en. Accessed on 9 March 2023. Laws of Bangladesh. (2019b). Upazila Parishad Ain, 1998 amended 2009 [Sub District Councils Act, 1998, amended 2009]. http://bdlaws.minlaw.gov.bd/ act-827.html. Accessed on 9 March 2023. Laws of Bangladesh. (2019c). Sthanio Shorkar (Union Parishads) Ain, 2009 [Local Government (Union Councils) Act, 2009]. http://bdlaws.minlaw.gov. bd/act-details-1027.html. Accessed on 9 March 2023. Laws of Bangladesh. (2019d). Sthanio Shorkar (Pourashavas) Ain, 2009 [Local Government (Municipal Councils) Act, 2009]. http://bdlaws.minlaw.gov. bd/act-details-1024.html. Accessed on 9 March 2023.

3

EVALUATION IN BANGLADESH

121

Laws of Bangladesh. (2019e). Rangamati Parbotto Zila Parishad Ain, 1989 [Rangamati Hill District Councils Act 1989]. http://bdlaws.minlaw.gov.bd/ act-details-724.html. Accessed on 9 March 2023. Laws of Bangladesh. (2019f). Government educational and training institutions (adaptation) order, 1972 (President’s order). http://bdlaws.minlaw.gov.bd/ act-details-400.html. Accessed on 2 April 2022. Laws of Bangladesh. (2019g). Government educational and training institutions ordinance, 1961 (East Pakistan ordinance). http://bdlaws.minlaw.gov.bd/act316.html?hl=1. Accessed on 2 April 2022. Legislative and Parliament Affairs Division. (2019a). The Constitution of the People’s Republic of Bangladesh (Act No of 1972) part VIII the Comptroller and Auditor General. http://bdlaws.minlaw.gov.bd/act-367/part-206.html. Accessed on 9 October 2021. Mahmood, M., & Nurul Absar, M. M. (2015). Human resource management practices in Bangladesh: Current scenario and future challenges. South Asian Journal of Human Resources Management, 2(2), 171–188. Mahmood, S., Hort, K., Ahmed, S., Salam, M., & Cravioto, A. (2011). Strategies for capacity building for health research in Bangladesh: Role of core funding and a common monitoring and evaluation framework. Health Research Policy and Systems, 9(1), 1–9. Mahmud, W. (2008). Social development in Bangladesh: Pathways, surprises and challenges. Indian Journal of Human Development, 2(1), 79–92. Majid, M. M. (2008). Monitoring, evaluation and sustainability. A.H. Development Publishing House. Manchanda, S. (2017). Process evaluation of component III of Bangladesh water PaCT: Partnership for Cleaner Textile (Midline assessment) (English). Executive summary of evaluation Washington, DC, World Bank Group. http://documents.worldbank.org/curated/en/451681500888848643/Pro cess-evaluation-of-component-III-of-Bangladesh-water-PaCT-partnership-forcleaner-textile-midline-assessment. Accessed on 31 August 2022 Measure Evaluation. (2021). Bangladesh. https://www.measureevaluation.org/ countries/bangladesh.html. Accessed on 1 September 2021. Ministry of Finance. (2022). Statement III: Broad details of operating expenditure year: 2021–22. https://mof.gov.bd/site/budget_mof/4dfe57eb-e00b-4c4494da-cd7056a77b34/Statement-IIIBroad-Details-of-Operating-Expenditure. Accessed on 2 April 2022. Ministry of Health and Family Welfare. (2008). Objectives of Ministry of Health and Family Welfare. http://www.mohfw.gov.bd/index.php?option=com_con tent&view=article&id=73&Itemid=81&lang=en. Accessed on 1 September 2021. Misra, U. (2022). Bangladesh@50: From a ‘basket case’ to a case study, how the country put itself on the fast track. Indian Express. Published 4 January

122

M. F. I. WALI ET AL.

2022. https://indianexpress.com/article/express-sunday-eye/from-a-basketcase-to-a-case-study-how-bangladesh-put-itself-on-the-fast-track-7667663/. Accessed 2 April 2022. Moazzem, K. G., Halim, S. B. F., & Shafayat, M. (2018). Can a sustainable inspection system be developed? The case of RMG. sector in Bangladesh. Centre for Policy Dialogue. https://think-asia.org/bitstream/handle/11540/ 8884/PB-12-Can-a-Sustainable-Inspection-System-be-Developed-The-Caseof-RMG-Sector-in-Bangladesh.pdf?sequence=1. Accessed on 29 September 2021. Morton, S., Pencheon, D., & Squires, N. (2017). Sustainable Development Goals (SDGs), and their implementation: A national global framework for health, development and equity needs a systems approach at every level. British Medical Bulletin, 24(1), 1–10. National Academy for Planning and Development. (2019a). Brief history. https:/ /research.napd.ac.bd/?page=about-us. Accessed on 2 April 2022. National Academy for Planning and Development. (2019b). Development review. https://research.napd.ac.bd/?page=journal-proposal. Accessed on 2 April 2022. Neogy, T. K. (2014a). Evaluation of the companies’ performance: A study on mobile telecommunication companies in Bangladesh. American Journal of Trade and Policy, 1(2), 79–84. Neogy, T. K. (2014b). Evaluation of efficiency of accounting information systems: A study on mobile telecommunication companies in Bangladesh. Global Disclosure of Economics and Business, 3(1), 40–55. Neogy, T. K. (2015). Disclosure of financial statements: A study on mobile telecommunication companies in Bangladesh. American Journal of Trade and Policy, 2(3), 143–152. PaCT. (2022). What is PaCT . https://www.textilepact.net/what-is-pact.html. Accessed on 31 August 2022. Pricewaterhouse Coopers. (2015). The world in 2050: Will the shift in global economic power continue? Pricewaterhouse Coopers. Quibria, M. G. (2019). Bangladesh’s road to long-term economic prosperity: Risks and challenges. Palgrave Pivot. Rahman, M. L. (2019). Participatory monitoring and evaluation in development projects of Bangladesh. International Journal of Project Organisation and Management, 11(2), 93–109. Raihan, S. (2010). Informal sector in Bangladesh: Implication for growth and poverty. Indian Journal of Labour Economics, 53(2), 251–265. Reza, S. (2017). Issues, constraints and countermeasures of development through non-renewable natural resources: Bangladesh country study. Journal of Environmental and Agricultural Sciences, 13, 40–50.

3

EVALUATION IN BANGLADESH

123

Social Impact. (2021). What we do. https://socialimpact.com/about/. Accessed on 1 September 2021. The Financial Express. (2021a). Bangladesh attains Asia’s highest GDP growth in FY 2020–2021. Published 3 June 2021. https://thefinancialexpress.com. bd/economy/bangladesh-attains-asias-highest-gdp-growth-in-fy-2020-20211622714483. Accessed on 31 July 2021. The Financial Express. (2021b). Bangladesh universities’ poor ranking. Published on 16 June 2021. https://thefinancialexpress.com.bd/editorial/bangladeshuniversities-poor-ranking-1623859480. Accessed on 12 October 2021. The Independent. (2020). IFC launches web portal to monitor resource usage in R.M.G. sector. Published on 12 February 2020a. https://m.theindepe ndentbd.com/arcprint/details/236418/2020-02-12. Accessed on 1 October 2021. UN Development Programme (UNDP). (2021). The SDGs in action. https:/ /www.undp.org/sustainable-development-goals. Accessed on 4 September 2021. United Nations Children’s Fund (UNICEF), Huynh, U. K., & Ravishanker, S. (2019). Rohingya refugee crisis—How effective is UNICEF’s response? https://gdc.unicef.org/resource/rohingya-refugee-crisis-how-effect ive-unicefs-response. Accessed on 30 March 2022. University Grant Commission (UGC). (2021). About UGC. http://www.ugc. gov.bd/#. Accessed on 1 September 2021. Waheduzzaman, W., & As-Saber, S. (2015). Politics and policy in achieving participatory governance in a developing country context. Politics & Policy, 43(4), 474–501. White, H. (2005). Comment on contributions regarding the impact of the Bangladesh integrated nutrition project. Health Policy and Planning, 20(6), 408–411. World Bank. (2020). World Bank provides Bangladesh over $1 billion to create quality jobs and respond to COVID-19 pandemic. https://www.worldb ank.org/en/news/press-release/2020/06/19/world-bank-provides-bangla desh-over-1-billion-to-create-quality-jobs-and-respond-to-covid-19-pandemic. Accessed on 8 October 2021. World Bank. (2021a). The World Bank in Bangladesh. https://www.worldbank. org/en/country/bangladesh/overview#1. Accessed on 5 October 2021. World Bank. (2021b). Bangladesh receives $500 million from World Bank to improve road and digital connectivity in Western region. https://www.worldb ank.org/en/news/press-release/2021/02/01/bangladesh-receives-500-mil lion-from-world-bank-to-improve-road-and-digital-connectivity-in-westernregion. Accessed on 8 October 2021.

124

M. F. I. WALI ET AL.

World Bank Group. (2020). Bangladesh country program evaluation. Published on 8 July 2020. https://ieg.worldbankgroup.org/sites/default/files/Data/ reports/ap_bangladeshcpe.pdf. Accessed on 2 April 2022. Wyatt, A. (2021). Strengthening parliamentary oversight in Bangladesh. Oxford Policy Management. https://www.opml.co.uk/projects/strengthening-parlia mentary-oversight-bangladesh. Accessed on 12 October 2021.

CHAPTER 4

Evaluation in China Wenxuan Yu

1

General Country Overview

In China, evaluation, “Ping Gu”, literally means “Comments” (Ping) and “Estimates” (Gu). Evaluation is a very elusive concept in the practice of public administration in China. In addition to being a noun, as a verb in Chinese, its connotation comprises various activities such as “measure”, “criticize”, “estimate”, and “feedback”. Evaluation as an essential part of governmental activities can be traced back to the Shang Dynasty, more than 3000 years ago. In the People’s Republic of China, established in 1949, evaluation of the Chinese government’s systematic efforts in modernising its government institutions and enhancing its governmental competence started in the late 1970s, when China initiated its “reform and open-up” policy (Burns & Zhiren, 2010). Now evaluation activities conducted by the Chinese government comprise three types of activities. The first is the performance appraisal in governmental human resource management. After the 3rd Plenary Session of the 11th Central Committee of the Chinese Communist

W. Yu (B) School of Public Affairs, Xiamen University, Xiamen, China e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_4

125

126

W. YU

Party, which convened in 1978, the ruling party, the Chinese Community Party (CCP), decided to explore various measures to motivate its civil servants (most of them are party members) and their organizations to develop its market economy and pursue social and economic prosperity. One of the key measures is to establish a nation-wide performance appraisal system. In November 1979, the Organisation Department of the Central Committee of CCP issued the “Opinions on the Implementation of the Cadre Appraisal System”, explicitly aiming to “encourage the advanced, motivate the backward and mobilize the enthusiasm of cadres”, and mandated that performance appraisal should mainly focus on civil servants’ direct and indirect contributions to the modernisation drive. The second is the performance management system at the organisational level, the well-known “Target-Responsibility System” (TRS). The TRS starts from the organisational performance goals (‘targets’) defined by the superior party and government offices and subsequently decomposes and cascades down those targets to the lower levels of government such as (province, city, county, and village). At year-end, the superior party and government offices will evaluate the performance of its subordinates according to the extent to which they meet the preset performance targets. Rewards and punishments (budget appropriation, performance bonus, career promotion, etc.) will be granted to government agencies, their leaders, and employees accordingly. The third is fiscal auditing, which refers to the activities of government audit offices in inspecting accounting statements, financial accounts, and relevant materials for ensuring the authenticity, legality, and efficiency of fiscal revenues and expenses. In China, the government auditing system is managed by a matrix structure. In general, the auditing system is within the executive branch, which is different from most Western countries. The National Audit Office (NAO) is under the State Council. Local audit offices (at Provincial, Municipal, and County levels) are led and managed vertically (the superior audit offices) and horizontally (local governments) at the same time. According to the Audit Law of the People’s Republic of China, the State Council and the local governments at or above the county level should produce annual reports by their relevant audit offices on budget implementation and other government revenues and expenditures. The audit reports will be presented in the national or local People’s Congress for monitoring and checking. The national and local People’s Congress will inquire about the wrongdoings and any problems of government agencies according to the problems identified by

4

EVALUATION IN CHINA

127

the annual auditing reports and have legal rights to mandate government offices and their leaders to rectify their problems. Scholars of political science and economics have touted the contributions of the three evaluation systems/activities to China’s stunning economic development in the past 40 years (Chan & Gao, 2008; Li & Zhou, 2005; Liu & Li, 2016).

2 Institutional Structures and Processes (Political System) The current political regime, the People’s Republic of China (PRC) was established in 1949. The government of the PRC is a unitary system consisting of the central government and the local governments, governing a land of 9.6 million square metres and a population of 1.4 billion. The local governments in China consist of five levels below the central government. They are provinces, prefectures, counties, townships, and villages. The provincial level encompasses 22 provinces, 5 autonomous regions, 4 municipalities under the direct control of the central government (Beijing, Shanghai, Tianjin, and Chongqing) and 2 Special Administrative Regions (Hong Kong and Macau). According to the Constitution of PRC (1982), the legislative branch of the government is the People’s Congress; the executive branch is the administrative system (the central and local governments in the narrow sense); the People’s Court and procuratorate are the judicial branch. At the central government level, The National People’s Congress (NPC) is the supreme legislature, the State Council is the central executive body, and the People’s Supreme Procuratorate and Court are judicial. PRC is a party state, which means although the State Council and the People’s Supreme Protectorate and court report to the NPC and its Standing Committee, all the three branches are under the leadership of the CCP and its central committee. The same structure is mirrored at the levels of province, municipality, and country. As required by the Constitution of the People’s Republic of China, The NAO was set up in 1983 as part of the State Council within the executive branch, which is very different from its counterpart in Western countries. Local audit offices were also established accordingly. A local audit office is subject to the management of both the audit office at a higher level and its supervising local government.

128

W. YU

2.1

Evaluation Regulations

Currently, in China, there are no universal laws or regulations on evaluation like the Government Performance and Results Act of 1993 in the U.S. However, in the area of performance appraisal and performance management, there are a series of administrative decrees and regulations enacted and implemented by the central government and the CPC. Some important administrative decrees and regulations are listed in Table 1. These administrative decrees and regulations contain not only principles or guidelines for performance appraisal or performance management, but, more often than not, also very detailed evaluation measures. Due to the top to down unitary system, while these administrative decrees and regulations are promulgated and enacted by the central government, it is also mandatory for local governments. Local governments promulgated their local implementation guidelines of these central government administrative decrees and regulations accordingly. 2.2

Parliamentarian and National Audit Structures

Even though the legislature of the PRC is the People’s Congress, which is constitutionally supposed to be the supreme state organ. However, the People’s Congress had been criticised to be a “rubber stamp” and did not play its role to check and balance administrative power (Truex, 2014). In the past, there were no official and institutionalised measures for the People’s Congress to “evaluates” government officials and government agencies. Performance appraisal and management were mainly carried out within the executive branch. The People’s Congress did organise its representatives to visit government agencies to “inspect” their operation. But these activities were not regular and institutionalised. Their main goals were not to evaluate the performance of government agencies. Instead, their purposes were to enrich People’s Congress Representatives’ understanding of how government agencies work and what they could do to help their operation. In addition, in the annual People’s Congress Convention, People’s Congress representatives symbolically discussed and approved the annual government report. Although they provided some comments or opinions on the performance of government officials or agencies, those feedbacks were only for reference and did not have much substantive influence on government appraisal and management system. Since the 1990s, as part of the efforts to modernise its government

4

EVALUATION IN CHINA

129

Table 1 Administrative and party decrees and regulations on performance appraisal and management Year

Administrative and party decrees and regulations

Enacted by

1979

The Opinions on the Implementation of the Cadre Performance Appraisal Institution The Notice on Establishing the Post Responsibility System for Personnel of the State Administrative Organs The Notice on Gradually Implementing the Post Responsibility System of Government Organs

The Organization Department of the Central Committee of CCP The Ministry of Labor and Personnel

1982

1984

1995

1996

2000

2002

2006

2009

2012 2015 2015

The Interim Regulations on the Selection and Appointment of Party and Government Leading Cadres The System of the methods and standards of the performance appraisal for the Members of the County Level Party and Government Leading Groups The Outline of Deepening the Reform of Cadre and Personnel Management Institution The Regulations on the Selection and Appointment of Party and government leading Cadres The Trial Measures reflect the requirements of the scientific outlook development for Comprehensive Assessment of Local Party and Government Leading Groups and Cadres The Opinions on the Establishment of an Assessment and Evaluation Mechanism for Party and Government Leading Groups and Cadres for Promoting Scientific Development The Quality Development Outline (2011–2012) The Outline for the Implementation of Government of Rule of Law (2015–2020) The Planning for the Development of the National Standard System (2016–2020)

Source Author’s own work

The Organization Department of the Central Committee of the CCP and the Ministry of Labor and Personnel The Central Committee of CCP

The Organization Department of the Central Committee of CCP

The General Office of the Central Committee of the CCP The Central Committee of CCP

The Organization Department of the Central Committee of CCP

The General Office of the Central Committee of the CCP

The State Council The State Council The State Council

130

W. YU

system, control the abuse of administrative power and prevent rampant corruption, the Central Committee of the CCP decided to ramp up the People’s Congress to strengthen its legislative power. On December 29th, 1998, the 6th meeting of the Standing Committee of the 9th NPC established its Budget Affairs Commission. As one of the working bodies of the Standing Committee of the National People’s Congress, the Budget Affairs Commission assists the Financial and Economic Affairs Committee of the NPC to inspect and examine the fiscal budget and financial accounts and monitor the execution of the fiscal budget. Approved by the leaders of the Standing Committee, the Commission also can launch an investigation on the suspicious misuse of the fiscal budget of administrative agencies and earmarked funds for specific government projects and programmes. At year-end, national government agencies are mandated to submit their auditing report prepared by the NAO to the Commission for approval. The Commission will require the reporting agencies to rectify identified problems and wrongdoings. This structure is mirrored in local government at the county level above. A local People’s Congress also has its budget affairs commission serving similar functions for the local government. The NAO was set up in 1983, required by the 1982 Constitution of the People’s Republic of China. Different from the Western countries, in which NAO is part of the legislative branch, in China the NAO is a department of the State Council, part of the administrative branch. The NAO manages a network of branch offices in central government ministries and agencies and very large municipalities to audit relevant government departments, agencies, and state-owned enterprises under the control of the central government. These branch audit offices are located in ministries and very large municipalities such as Guangzhou and Shanghai, under the direct leadership of the NAO. A local government audit office was set up at the county level above accordingly. It is intriguing for Western observers that a local government audit office serves two bosses, its upper-level audit office, and the local government. Under the dual leadership system, a higher-level audit office provides the local audit office with auditing and regulatory guidance and professional support, while the local government provides resources and exercises personnel control. The dual management system of the Audit offices in China has been criticised by scholars due to its lack of independence (Chong, 2010; Lin & Chan, 2000). However, the advantages of the system are also

4

EVALUATION IN CHINA

131

recognised by scholars and practitioners and considered suitable for the current Chinese political and administrative system (Cao et al., 2022). For keeping a balance between the pros and cons, some scholars called for a “Dual-Track System” of government audit offices in China (Yang et al., 2008). 2.3

Organisational Structure

In addition to the government audit system, the performance appraisal and performance management system are managed by two systems, the party system, and the executive system (government in its narrow sense). It should be noted that China is a party state. The two systems are seamlessly integrated, and the party system leads the executive system in every aspect. As for human resource management, within the Departments of the Party at each level from the central down to the county level, the Organization Department oversees the selection, promotion, and attrition of officials with a leadership position. The Discipline Depart proactively investigates corruption or the wrongdoings of party members and officials with leadership roles or reactively investigates them according to the whistleblowers’ reports. The Personnel Department in the executive branch oversees managing human resource matters and evaluating public officials’ performance regularly. As for performance management, partially driven by the New Public Management (NPM) movement, in the 1990s, the Chinese government adopted its performance measurement-based performance management system, Targets and Responsibilities System (TRS). Mainly according to the 5-year social and economic plan, the central government decomposes and cascades down performance indicators to its ministries, agencies, and local governments. The central government ministries and agencies evaluate its line offices’ performance and upper-level government evaluates its direct lower-level government’s performance intermittently and annually according to the preset performance indicators (“targets”). The results of the evaluation will be used for the reward and punishment of local government leaders, agency heads, local government as a whole, and individual administrative agencies and offices. Organisations involved in performance activities include the Department of Finance, the Department of Personnel, the Audit Office, and the General Office

132

W. YU

of Government. At the central government level, within each administrative department, there is a performance management office or a Leading Small Group (LSG) in charge of managing performance appraisal and performance management-related matters. Although it is not a national practice, in some provincial or municipal governments such as Hangzhou City in Zhejiang province, a designated performance management office is established for organising and managing performance management activities. In some local governments such as local governments in Fujian Province, an office in charge of performance management is set up within the Discipline Department of the Party (Liu & Li, 2016). 2.4

Evaluation Practice and Use

Chinese government adopts a performance appraisal system that consists of four categories of standards, Diligence (Qin), Ethics (De), Competence (Neng), Performance (Ji). Each category includes a myriad of qualitative and quantitative measures. Government employees are subject to performance evaluation on a regular or ad hoc basis. Although scholars have been questioning the effects of performance appraisal on performance rewards and attrition, recent studies show that in addition to political connections, public officials’ performance does matter for their promotion (Li & Gore, 2018). Chan and Gao provide a detailed account of the performance indicators of the TRS. According to their 3 years of fieldwork, their study shows that in Zhouzhi County of Shanxi Province, an exemplar of the implementation of the TRS system in China, the County Party Committee and County Government sign performance contracts with the leadership corps of the party departments, the departments of the county government and municipalities within their jurisdiction. Within the performance contract there are two types of targets, functional targets and common targets. Functional targets are those pertinent to the individual department’s tasks. More than one thousand indicators have been developed in the category. Functional targets include indicators on public service delivery, and promoting economic development. Developing CCP’s capacity, bureaucratic institution-building, promoting a stable, safe, and harmonious society, and other targets deemed to be necessary by the party and the government (Chan & Gao, 2008). The common targets consist of indicators all party and government departments and agencies are mandated to meet. These indicators often

4

EVALUATION IN CHINA

133

include to what extent local officials can handle public complaints, maintain comprehensive social security, ensure production safety, population control, construct spiritual civilisation, party building, and anticorruption. Chan and Gao also found out that Zhouzhi country prioritised four common goals, population control, product safety, handling of public complaints, and maintaining social security. These goals are commonly known as “veto”, which means if officials fail to achieve any of these targets, they will fail in the annual evaluation, no matter how well they have been performed on other targets. Studies have shown that the “veto” targets have become the dark side of the performance management system in China and stimulate public officials’ gaming behaviour (Chan & Gao, 2008). In China, a government audit office performs five types of activities. The first is budget auditing which started in 1996 as required by the Budget Law and the Audit. The purpose of budget auditing is to assure the legality of fiscal budget implementation, the reliability of its outcomes, and the effectiveness of budgetary control at all levels of government. The government audit office is required to report the budget audit results to the government body of which it is a part and, if necessary, to a higherlevel government audit office. After being examined and approved by the government, the government audit office must also report the findings of the budget audit to the standing committee of the People’s Congress. The standing committee of the People’s Congress will then review these findings and mandate the pertinent government offices to explain and rectify their problems. The second main category of a government audit is economic responsibility auditing. This is required by the Regulation on Auditing the Economic Responsibilities of Party and Government Officials at or above the county level and the Regulation on Auditing the Economic Responsibilities of State-Controlled-Enterprise Managers. The audited are party officials, government officials, or managers of state-owned enterprises. The audit aims to what extent these individual officials or managers are performing their official duties. The audit report is mainly used by the disciplinary and organisational departments of the Party and government for selection and promotion purposes. The third is the audit of special-purpose funds, including poverty relief funds, government bonds, social welfare funds, railway construction funds, and basic education funds. The fourth type of government audit is financial services auditing, including the audit of banks, insurance

134

W. YU

companies, and international loans. The final main type of government audit is the audit of state-owned enterprises (Yang et al., 2008).

3 Societal Dissemination/ Acceptance (Social System) 3.1

Use of Evaluations by Civil Society

China is considered an authoritarian regime. As part of its modernisation reform, government evaluation activities only began in the 1990s. Before the fourth round of China’s administrative reform in 1998, the central government’s focus was central planning and bureaucratic command-andcontrol. After 1998, the Chinese central government began to reform its administrative system to establish an effective and responsive government. Partially influenced by the NPM Movement, performance management as an administrative and policy instrument gradually attracted the Chinese government’s attention. In March 2008, Premier Wen Jiabao stated in his report to the 11th NPC that the Chinese government would introduce performance management. Performance management began to be implemented on a trial basis. In 2011, the institutions of the Joint Conference on Performance Management were set up at the central level and the General Office of Performance Management was established under the Ministry of Supervision. Thus, the policy instrument of performance management was under formation (Liu & Li, 2016). To a large degree evaluation activities in China are government internal activities. For citizens and nonprofit organisations, the content, the procedure, and the results of government evaluation activities are still in a black box. In 2007, the Chinese government enacted the Regulations on Open Government Information of the People’s Republic of China mandated central government and local government to release government information to the public. Since then, more and more government information has been released. There is also an increasing trend for civil society to request government information. However, there is still a long way to go for the open government information movement. Information released and requested by civil society is mainly on public service delivery and the outcomes of public projects and programmes. The internal performance appraisal and management results are kept within the government and considered confidential (Lu, 2021; Yu, 2011).

4

EVALUATION IN CHINA

135

In 2001, the NAO issued a Standard on the Disclosure of Auditing Results, asking government audit offices to disclose the audit results to the public if it is requested by the same-level government or the upperlevel government audit office. In June 2003, the NAO published its first audit report to the public and revealed the problems it identified in its audit process. The report for the first time specifically pointed out the wrongdoings of four central government ministries. In subsequent years, 2004 and 2005, the NAO published a series of Statements of Auditing Results, exposing illegal and wasteful fiscal and managerial activities in more than 30 central government departments or other national stateowned enterprises. The 2003 audit report released to the public by NAO was reported by all major media outlets and made a national stir. Journalists coined the term, “Audit Storm” to describe its impact on civil society’s perception of the central government’s performance. Although the NAO has begun to continually release its audit report to the public, research has shown that the audit reports were only able to reveal about 30% of the identified problems (Mo, 2004). In addition, local audit offices find it very difficult to release their audit report to the public because local audit offices rely on the local government for financial and human resources. 3.2

Public Perception and Discourse on Evaluation and Evaluation Results

China had been ruled by Emperors for more than 3000 years. “Rule by Secrecy” has already become part of its national culture. Since the establishment of the People’s Republic of China in 1949, the legitimacy of the revolutionary regime has been based on the performance of the ruling party, the Chinese Communist Party. The performance-based legitimacy is vividly manifested by the public discourse from “overthrow the three great mountains of imperialism, feudalism and bureaucrat-capitalism” to the pre- “Reform and Open up” policy to the “Three representatives” in the administration of President Jiang Zemin. Even though Chinese people experience the pains of the Great Leap and the Culture Revolution, the stunning economic development and the improvement of their life, particularly after the late 1970s, have persuaded most of the Chinese people the current ruling party and political institutions are their best choice. In 2012, the current President Xi Jinping promised to eradicate extreme poverty in China. Over eight years, nearly 100 million people

136

W. YU

have been lifted out of poverty. The fact that the Chinese government successfully led its people to escape the poverty trap has been touted as a miracle by international scholars (Ang, 2016). Largely due to the performance-based legitimacy, the Chinese government has ranked as the highest political trusted government in the world, even with rampant corruption in the 2000s, as the World Value Barometer Survey indicated (Hutchison & Xu, 2017). The traditional culture of “rule by secrecy” and the performance-based legitimacy, to a large degree, have fostered a culture of pragmatism. For the general public, the best evaluation results of the Chinese government are outcome-based, namely their quality of life. As long as the Chinese government can continuously improve their wellbeing, the general public does not care much about how the performance of the Chinese government is evaluated. In some areas related to essential public services, some evaluations are well known to the public. The two most salient evaluation areas are public education and environmental protection. University Rankings conducted by the Ministry of Education, newspapers, and university research institutions are hotly discussed in public when they are released. Rankings of environmental pollution have been released by the Ministry of Ecology and Environment on a daily basis. Evaluations related to the quality of life issues are discussed in the media. Every five years, the Ministry of Education released its evaluation reports on Chinese public universities. The reports rank Chinese universities according to academic disciplines and subjects, respectively. The results have profound impacts on the reputation of the universities and resource allocation and faculty recruitments. 3.3

Participation of Civil Society in Evaluations

Evaluation in China is largely government internal activities. Recently Chinese government has integrated a citizen participation component in their administrative, personnel and programme evaluation activities. But they are still government-led and controlled. The TRS is an overarching evaluation form that integrates all three types of government evaluation activities, performance appraisal, performance management, and government audit. It is the dominant form of government performance management in China. The TRS system has effectively and significantly improved the hierarchal control of policy implementation, motivated

4

EVALUATION IN CHINA

137

public officials, and enhanced hierarchical accountability (Walker & Wu, 2010). Economists have attributed China’s stunning economic development to this system (Li & Zhou, 2005). However, the TRS has also inherited dysfunctions deeply rooted in China’s political culture, unitary and authoritarian political institutions. Compounded by the pecking order of strict and rigid performance appraisal and performance management mechanisms and China’s Guanxi (social relation) culture, TRS induces pernicious gaming behaviour of public officials such as goal displacement, tunnel vision, sub-optimisation, and cheating (Gao, 2015). In addition, due to the insufficiency of checks and balances between the legislative, judicial, and executive branches and the strict top-down carrotand-stick incentive system, on the one hand, TRS strengthened the top to down hierarchical control and accountability, on the other hand, it overlooked the horizontal accountability. The myriad of political, social, and economic problems caused by the TRS’ overemphasis on economic growth and social stability (O’brien & Li, 2017; Shukai, 2007). The problems of the TRS system had been noticed and addressed by the Chinese government in the late 1990s. To complement the top to down TRS, the Chinese government has been inviting citizens, mass media, and academic and nonprofit organisations to assess its performance and then incorporate their evaluation results into the TRS system. For example, in the 1990s, local governments such as Zhuhai City, Nanjing City, and Shenyang City, initiated citizen survey projects. Government officials conducted face to face interviews with more than 10,000 residents and businesses to obtain their opinion on government performance and public service quality. In addition, local governments also employ professional survey companies, media, nonprofit organisations, or universities to design performance measurement metrics, and surveys, conduct citizen surveys and analyse performance data (Yang & Wu, 2013). However, some scholars criticised these practices as “a manoeuvre by government leaders at the top echelon to enhance the vertical accountability of subordinate bureaucrats. It shares the spirit of the past practice of combating bureaucratism by mobilizing the masses during the prereform era” (So, 2014, p. 2). After examining the “Democratic Review of Administrative and Business Style” project in Wuhan, So (2014) found that this third-party evaluation project, initiated and dominated by the government, failed to measure real government performance and was, in essence, a managerial rather than a democratic means to strengthen hierarchical control.

138

W. YU

The so-called “third-party” evaluation activities are emerging in China. But their independence has been questioned. Against the background, in the early 2000s, nongovernment entities including academic institutions, mass media, think tanks, and consulting firms began to initiate and carry out their own evaluation projects to assess government performance. Public administration scholars in China have documented notable cases and practices such as the evaluation of provincial departments and prefecture-level city government in northwestern Gansu Province in 2003, 2005 and 2006 carried out by Lanzhou University. In 2007 Oriental Outlook Magazine began its nationwide online survey to identify the “Happiest Cities in China”. A private consulting firm, The Horizontal Research Consultancy Group, has been continually publishing reports on citizen satisfaction with public service delivery according to the citizen interviews they conducted across key Chinese cities. Yu and Ma (2015) developed a framework to evaluate 11 “third-party” evaluation projects in China and found out that while these projects have great potential to supplement the exiting TRS and improve government transparency and external accountability, the independence and the quality of these assessments need to be further improved (Yu & Ma, 2015).

4 Professionalisation of Evaluation (System of Professions) 4.1

Academic Study Courses and Training Practices

There are no universities and research institutes specialising in evaluation in China. However, there are schools of public management and public policy providing academic courses at undergraduate, graduate, and PhD levels and short-term professional training for public officials. Performance and project evaluation courses are mandated courses in academic degree education. The Undergraduate degree programmes in Public Administration were established in the early 1980s. The professional master’s degree programmes, Master of Public Administration (MPA), were established in 2001. The first batch of graduates with MPA degrees graduated in 2005. Till 2019, there were 241 MPA programmes in China with 9029 full-time faculty members and 4782 adjunct faculty members. In 1998, Renmin University in Beijing, Sun Yat-sen University in Guangzhou, and Fudan University in Shanghai were approved by the Ministry of

4

EVALUATION IN CHINA

139

Education to offer a doctoral degree in Public Administration. After more than 20 years’ development, so far 50 universities offer PhD degree programmes in public administration. Courses on government performance evaluation and appraisal are mandatory in most of these programmes. In China, there is a university that especially provides training in audit, which is the Nanjing Audit University jointly established by the Ministry of Education, the Ministry of Finance and the NAO, and the Jiangsu Provincial government. Currently, it has 18,000 students and more than 1000 full-time faculty members. Graduate and undergraduate students study subjects such as audit, accounting, finance, and management. Courses on evaluation are an essential part of their curriculum. 4.2

Journals and Communication Platforms

In China, in the field of public administration, there are three key journals, specialised in publishing academic articles on public administration. Public performance appraisal and management are two key areas of research published in the three journals. Chinese Public Administration Review,1 is the official journal of the Chinese Public Administration Society (CPAS), and can be seen as a counterpart to the journal Public Administrative Review in the U.S. The Journal of Public Management,2 is a quarterly journal published by the School of Management of Harbin Institute of Technology. It is touted as the best and most influential academic journal in the field. The Journal of Public Administration is a bi-monthly journal published by the School of Politics and Public Affairs, Sun Yat-sen University in Guangzhou.3 Moreover, in China, there are two evaluation-specific journals, the Journal of Higher Education Development and Evaluation, and Appraisal Journal of China. The former is an academic journal in the field of education, while the latter is a practice-oriented professional journal published by the China Appraisal Society, in the area of financial assets management. The annual conference of CPAS, the counterpart of the annual conference of the American Society for Public Administration (ASPA), is

1 http://www.cpaj.com.cn/. Accessed on 23 January 2023. 2 http://som.hit.edu.cn/qkhy/xyqk/ggglxb1/qkjj.htm. Accessed on 23 January 2023. 3 https://jpa.sysu.edu.cn/index1.htm. Accessed on 13 February 2023.

140

W. YU

the largest conference on public administration in China. Performance appraisal, performance management, government finance, and audit are key themes of the conference every year. Different from hundreds of academic conferences organised by academic institutions, the conference convenes thousands of academic scholars and government officials from the central government and local governments all over China together for exchanging information on the latest policy development, best practices, and cutting-edge research. 4.3

Professional Organisations

The Chinese Public Administration Society (CPAS) is a national professional organisation in China, the counterpart of the American Society for Public Administration (ASPA). However different from ASPA, CPAS is a governmental organisation, part of the State Council. CPAS’ president is a senior official of the central government, normally the Director of the General Office of the State Council. Chinese Public Administration Review is its official journal. The high-level government status of the CPAS enables CPAS to have profound practical influence. Its subscription is mandated by the State Council, and it is put on the desks of senior public officials in the central and local governments all over China. CPAS has a section, performance management research, specifically focusing on performance appraisal and management. 4.4

Compliance with Quality Standards and Obligations

As mentioned in the previous section, there is a myriad of laws, ruling party guidelines, government decrees, and implementation methods for evaluation activities including performance appraisal, performance management, and performance audit, at the Central government and local government levels in China. These standards and guidelines are imposed and enforced by the Chinese government and the ruling party. The CPAS also publishes its Charter and “Codes of Conduct” to encourage compliance and promote the quality and professionalism of its member organisations and members including the Section on Performance Management Research.

4

5

EVALUATION IN CHINA

141

Conclusions

As aforementioned, evaluation is an elusive concept in China. Its activities permeate every part and every level of Chinese government from the central to the local. This chapter focuses on the three types of evaluation activities the Chinese government is carrying out on a daily basis. The three types of evaluation activities, performance management, performance appraisal, and performance audit, have been extensively studied in the field of Chinese public administration. In China, the evaluation of government performance is still mainly an internal and top to down administrative activitiy within the executive branch. The involvement and the impacts of the legislative branch and civil society on government evaluations are very limited. In recent years, as part of its efforts to modernise its government system and improve its governmental competences, Chinese government has begun to further institutionalise evaluation activities and kept emphasising the importance of evaluation activities for reducing rampant corruption, improving administrative control and responsiveness, and enhancing internal and external accountability. To address the drawbacks of internal evaluation system, Chinese government is encouraging societal stakeholders to initiate and carry out evaluation activities on government performance. However, to what extent these external evaluation activities can truly influence government decision-making and policy implementation needs to be further studied. Recently Chinese government began to emphasise the use of IT technologies including big data, artificial intelligence, and cloud storage, etc., to streamline its administrative system, reduce administrative costs, strengthen hierarchical and social control. Empowered by IT technology and big data, evaluation activities are carried out in a more frequent and efficient way. However, a more accurate and more efficient data collection and analysis system for evaluation may discourage the willingness of the Chinese government to tolerate and foster external government performance evaluation organisations and activities, which may produce negative impacts on the democratisation process in China.

References Ang, Y. Y. (2016). How China escaped the poverty trap. Cornell University Press.

142

W. YU

Burns, J. P., & Zhiren, Z. (2010). Performance management in the government of the People’s Republic of China: Accountability and control in the implementation of public policy. OECD Journal on Budgeting, 10(2), 1–28. Cao, H., Zhang, L., Qi, Y., Yang, Z., & Li, X. (2022). Government auditing and environmental governance: Evidence from China’s auditing system reform. Environmental Impact Assessment Review, 93, 106705. Chan, H. S., & Gao, J. (2008). Performance measurement in Chinese local governments: Guest editors’ introduction. Chinese Law & Government, 41(2), 4–9. Chong, G. (2010). Legal cases and auditing in China. In Satyendra Singh (Ed.), Handbook of business practices and growth in emerging markets (pp. 101–116). World Scientific. Gao, J. (2015). Pernicious manipulation of performance measures in China’s cadre evaluation system. The China Quarterly, 223, 618–637. Hutchison, M. L., & Xu, P. (2017). Trust in China? The impact of development, inequality, and openness on political trust across China’s provinces, 2001– 2012. Asian Journal of Comparative Politics, 2(2), 176–195. Li, H., & Gore, L. L. (2018). Merit-based patronage: Career incentives of local leading cadres in China. Journal of Contemporary China, 27 (109), 85–102. Li, H., & Zhou, L.-A. (2005). Political turnover and economic performance: The incentive role of personnel control in China. Journal of Public Economics, 89(9–10), 1743–1762. Lin, K. Z., & Chan, K. H. (2000). Auditing standards in China—A comparative analysis with relevant international standards and guidelines. The International Journal of Accounting, 35(4), 559–577. Liu, W., & Li, W. (2016). Divergence and convergence in the diffusion of performance management in China. Public Performance & Management Review, 39(3), 630–654. Lu, Y. (2021). Negotiating the right to information: Citizen-government interactions in the implementation of the regulations on open government information in China. Europe-Asia Studies, 73(1), 81–100. Mo, Y. (2004, July 19). Aftermath pondering on the audit storm. China Economic Weekly (Zhongguo Jingji Zhoukan). O’brien, K. J., & Li, L. (2017). Selective policy implementation in rural China. In Critical readings on the Communist Party of China (4 Vols. Set, pp. 437– 460). Brill. Shukai, Z. (2007). The accountability system of township governments. Chinese Sociology & Anthropology, 39(2), 64–73. So, B. W. Y. (2014). Civic engagement in the performance evaluation of the public sector in China: Building horizontal accountability to enhance vertical accountability. Public Management Review, 16(3), 341–357.

4

EVALUATION IN CHINA

143

Truex, R. (2014). The returns to office in a “rubber stamp” parliament. American Political Science Review, 108(2), 235–251. Walker, R. M., & Wu, J. (2010). Future prospects for performance management in Chinese city governments. Administration & Society, 42(1_suppl.), 34–55. Yang, S., Xiao, J. Z., & Pendlebury, M. (2008). Government auditing in China: Problems and reform. Advances in Accounting, 24(1), 119–127. Yang, Y., & Wu, J. (2013). Are the “bigger fish” caught? China’s experience of engaging citizens in performance measurement system. Public Administration Quarterly, 37 , 144–182. Yu, W. (2011). Open government information: Challenges faced by public human resource management in China. International Journal of Public Administration, 34(13), 879–888. Yu, W., & Ma, L. (2015). External government performance evaluation in China: Evaluating the evaluations. Public Performance & Management Review, 39(1), 144–171.

CHAPTER 5

Evaluation in India Rashmi Agrawal, Banda Rao, and Rajib Nandi

1

Introduction 1.1

Governance

India has a long historical and cultural past stretching over a few millennia before CE. Administratively, it is a union of 28 States and 8 Union Territories, and is, according to its Constitution, a sovereign, socialist, secular and democratic Republic, with a parliamentary system of government. The President is the Head of the State, while the Prime Minister is

R. Agrawal (B) · B. Rao · R. Nandi Evaluation Community of India, New Delhi, India e-mail: [email protected] B. Rao e-mail: [email protected] R. Nandi e-mail: [email protected] R. Agrawal Asia Pacific Evaluation Association Board, Bulacan, Philippines

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_5

145

146

R. AGRAWAL ET AL.

the head of the Union (or Central) Government, and runs office with the support of a Cabinet of ministers. The Cabinet is responsible to the bicameral Legislature which comprises the Lok Sabha (House of the People) and the Rajya Sabha (Council of States), the two together forming the Parliament. The Supreme Court of India is the apex judicial institution in the country. Generally, the State governments have similar institutions of executive, legislature and judiciary. The Union government can only legislate on some subjects like defence and foreign affairs. Most of the other subjects are in the domain of the State governments only, while both governments can legislate on the rest. There is also a third tier of governance in the form of local administrations for cities, towns and villages, in the form of corporations, municipalities and village panchayats. The country is home to an estimated population of 13.80 billion (United Nations, 2020), currently growing at one per cent per annum, professing different religious faiths, speaking different languages and following different socio-cultural traditions and ethos. 1.2

Monitoring and Evaluation in India—An Overview

Monitoring and evaluation in different spheres of public life have been in practice in India for years. Monitoring progress and performance is particularly common, and is supported by elaborate management information systems (MIS) in various sectoral programmes, through administrative arrangements and executive instructions, and the national statistical system. Some of the monitoring systems provide daily information and even real time, for example on the recent incidence of the Covid-19 pandemic. Evaluation, for obvious reasons of need and costs, is less frequent. Evaluation of public policies is rare in India, but not totally absent. Policies are reviewed from time to time by ministries, often supported by a consultative process with experts and at times even the public. The industrial policy has been reviewed four times since Independence, and the export–import policy annually. A recent example is

EvalGender+, New Delhi, India R. Nandi Institute of Social Studies Trust, New Delhi, India Community of Evaluators—South Asia, New Delhi, India International Evaluation Academy (IEAc), Prague, Czech Republic

5

EVALUATION IN INDIA

147

the adoption of the National Education Policy 2021 after a review and replacement of the earlier National Education Policy 1986/1992. Evaluation of programmes is very extensive in spite of the fact that there is no national evaluation policy or statutes demanding evaluation. It is not confined only to the public sector but extends to non-government organisations and the corporate sector pursuing welfare programmes under their corporate social responsibility (CSR). Industries evaluate their products through evaluative methods. An example is the recent evaluation of the Covid vaccines through quasi-experimental and non-experimental methods. Experimental designs are used in assessing the treatment effects in agriculture. The practice of evaluation, however, is particularly visible in the national development sphere in relation to social and economic programmes. The state of evaluation in the country will, therefore, be discussed in this chapter elaborately in relation to economic and social development policies and programmes. 1.3

Development Planning

Soon after its independence from colonial rule in 1947, India embarked on a mission of national economic and social development. Beginning with the First Five Year Plan (1951–1956), a number of ambitious development policies and programmes were launched with massive investments through the National Planning Commission. Even though the Planning Commission and the Five-Year Plans were abolished in 2014 and replaced by a policy think tank titled ‘National Institution for Transforming India’ (NITI Aayog), India’s journey along the path of planned development continues. Monitoring and evaluation have been an important component of the process of India’s planned development right from the beginning of the 1950s. Recognising the importance of M&E in development, the First Five Year Plan (1951–1956) states that “With increased investment on development, much more attention to the systematic assessment and evaluation of results from public expenditure is now called for than was probably necessary in the past. The problem arises in almost every project included in the Five-Year Plan in the Central Government as well as in the States. In each case, the machinery for the review of results has to be related to its nature and organisation. With every important programme, provision should always be made for the assessment of

148

R. AGRAWAL ET AL.

results. For instance, there should be an annual report and also intermediate periodical reports. In this connection, it may be suggested that reports should confine themselves to items of information which are intrinsically important and should not become too elaborate. Frequently, reports which are received do not receive the necessary study and analysis, and no attempt is made to draw practical conclusions from them. Reports which are called for by the Central Ministries from the States or by departments in the States from the districts should invariably be the basis of documents which are submitted to the higher authorities and are also made available to the project authorities from whose material they are compiled and later to the public. For important projects or aspects which need scrutiny, wherever possible, there should be arrangements for independent inspection followed by detailed reviews in consultation with the project authorities. Systematic evaluation should become a normal administrative practice in all branches of public activity. “With the objective of developing the techniques of evaluation, a beginning has now been made with the establishment of an independent evaluation organisation under the Planning Commission for community projects and other intensive area development programmes” (Planning Commission, Government of India, 1951, Chapter 6, Paragraph 29).1 Thus, M&E has been an integral part of India through its successive development plans. The 11th Five Year Plan (2007–2012), for instance, states that “quality evaluation of various programmes and projects would not only bring improvement in public sector performance but would also address a broad range of issues relating to economy, efficiency, sustainability and relevance of public sector funding and development intervention” (Planning Commission,Government of India, 2008, p. 230). The document of the abandoned 12th Plan (2012–2017) also recognised the importance of M&E. It states that “There is a need for strong web-based monitoring and evaluation system to promote transparency and accountability and facilitate regular tracking of physical and financial performance of individual programmes/schemes, particularly the ongoing clusters” (Planning Commission, Government of India, 2013, p. 398). Given the size of the country and the state of the economy at the time of its Independence in 1947, the developmental effort to raise the

1 See a recent article by Diwakar and Rao (2023) that traces the history of monitoring and evaluation in India and the world. Also, see Pal (2011).

5

EVALUATION IN INDIA

149

living standards of the large population had to be gigantic. Massive investments have been made since then in various socio-economic development programmes in infrastructure, agriculture, industry, and various services, particularly in health, education and employment in the social sector, programmes to bring the relatively backward sections of society into the mainstream development, scientific development, etc. Strict monitoring of progress and evaluation of results was therefore imperative to ensure that the scarce resources so deployed bear adequate fruit to justify the investments. The commitment of India to the Global Development Agenda 2030 and the monitoring of progress towards the 17 Sustainable Development Goals (SDGs) required the strengthening of the already elaborate national statistical system and monitoring systems. Besides, the complex nature of SDGs and the underlying principle of No One Left Behind adopted by the Agenda 2030 meant that it might not be possible to find evidence of achieving this principle purely through the national statistical system, and supplementary evidence will have to be generated from evaluations. However, the Voluntary National Reports of India presented to United Nations High Level Political Forums so far depended mainly on monitored information and national statistical system with little evidence of the use of evaluative information.

2

Institutional Structures and Processes 2.1

Constitution of India

The Constitution of India does not expressly provide for any evaluations of policies, programmes or projects of the government. In fact, the word ‘evaluation’ appears in the Indian Constitution just once, and not in connection with the evaluation of development interventions but in the context of providing lower evaluation standards for the socially weaker sections of the society in examinations or employment (Ministry of Law and Justice. Government of India, 2020). It prescribes, among other things certain Fundamental Rights and Duties for citizens which are enforceable in courts of law, and certain Directive Principles of State Policy, which while not so enforceable in courts of law, are to guide government’s policies and in making laws. In particular, Article 39 of the Constitution prescribes that

150

R. AGRAWAL ET AL.

The State shall, in particular, direct its policy towards securing — (a) that the citizens, men and women equally, have the right to an adequate means, of livelihood; (b) that the ownership and control of the material resources of the community are so distributed as best to subserve the common good; (c) that the operation of the economic system does not result in the concentration of wealth and means of production to the common detriment; (d) that there is equal pay for equal work for both men and women; (e) that the health and strength of workers, men and women, and the tender age of children are not abused and that citizens are not forced by economic necessity to enter avocations unsuited to their age or strength; (f) that children are given opportunities and facilities to develop in a healthy manner and in conditions of freedom and dignity and that childhood and youth are protected against exploitation and against moral and material abandonment. (Ministry of Law and Justice India, 2020, Part XVI, Article 335)

Article 41 of the Indian Constitution stipulates that the State shall, within the limits of its economic capacity and development, make effective provision for securing the right to work, to education and to public assistance in cases of unemployment, old age, sickness and disablement, and in other cases of undeserved want (Ministry of Law and Justice India, 2020, Chapter IV, Directive Principles of State Policy, Article 41). It can be observed from the above that the realisation of the provisions in the Directive Principles demands regular monitoring and evaluation of all welfare policies and programmes. 2.2

National Laws and Administrative Orders

There are no national laws enjoining Ministries/Departments to conduct regular evaluations or making a reference to M&E. However, in recent times, instructions issued by the Department of Expenditure, Ministry of Finance, to all Ministries/Departments (D.O.No. 66(01)/PF.II/2015 dated 18 May 2016) require, inter alia, that (a) measurable outcomes need to be defined for each scheme over the medium term, and (b) an evaluation framework would be designed for each scheme, any continuation of the scheme beyond the Fourteenth Finance Commission period 2019–2020 would be contingent on the result of such evaluation by the NITI. NITI, accordingly, commissioned evaluations of all schemes in ten development sectors in 2020.

5

EVALUATION IN INDIA

151

The Ministry of Rural Development and Panchayat Raj organises evaluations and impact assessment studies on a regular basis. Also, the ministry treats the evaluation of its policies, programmes, schemes, projects and interventions as an instrument of accountability. It has issued detailed guidelines for Evaluation, Impact Assessment and Research Studies to lay down in clear terms, the methods of organising/conducting these studies. These Guidelines are primarily for the evaluation commissioners and managers in processing evaluation proposals.2 A number of other Ministries, like the Ministry of Education or the Ministry of Health and Family Welfare, commission evaluations from time to time, but the presence of laws, executive orders or guidance documents on the conduct of evaluations is not in evidence. 2.3

Comptroller and Auditor General (C&AG) of India (Supreme Audit Authority of India)

Article 148 of the Constitution provides for the constitutional position of Comptroller and Auditor General of India (C&AG), who submits reports on government programmes and expenditures directly to the President of India. C&AG is the Supreme Audit Institution of India (SAI). Mandated by the Constitution of India, the SAI promotes accountability, transparency and good governance through high-quality auditing and accounting and provides independent assurance to the Legislature, the Executive, and the Public, so that public funds are being used efficiently and for the intended purposes. The office conducts financial, performance, and compliance audits of offices of Central and State Governments and also public sector undertakings, of which performance audits address the issues of economy, efficiency, and effectiveness of an organisation, programme or scheme. As stated in the C&AG’s Performance Audit Guidelines (C&AG, 2014), “Performance audit is an independent assessment or examination of the extent to which an organisation, programme or scheme operates economically, efficiently and effectively” (C&AG, 2014, p. 2). These Guidelines further elaborate that performance audits seek answers to the basic questions of (a) Are the things being done in the right way, and (b) Are the right things being done, thus conceptually approximating the main aspects of evaluation (C&AG, 2014, 2 See rural.nic.in/sites/default/files/guidelines_evaluation_impact_assessment_research_ studies_economic_monitoring_wing.pdf. Accessed on 29 September 2022.

152

R. AGRAWAL ET AL.

p. 5). The Guidelines further list “External sources - independent surveys, evaluation, research, etc.” among the sources of evidence for preparing performance audit reports (C&AG, 2014, p. 55). Thus, C&AG’s performance audits reflect evaluative aspects and also use independent evidence from evaluations in such audits. C&AG’s reports are first presented to the President who sends them to Parliament and are normally discussed in detail in the Public Accounts Committee (PAC, comprising members of Parliament) when the ministry concerned responds to the issues raised. The PAC then submits its report to the Speaker of the House and that report is then discussed in Parliament. Not all C&AG reports may get equal attention from the Parliament, but some, which are politically significant, are extensively discussed and debated upon (e.g. allocation of coal blocks to the private sector, or purchase of defence aircraft). C&AG prepares Compliance Reports on the extent to which its findings were attended to by the government. Some recent reports of C&AG are: • Report No. 13 of 2020: Performance Audit of National Pension System3 • Report No. 14 of 2019: Performance Audit of Pradhan Mantri Ujjwala Yojana4 • Report No. 8 of 2019: Preparedness for the Implementation of SDGs5 • Report No. 15 of 2006: Performance Audit of the Implementation of Sarva Shiksha Abhiyan (SSA).6 The Performance Audit of the National Pension Scheme (NPS) was conducted to assess whether: (i) the system for NPS was established as envisaged; (ii) all eligible subscribers had been covered under NPS; (iii)

3 Read the complete report at https://cag.gov.in/en/audit-report/details/112173. Accessed on 13 March 2023. 4 Read the complete report at https://cag.gov.in/en/audit-report/details/55961. Accessed on 13 March 2023. 5 Read the complete report at https://cag.gov.in/uploads/download_audit_report/ 2019/Report_No_8_of_2019_Preparedness_for_the_Implementation_of_Sustainable_Dev elopment_Goals_Union_Government_Civil_NITI_Aayog.pdf. Accessed on 20 February 2023. 6 Read the complete report at https://cag.gov.in/en/old-audit-reports/view/13766. Accessed on 20 February 2023.

5

EVALUATION IN INDIA

153

subscribers were registered within the prescribed timelines, if any; (iv) SCFs were uploaded timely, total contributions (subscribers’ and employers’) were remitted to Trustee Bank, within prescribed timelines, if any; and (v) NPS is being regulated and monitored in accordance with relevant Acts/regulations/orders etc. The Audit, which was primarily focused on the planning of NPS, its implementation and monitoring, was based on a sample of Ministries/Departments/Autonomous Bodies, in the Central Government as well as in the States. The report pointed to a number of shortcomings in the planning, implementation and monitoring of the scheme. Similarly, the Performance audit of Preparedness for the Implementation of SDGs was undertaken with the objective of covering aspects such as the extent to which the 2030 Agenda has been adapted in the national context; identification and mobilisation of resources and capacities, and creation of mechanism for monitoring and reporting progress. In each of these areas, the report pointed to achievements and areas of concern. 2.4

National Evaluation Policy

In recent years, a few countries in the world have formulated national evaluation policies to guide their evaluative work.7 Even though evaluation has been practised for seven decades nationally and in various sectors and states in the country, the Government of India has not adopted any specific national evaluation policy till date. One of the constituent States of India, Karnataka, however, adopted an evaluation policy in the year 20008 and a specialised institution called Karnataka Evaluation Authority that oversees the evaluative work of the State government’s programmes was established. According to the Policy “all schemes (PLAN) with more than Rs. 1 crore9 outlay were required to be evaluated at least once in 5 years by all the government departments, corporations, boards, local

7 1. Status of National Evaluation Policies: Global Mapping Report by Barbara Rosenstein, 2015 for Parliamentarians Forum for Development Evaluation in South Asia and EvalPartners; last update 2021 by Barbara Rosenstein and Asela Kalugampitiya (Rosenstein 2015; Rosenstein & Kalugampitiya, 2021). 2. A Study on the Status of National Evaluation Policies and Systems in Asia Pacific Region 2021, APEA; last update 2022 (Diwakar et al., 2021, 2022). 8 Karnataka is the only State in India to have an explicit evaluation policy. 9 1 crore is 10 million.

154

R. AGRAWAL ET AL.

bodies and other publicly funded entities and 1% of the total outlay of a project/scheme, with an upper limit of Rs. 5 lakhs,10 is to be kept for evaluation purpose. It is mandatory for a scheme/project to continue beyond plan period to be justified by evaluation”.11 The Policy stipulates that all evaluations, irrespective of their nature, are entrusted to independent external evaluators. There is no evidence or information as to whether any of the other State Governments intend to follow suit and adopt such policies. Nevertheless, they do commission need-based evaluations from time to time, mostly to external evaluators, and have their own administrative systems to manage such evaluations. The Evaluation Community of India (ECOI), an association of professional evaluators, has in the recent past been pushing for a national evaluation policy for the country. In the EvalFest 2020, a biennial conference of evaluators organised by ECOI in February 2020, particularly in a session of parliamentarians, it was strongly urged that such a policy was imperative. Since then, ECOI has been working on a draft of such a policy. In 2021, a draft has been presented to DMEO. It broadly presents a framework for evaluation, calls for an annual evaluation plan, and presses for methodological standards and professionalisation, evaluation capacitybuilding, institutional mechanisms for follow-up of results, the role of professional associations of evaluators and other aspects. As the document is still to be considered by the government, it is not possible to say at present if and when it, with or without modifications, will be adopted. 2.5

Parliament and Evaluation

Parliament members, while not directly participating in any evaluations, evince keen interest in evaluative evidence. They frame their questions to various Ministries regarding policies, programmes, and projects of the government, effectiveness of governance, efficacy of development policies and programmes, and optimality of expenditure for the government to provide answers for. They press for accountability on the part of government and directly and indirectly promote the demand for evaluations. Parliament questions are taken very seriously by the government and all

10 1 lakh is 100,000. 11 Official website of Karnataka Evaluation Authority can be accessed at https://kmea.

karnataka.gov.in/info-1/About+Us/en. Accessed on 11 July 2022.

5

EVALUATION IN INDIA

155

efforts are made to provide answers. A few questions specifically relating to evaluation posed by Members of Rajya Sabha (Upper House) and Lok Sabha (Lower House) in recent years are given in the box below, which deal with development programmes as well as general administration and governance. This clearly shows that members of the parliament do ask for evaluations of specific schemes: Some Questions of Members of Rajya Sabha and Lok Sabha Relating to Evaluation in Recent Years RAJYA SABHA UNSTARRED QUESTION No-577 ANSWERED ON 07.02.2019 BY MINISTRY OF WOMEN AND CHILD DEVELOPMENT Monitoring and Evaluation of Mahila Shakti Kendra a. the impact of the Pradhan Mantri Mahila Shakti Kendra since its inception; b. whether any monitoring and evaluation of these kendras has been undertaken; c. if so, whether there are any course corrections as a result; and d. If no; why was such an evaluation mechanism not set up? RAJYA SABHA UNSTARRED QUESTION No-2090 ANSWERED ON 06.12.2019 BY MINISTRY OF AGRICULTURE AND FARMERS’ WELFARE Performance of Agricultural Schemes a. whether despite various ongoing schemes for promotion of agriculture in the country, more than 40% of farmers are living below the poverty line; b. if so, whether Government has made any study to assess the performance of each of these schemes during the last three years; c. if so, the outcome of such study; and d. whether Government proposes to reform such schemes and if so, the details thereof? LOK SABHA UNSTARRED QUESTION No-5024 ANSWERED ON 24.07.2019 BY MINISTRY OF PERSONNEL, PUBLIC GRIEVANCES AND PENSIONS Evaluation of Ministries and Department

156

R. AGRAWAL ET AL.

a. Whether the Government has set up any Committee to monitor and evaluate the functioning of each Ministry/Department; b. if so, the details thereof along with the aims and objectives of setting up of the Committee; c. whether the Results-Framework Documents of these Ministries/ Departments have been received; and d. if so, the percentage of targets met by these Ministries/Departments? LOK SABHA UNSTARRED QUESTION No-4952 ANSWERED ON 25.03.2021 BY MINISTRY OF HOUSING AND URBAN AFFAIRS Evaluation of PMAY a. Whether the Government has announced Pradhan Mantri Awas Yojana (PMAY) under scheme “Housing For All’ in 2015; b. if so, the details including the applications received and beneficiaries thereof since 2017, State-wise; c. Whether the Government has failed to support at par their target of PMAY (Urban); d. if so, the projected/estimated targets and achievements therein, State-wise; and e. the details of the Government’s plan to ensure hundred percent housing for all by 2025? Sources https://loksabhaph.nic.in/Questions/Qtextsearch.aspx and https://rajyasabha.nic.in/rsnew/rs_rule/question.asp. Accessed on 29 September 2021.

Parliamentary committees comprising members of parliament are established by the parliament to study and deal with various matters that cannot be directly handled by the legislature due to their volume. They also monitor the functioning of the executive. There are departmental parliamentary standing committees looking into the functioning and issues relating to various ministries/departments. Some of the main committees, among others, are: General PAC, Estimates Committee, and Departmental Committees for Agriculture, Information and Technology, Rural Development, Social Justice and Empowerment. Members of Parliament and State legislatures also form their own groups to examine evidence on the nation’s development. For instance,

5

EVALUATION IN INDIA

157

there is a Parliamentary Forum on SDGs. It is supported by a civil society organisation titled Centre for Legislative Research. 2.6

Evaluation Structures in Government

While there is no national evaluation policy in India, the practice of evaluation is as old as planned development itself. As mentioned in the introduction above, the importance of evaluation was emphasised at the very beginning of the development process in the early 50s. Accordingly an institution named Programme Evaluation Organisation (PEO) was created in 1952 under the national Planning Commission with regional offices and field units to carry out periodic evaluations initially of community development programmes and later extended to all government development programmes. Since then, the PEO conducted a number of evaluations in the following two decades. The PEO flourished from 1952 to 1970 but gradually became weakened in subsequent years, with a reduction in its field offices and staff (Mehrotra, 2013). The practice of assigning evaluations by individual Ministries to independent research and civil society organisations became common. Some details of its evaluations are available in their publication of 2004 (PEO, 2004). Later, an Independent Evaluation Office (IEO) was created in 2013 with a mandate to evaluate plan programmes, especially the large flagship programmes to assess their effectiveness, relevance, and impact. Unlike the PEO, the IEO remained outside the Planning Commission. Subsequently, on the abolition of the National Planning Commission and its replacement in 2014 by the NITI Aayog, the PEO and the IEO were merged into a single entity called the Development Monitoring and Evaluation Office (DMEO), an attached office of NITI Aayog under a Director General. Under the present dispensation, therefore, the DMEO in NITI Aayog at national level is mandated with the mission “To institutionalize application and use of monitoring and evaluation at all levels of government policy and programs and help improve the efficiency, effectiveness, equity, sustainability and achievement of results ”12 Its work includes independent evaluation of Centrally Sponsored development schemes, and other schemes suo-motu or on the request of implementing Ministries/Departments of the Central Government.

158

R. AGRAWAL ET AL.

The objectives of DMEO are shown in Fig. 1. To pursue these objectives, the DMEO performs the functions shown in Fig. 2.13 As a part of NITI Aayog, DMEO has advisory powers across the Ministries and Departments of the Union Government and is one of few institutions within the government to provide a cross- and interministerial perspective. Its mandate also expands to technical advisory to States, under NITI Aayog’s mandate of cooperative and competitive federalism. DMEO also has the ability to share reports and reviews directly with key decision-makers in the government to initiate action.

Fig. 1 Objectives of DMEO, NITI Aayog, Government of India (Source Designed by the authors based on the info provided on the DMEO website https://dmeo.gov.in/)

12 Vision, Mission and Strategy | DMEO | Development Monitoring and Evaluation Office | Attached office under NITI Aayog. See https://dmeo.gov.in/. Accessed on 14 March 2023. 13 The diagram is taken from the Development Monitoring & Evaluation Office (DMEO) | NITI Aayog, Functions of DMEO, at https://www.niti.gov.in/developmentmonitoring-evaluation-office-dmeo. Accessed on 29 September 2021.

5

EVALUATION IN INDIA

159

Fig. 2 Functions of the DMEO (Source Designed by the authors based on the info provided on the DMEO website https://dmeo.gov.in/)

This institutional positioning also gives the organisation convening power, to create a platform for M&E advancement in the country. While, thus, the DMEO holds a central place in matters relating to evaluation in the country, the evaluation function in the public sphere is spread across different Ministries/Departments in the Central Government as well as departments in State Governments. The Department of Programme Implementation in the Ministry of Statistics and Programme Implementation is responsible for M&E of a special programme called the 20-Point Programme. Other major Ministries have their own M&E institutions and practices. For instance, the Ministries of Rural Development, Education, Social Welfare, and Health have their own elaborate M&E systems. They commission evaluations of their development schemes and assign the task to research and civil society organisations selected through an objective process of selection. The Ministry of Rural Development, which implements a wide range of rural development schemes, has a M&E Division which issued detailed M&E Guidelines.14 It has a MIS, a system of National Monitors and also commissions and funds detailed

14 https://rural.nic.in/sites/default/files/Guidelines_Evaluation_Impact_Assessment_R esearch_Studies_Economic_Monitoring_Wing.pdf. Accessed on 13 March 2023.

160

R. AGRAWAL ET AL.

evaluation studies. There are detailed guidelines on the National Monitors scheme.15 The National Institute of Rural Development and Panchayat Raj supports the Ministry through various evaluations. The Ministry of Education, too, has an MIS maintaining data uploaded at the district level, and verified by technology-supported systems. Its flagship scheme Sarva Shiksha Abhiyan (Universal Secondary Education) is monitored and also subjected to independent evaluations from time to time.16 In the Ministry of Women and Child Development, again, there are institutional arrangements for MIS, monitoring and supervision, and evaluation of the Ministry’s major programme of Integrated Child Development Programme at various levels of administrative divisions. Similar mechanisms exist in State Governments also. The institutional arrangements for M&E in State Governments came into existence almost at the same time as the PEO at the national level. During the 1960s, many State governments established M&E institutions. Some of them took the form of independent full-fledged M&E Directorates, in some States they were separate Divisions in the Directorate of Economics and Statistics, and in others M&E functions, if performed, were assigned to the normal statistical personnel. In some States, the institutions were strong and effectively performed the evaluation functions, in others it was less so. In the State of Karnataka, an Evaluation Authority was established and an evaluation policy was laid down.17 In Rajasthan, there is an independent M&E Directorate which functioned efficiently.18 In Tamil Nadu too, the Department of Evaluation and Applied Research (DEAR)19 originally functioned as Economic Analysis and Research Division in the 15 Ministry of Rural Development (2016). Guidelines for Scheme of National Level Monitors. Government of India. Ministry of Rural Development. Department of Rural Development. Monitoring Division. New Delhi. https://rural.nic.in/sites/default/files/ NLM_Guidelines.pdf. Accessed on 17 March 2023. 16 Government of India NITI Aayog, List of Evaluation Studies Conducted by Programme Evaluation Organisation https://niti.gov.in/planningcommission.gov. in/docs/reports/peoreport/cmpdmpeo/index.php?repts=peo_yrwis.htm. Accessed on 31 May 2021. 17 The website of Karnataka Evaluation Authority can be accessed at https://kmea.Kar nataka.gov.in. Accessed on 20 February 2023. 18 The website of the Rajasthan Directorate of Evaluation Organisation can be accessed at https://plan.rajasthan.gov.in/evaluation. Accessed on 20 February 2023. 19 The website of the Department of Evaluation and Applied Research of Tamilnadu can be accessed at https://direar.tn.gov.in/. Accessed on 14 March 2023.

5

EVALUATION IN INDIA

161

Finance Department and is now functioning as a separate Department since 1974 under the overall administrative control of Planning, Development, and Special Initiatives Department, charged with the responsibility to take up various evaluation studies assigned by the Government from time to time. Thus, though the positioning and structure of the institutions are different in different States, institutional mechanisms have been functioning in the Central and various State Governments. 2.7

Evaluation Practice

Despite the absence of any constitutional mandate or national evaluation policy or legislative requirement for evaluation of development instruments (policies, programmes, and projects), the institutional infrastructure described above has promoted the practice of evaluations extensively—temporally and geographically, a process that permeated different sectors of economy. The Programme Evaluation Organization (PEO), for instance, conducted 236 evaluation studies between 1954 and 2016,20 with an average of 4 evaluations per year. The sectoral distribution of evaluations (185) done by the PEO during 1954–2010 shows the following distribution by sectors (Table 1). Some more recent evaluations done by PEO were: Table 1 Sectoral distribution of PEO evaluations (1954–2010)

Agriculture, fisheries, and allied Rural/community/area development Social and tribal welfare Labour and employment Industry/small industry Health and family welfare Cooperation Education Women and child development All other sectors Total

39 43 19 13 11 9 8 7 5 31 185

20 Government of India, NITI Aayog, List of Evaluation Studies Conducted by the Programme Evaluation Organisation (PEO). See https://niti.gov.in/planningcommission. gov.in/docs/reports/peoreport/cmpdmpeo/index.php?repts=peo_yrwis.htm. Accessed on 31 May 2021.

162

R. AGRAWAL ET AL.

i. Evaluation Study on Efficiency of Minimum Support Price for Farmers (MSP) (2016) ii. Evaluation Study of Backward Regions Grant Fund (BRGF) (2014) iii. Evaluation Study on Integrated Scheme of Micro-irrigation (2014) iv. Evaluation Study on Kasturba Gandhi Balika Vidyalaya (KGBV) (2015) v. Evaluation Report on Sarva Shiksha Abhiyan (2010) vi. Evaluation of National Health Mission in 7 States (2011). While it is quite natural that the PEO, originally set up to evaluate community development programmes, evaluated the largest number of evaluations for that sector, its reach extended to all other sectors, particularly agriculture, various components of the social sector (education, health, social welfare and employment, women and child development), and to the environment. The DMEO, which subsumed the PEO within its organisation, has commissioned detailed evaluation of 28 umbrella Centrally Sponsored Schemes (programmes/schemes nationally conceived and designed, implemented by the State governments, and expenditure shared between the two in prescribed ratio) in the following ten sectors in 2020: agriculture, women and child development, human resource development, urban transformation, rural development, drinking water and sanitation, health, jobs and skills, water resources, environment and forest and social inclusion, law and order and justice delivery (Sharma, 2019). These evaluations are currently going on. The DMEO is also striving to develop outcome indicators. A system of outcome-budgeting is now in place across Ministries. As stated earlier, it is not only the PEO and the DMEO that conducted evaluations of public schemes, but individual Ministries like that of Rural Development, Education, Social Welfare, Women and Child Development, etc. did too. Some of these Ministries make enormous amounts of investments in their programmes, and have established elaborate monitoring systems. The monitoring systems of the Ministries of Rural Development and Education extend down to the district level, measuring progress at that level. They also employ technology in monitoring overall impacts, as, for instance, geo-spatial mapping to assess rural development. Other departments of the government also commission evaluations of their schemes from time to time. The Indian Council of Agricultural Research (ICAR) (a wing of the Ministry of Agriculture), for instance,

5

EVALUATION IN INDIA

163

had commissioned the National Institute of Labour and Employment Research and Development in 2015 to carry out an evaluation of the functioning of its Krishi Vigyan Kendras (extension centres in the districts for disseminating farm technologies and knowledge). The Ministry of Micro, Small, and Medium Industries carried out three evaluations of the Prime Minister’s Rozgar Yojana (a scheme for providing self-employment support via finances and technical training), the last being earlier this century. State governments also have been subjecting their schemes to evaluation periodically. For instance, the Directorate of Evaluation, Government of Rajasthan, has conducted over 541 evaluations of varied programmes/ schemes of the State government, in about 60 years since the organisation came into existence in 1960, about 9 per year.21 The Karnataka Evaluation Authority commissioned 132 evaluations in 9 years (2011– 2012 to 2019–2020) of its existence,22 on programmes as diverse as the Evaluation of the National Bio-gas and Manure Management programme, Impact of implementation of the Western Ghats Development Programme in Karnataka and Evaluation of the Rapid Appraisal of Result Framework Documents. Thus, in spite of the absence of any national evaluation policy, national evaluation system, national evaluation plan or national guidelines on standards system, evaluation is widely practised in the whole country and extends to various policy fields and sectors of the economy. The other main features of the evolution of evaluation practice in India over the past six decades are: a. India’s dependence on external aid has decreased over the years and today, most programmes are being financed from national or State budgetary resources. As a consequence, evaluations have almost ceased to be external donor-driven and have become country-led. Thus, evaluations may no longer be a response to donor requirement, but tend to be driven by an urge to learn from evidence and enhance programme performance. It is, however, possible that there may be cases where evaluations are still carried out to 21 The reports can be accessed here at https://plan.rajasthan.gov.in/evaluation. Accessed on 8 June 2020. 22 The report can be accessed here at https://kmea.karnataka.gov.in. Accessed on 8 June 2020.

164

R. AGRAWAL ET AL.

fulfil an internal requirement from resource controlling authorities for continuing or upscaling the programme. As mentioned by Mehrotra, “Today, most development schemes of the government of India, if not of the state governments, are evaluated from time to time (in some form or another), but only rarely because of donor insistence. Evaluation is being driven mainly because of the value planners, policy makers, and program implementers are placing on it as an instrument for accountability and improvement of public programs. In India, public expenditure is much more under the scanner, and the slow progress toward the achievement of Millennium Development Goals has also led to a greater stress on effectiveness of public expenditure” (2013, p. 1). The adoption of SDGs and the national framework of indicators has injected a sense of urgency in M&E activity in the country. b. Evaluations that used to be done mostly by internal agencies like PEO and State Departments of Evaluation, are now mostly done by external independent research organisations or civil society organisations. This has added to the credibility of evidence collected and reported on. c. With the entry of independent organisations, there has also been an effect on the methodological aspects of evaluation. In the past, most evaluations employed exclusively quantitative methods using RCTs, quasi-experimental and often non-experimental designs. It was perhaps considered that qualitative methods do not allow generalising results. Today, that prejudice against qualitative approaches and independent organisations commissioned to do evaluations generally employ mixed methods including surveys, Focus Group Discussions, and individual expert interviews. As a corollary, the practice of engaging persons mainly from statistical and economic disciplines to conduct evaluations has yielded place to a multi-disciplinary approach. d. A number of programmes in the public sector with large investments are in the social sector that includes health, education, employment, social welfare, women and child development, etc. This factor and India’s commitment to international goals, MDGs earlier and SDGs now, and international conventions like CEDAW, which demand monitoring of progress towards these instruments, make it necessary that evaluative evidence that focuses on gender, equity and related issues becomes available. Most evaluations now take these aspects into account.

5

2.8

EVALUATION IN INDIA

165

Evaluation Use

It is one thing to conduct evaluations and quite another to utilise the results. Internationally also, the level of evaluation use is relatively low. Citing Patton (2008), a survey of members of the American Evaluation Association conducted in 2006, two-thirds (68%) of the members selfreported that their evaluation results were not used (CDC, 2013). It is not easy to assess the extent of utilisation of evaluation findings as it bristles with definitional problems. Whether utilisation refers to the adoption of entire evaluation reports and acting on their recommendations in toto or to the acceptance of the recommendations on a selective basis and their implementation is a moot point (Agrawal & Rao, 2011). Moreover, as observed by Pal (2012), it is difficult to say which aspects of the findings were acted on and whether the substantive findings about the systemic weaknesses were taken into consideration, as restructuring of programmes may not take place at one point of time and all follow-up actions are not formally announced. The usual tendency among commissioners of evaluation, however, is to accept recommendations supporting their scheme of things and ignore negative references to the programme implementation and performance. While India does not have any national mechanism to track the implementation of recommendations of evaluations, nor conducted any survey to assess the situation, it is not likely that it has a better implementation rate than one-third. There are, however, some positive indications. In response to queries from members of the Evaluation Advisory Committee of the Planning Commission, PEO began gathering evidence of the use of its reports from various sources. According to the report of the PEO of 2004 cited above, the following are some examples of follow-up actions taken • The Mahila Samriddhi Yojana, a programme for the empowerment of poor rural women, was thoroughly restructured and redesigned following the PEO’s evaluation report of 1996.23

23 Planning Commission (2004). Annual Report 2003/04. Planning Commission. Government of India. New Delhi, pp. 141f. Also please see the webpage on the Mahila Samriddhi Yojana at https://agriinfo.in/ mahila-samridhi-yojana-m-s-y-1466/. Accessed on 16 March 2023.

166

R. AGRAWAL ET AL.

• The Non-Formal Education programme of the Ministry of Human Resource Development was revamped, with a new strategy. In restructuring the programme, the PEO’s findings of the evaluation of the programme of 1998 were considered.24 • Based on the findings of the PEO’s evaluation 2001 of the National Project on Bio-gas Development, the Ministry of Non-Conventional Energy Sources modified the guidelines for implementation of the scheme and the revised guidelines were issued in 2002–2003.25 • The findings of the evaluation on the functioning of community/ primary health centres were extensively used in the midterm appraisal of the ninth plan (1997–2002), formulation of the tenth five-year Plan (2002–2007), and formulation of the National Health Policy in 2002.26 The Prime Minister’s Rozgar Yojana, a scheme to provide financial and technical training to educated unemployment youth, started in 1993– 1994, was evaluated three times in 1996, 2001 and 2005. It was found that only 10% of the recommendations were implemented after the first evaluation and about 30% in case of the second evaluation. Detailed discussions between the sponsors and evaluators initially and at the end helped improve the acceptance (Agrawal & Rao, 2011, pp. 123–139). A more recent example of evaluation use has been cited by Mishra and Avinandan (2020), it is the evaluation of the Bhagyalakshmi CCT scheme in Karnataka, under which a girl child belonging to a below poverty line (BPL) family is eligible for financial assistance after completing 18 years of age. While evaluation results found that the scheme reduced gender disparity in school attendance rates, it also noted that the scheme was somewhat marriage-oriented, i.e. parents expressed that they would like 24 Ministry of Human Resource Development (2001). Evaluation Study on the Impact of Nonformal Education. Government of India. Also see Planning Commission (2004). Annual Report 2003/04. Planning Commission. Government of India. New Delhi, pp. 141f. 25 PEO (2002). Evaluation Study on National Project on Biogas Development. Programme Evaluation Organisation. Planning Commission. Government of India. New Delhi. Online at: https://dmeo.gov.in/sites/default/files/2019-10/Evaluation%20study% 20on%20National%20Project%20on%20Biogas%20Developmen.pdf. Accessed on 16 March 2023. 26 See the National Health Policy 2002 at https://nhm.gov.in/images/pdf/guidelines/ nrhm-guidelines/national_nealth_policy_2002.pdf. Accessed on 17 March 2023.

5

EVALUATION IN INDIA

167

to use the money from the scheme for their daughters’ marriages. Based on the findings, the age limit to withdraw the full amount under this scheme was increased from 18 to 21 years. Issues of quality of evaluations conducted, emanate both from the demand and supply sides. Factors like resources allocated, money, time and other facilities, method of selecting evaluators, and allowing independence to evaluators, all influence the quality on the demand side. The competencies of the evaluators and adherence to quality and ethical standards greatly affect the quality from the supply side. The practice of allocating the evaluation to the evaluator with the lowest financial bid prevalent in India puts constraints on the quality. A meta-analysis of 110 evaluations (that were conducted before 2010) by the Institute of Human Development as a part of a project (State of India Development Evaluation) supported by International Development Research Council (IHD 2013), using various internationally adopted quality checklists adapted to the Indian context, indicates (a) that the quality of methodology was satisfactory in two-thirds of the evaluations, (b) about three-fourths of the reports were satisfactory in quality of conclusions, lessons learnt and recommendations, (c) 50–60% of the evaluations were rated satisfactory in assessing efficiency, sustainability, and impact, while the percentage was above, while the percentage of the reports that crossed the benchmark was 73 in the case of relevance and 66 in the case of effectiveness, and (d) only a quarter of the reports were satisfactory in their assessment of human rights and gender mainstreaming, though a large proportion of the evaluations were rated neither satisfactory nor unsatisfactory on these parameters. It seems reasonable to assume that the quality of evaluations since 2010 has improved with greater stress on quality evaluations in recent years and improvement in awareness among evaluation commissioners, and in evaluator capabilities with frequent exposure to national and international capacity-building efforts. Terms of Reference (TOR) have become more comprehensive with the commissioners and managers being more familiar with the importance and the nuts and bolts of evaluation. Evaluators too are now familiar with issues like the integration of gender and equity into evaluations, the available wide methodological spectrum, and access to technology in the collection and analysis of data. However, there are as yet no institutional mechanisms to assess the quality of evaluations (such

168

R. AGRAWAL ET AL.

as those by the United Nations, e.g. Global Evaluation Report Oversight System of UNICEF) or track the utilisation of evaluation results that could throw light on the impact of quality on utilisation.

3 Societal Dissemination/ Acceptance (Social System) 3.1

Use of Evaluations by Civil Society

The contemporary development discourse in India acknowledges the prominent role of partnerships with civil society organisations in ensuring inclusive development (Gupta, 2016). In India, Civil society comprises formally constituted non-governmental organisations (NGOs) along with a wider range of informal organisations, networks, and citizens’ groups. The recent addition to the CSOs is the CSRFoundations in India which could be otherwise called a part of the domestic philanthropy organisations (Patra & Behar, 2015).27 CSOs in India play many and varied roles in development cooperation, including conducting and influencing and monitoring development policies and practices. India is currently home to 3.3 million registered NGOs and about one-third are estimated to be fully functional and ready to support the implementation of the law (B the Change, 2016). As per the data provided by the government portal called NGO Darpan that manages the registered NGOs as of date one hundred fourteen thousand NGOs are signed up to the portal.28 It is more or less a common practice among the Indian NGOs to have evaluations carried out by themselves and use them, or by external agencies for their own purposes or as per the requirement of the donors—both

27 For implications of corporate social responsibility on civil society in India, see https:/ /ngosindia.com/wp-content/uploads/2018/08/CSRIndia.pdf. Accessed on 20 February 2023. 28 The NGO DARPAN Portal that is maintained under the aegis of NITI Aayog, invites all Voluntary Organisations (VOs)/Non-Governmental Organisations (NGOs) to sign up on the Portal. The NGO DARPAN is a platform that provides space for interface between NGOs and key Government Ministries, Departments, and other Government Bodies. This portal enables VOs/NGOs to enrol centrally and thus facilitates creation of a repository of information about VOs/NGOs, Sector/State wise. The Portal facilitates VOs/NGOs to obtain a system generated Unique ID. The Unique ID is mandatory to apply for grants under various schemes of Ministries/Departments/Governments Bodies.

5

EVALUATION IN INDIA

169

government and private. However, Indian civil society organisations face huge challenges in carrying out the evaluations. A recent online survey with 200 NGOs reveals that only 58% of the CSOs in India have their own internal M&E department.29 The rest have little or no internal capacity and completely rely on external consultants. A follow-up telephonic survey with the 10 selected Monitoring Evaluation and Learning (MEL) officers of the NGOs further reveal that M&E is more of a tokenistic activity by the majority of the Indian organisations as this is a requirement by the donors.30 Some of the challenges that the Indian CSOs face are (1) lack of technical capacity; (2) lack of technical and financial resources; (3) lack of awareness regarding building up of evidence and use of evidence for programme designing and policy-making. However, there is a number of CSOs who follow organisational systems and guidelines, that are already in place. A key informant from an international NGO that collaborates with more than 100 Indian NGOs as implementation partners opined that there is a huge gap among the Indian organisations as far as the awareness, M&E capacity, review systems, and use of evidences are concerned. The ECOI, the national VOPE has been working with many CSOs and individuals since 2015 for developing the awareness and evaluation culture towards evidence-based policy-making and programme designing. A study conducted in 2016 investigated users’ experiences of Monitoring and Evaluation (M&E) systems in the context of HIV interventions in western India. Based on interactions with the employees of different NGOs and the donor organisations, the study concluded that M&E appears an ‘empty ritual’, enacted because donors require it, but not put to local use. This paper reveals a remarkable congruence between the views of NGO and donor users of M&E systems in the HIV/ AIDS sector in India. In terms of the purported aims of M&E, this study suggests that monitoring may be playing a positive role in developing financial accountability; that monitoring is being used to assess 29 Based on an unpublished Survey Report in 2021. The survey was conducted by Satyam Vyas, Neeta Goel and Rajib Nandi with an assistance of the research team at Arthan—A social enterprise based in New Delhi. 30 Based on interviews conducted in 2021 with 20 senior MEL officers of 20 different NGOs and evaluation consulting firms in India. The survey was conducted by Satyam Vyas, Neeta Goel and Rajib Nandi with an assistance of the research team at Arthan—A social enterprise based in New Delhi. Also see the paper by Jacob and Mukhopadhyay (2023).

170

R. AGRAWAL ET AL.

programme effectiveness, though participants question the validity of the effectiveness data; and that monitoring is not being used as a means of feedback and programme improvement. The authors argue that the activity of monitoring is being constituted as an instrument of performance management and disciplining of NGOs, and not as an instrument of rational programme improvement (Shukla et al., 2016). Another recent study conducted by Singh et al. (2018) examines the continuum of sustainability and CSR policies and analyses the broad patterns that have emerged with respect to monitoring and evaluation practices in the CSR programmes of Indian companies. The paper scrutinises the annual sustainability reports and the monitoring and evaluation practices in CSR interventions. The study uses both primary and secondary data sources, and employs text network analysis and narrativesbased content analysis to analyse the data. The authors found that the conceptualisation of sustainability is a largely rhetoric and customary exercise that does not take into account variations in firms’ businesses. This approach towards sustainability initiatives presents serious challenges to sustainability, including social sustainability. The study also finds that there is lack of ‘willingness’ and ‘readiness’ among Indian companies to measure and monitor the outcomes of CSR interventions. Consequently, they do not find value in monitoring and measuring CSR outcomes. This also explains the lack of capacity among CSR professionals to design programspecific M&E frameworks and evaluation strategies that can assist them to improve the social returns from CSR spending. However, there are many promising stories too. Pratham, one of India’s largest NGOs, works to provide quality education for millions of underprivileged children in rural and urban areas.31 CLEAR-SA, with Abdul Latif Jameel Poverty Action Lab (J-PAL), has developed a partnership with Pratham based on a shared belief in evidence-based policymaking. Over the last decade, J-PAL has conducted several randomised evaluations of Pratham’s pilot programmes. Results from these evaluations have helped the NGO improve the design, delivery and impact of its programmes. As an organisation that is committed to improving the quality of education for children from socially and economically disadvantaged communities, Pratham was interested in integrating impact evaluation into its M&E systems and requested support to help build the 31 Annual Report 2019–20, https://www.pratham.org/ar/ar1920.pdf. Accessed on 30 September 2021.

5

EVALUATION IN INDIA

171

capacity of its management, programme, and M&E teams to design, run, and interpret the results of randomised evaluations.32 3.2

Public Perception and Public Discourse

The first decade of this century which is often termed as the regime of economic growth and high public expenditure in India has propelled the need for effective monitoring, evaluation and accounting systems in the country. Since 2010, a number of donor organisations and independent research organisations in India started highlighting the need for independent evaluation studies and strengthening the research organisations for conducting development evaluations. Consequently, a study entitled, “India: State of Development Evaluation Report 2010” was conducted in 2010 by a private not-for-profit research organisation. The study provided an evidence-based analysis of the state of development evaluation processes and systems in India to inform various stakeholders involved in development evaluation. The study highlighted the issue of evaluation capacity among the independent organisations that conduct evaluations in India. Another key problem that the study highlighted was the budget and financial allocation for M&E activities. Interestingly, a key point raised by participant stakeholders of the study was the negative consequences if evaluation becomes completely a government-oriented process. The study recommended that the feedback mechanism needs to be strengthened to incorporate the results of the evaluation study in the planning and implementing process and care should be taken to avoid inordinate delays in the completion of studies, often cited as one of the stumbling blocks on the path to effective evaluation. In principle, evaluation is a critical issue for civil society organisations. At times, there are queries by policy-makers, funders, activists and scholars, who want to know whether CSOs are having an impact on the issues they seek to address. Evaluative demands are only likely to increase as CSOs continue to play a central role in development intervention in India. At the same time, larger questions about the broader impact of civil society in democratic development loom large in a policy environment that continues to favour private action to address social problems 32 Please read the Evaluation Summaries by J-PAL of Pratham’s Education Programs 2001–2016 at https://prathamusa.org/website-admin/wp-content/uploads/2017/04/ JPAL.pdf. Accessed on 05 October 2021.

172

R. AGRAWAL ET AL.

(UNICEF et al., 2013). But evaluating the interventions of the civil society organisations in India is no doubt a difficult task. Many of the outcomes of CSOs, programme interventions are long term and difficult to measure. At the same time, CSOs make other contributions to society beyond service delivery—they advocate for policy change, provide opportunities for civic engagement, organise and mobilise communities—all of which are even less measurable (ibid.). Though evaluation as a development instrument is not completely unknown to Indian society, unfortunately, could never grab the limelight within the public discourse on development in this country, except for a few sporadic and random discussions. But the good thing is that in recent years one can see that there has been a constant effort to bring the issue of data, evidence, and evaluation for effective policy-making by some web-based media platforms. India Development Review (IDR)33 is one such media platform that has been raising the issue of monitoring and evaluation in policy making and programme implementation. A recent blog article in the IDR talks about the importance of monitoring and evaluation in effective policy-making. The article on the one hand draws attention towards specific missing information on the health status of children which would be crucial to reorient current programmes under the Integrated Child Development Services of the government of India. On the other hand, the article appreciates the government initiative of establishing the DMEO under the NITI Aayog—the policy thinktank of the government of India (Bhunia, 2021). It would be extremely difficult to comment on the frequency of the use of the word ‘evaluation’ by Indian media. However, there are ample examples where the Indian media highlights the role of evaluation in programme implementation or the adverse evaluation reports on any government programme. There are a number of mainstream newspapers in India who publish the highlights of government’s flagship programmes. For example, The Hindu—one popular national daily newspaper of India—published the highlights of the evaluation of the government’s flagship Bhagyalakshmi scheme, a welfare measure linked to the conditional cash transfer in the name of the girl child (Prabhu, 2020). Times of India, one of the most popular national English daily newspapers published from almost all big towns of India, provided the news of 33 Please visit the website of IDR at https://idronline.org/. Accessed on 05 October 2021.

5

EVALUATION IN INDIA

173

the Kerala state government’s policy decision of implementing concurrent evaluation and monitoring scheme (CEMS) for all government-funded public programmes in the state (TNN, 2019). Hindustan Times of India, another popular English daily newspaper of India, publishes the news of the respective ministries of the monitoring evaluation from time to time. In the year 2013, the newspaper published a big story on the evaluation report of the Comptroller and Auditor General (CAG) on the flawed implementation, poor monitoring and evaluation of the government of India’s flagship programme Mahatma Gandhi-National Rural Employment Guarantee Act (MGNREGA) (Nichenametla, 2013). More recently, Times of India published an overview of the SDG India Index 2020– 2021 providing the status of development in connection with SDGs in each state (Pandit, 2021). Such initiatives by Indian media create a lot of awareness about development, monitoring and evaluation, and people, in general, get inquisitive to know the status of their place. A number of evaluation reports of the public programmes are available online through the website of the DMEO of the NITI Aayog.34 The evaluation reports of the programmes run by the respective ministries are available through the webpage of the respective ministries. A number of Indian states have done well in bringing the evaluation discourse in the public. One such initiative by the government of Karnataka is the Karnataka Evaluation Authority which publishes all evaluation reports online.35 A recent blog article published by the IDR emphasises the importance of programme evaluations which could improve the effectiveness of public policies and drive more efficient use of public funds. The article mentions that India needs to strengthen its evaluation system for government programmes. The authors also question and identify the major challenges in front of the government in enhancing the evaluation ecosystem of India. The authors observed three major reasons for this: (1) Several ministries and departments continue to view evaluations conservatively, treating them as an exercise in accountability rather than learning. (2) India is yet to establish minimum standards of evaluation for public programmes. Their absence leads to a wide variation in the quality 34 The Programme Evaluation Studies can be accessed at https://dmeo.gov.in/evalua tion/dmeo-evaluation-studies. Accessed on 20 February 2023. 35 All evaluation reports can be accessed here at https://kmea.karnataka.gov.in/info4/Reports/kn. Accessed on 20 February 2023.

174

R. AGRAWAL ET AL.

and frequency of evaluations and comparing the same programme across states and time becomes challenging. And (3) evaluations in India are predominantly used to validate the successes of programmes, rather than for policy and budgetary planning, and this needs to change (Mishra & Avinandan, 2020). 3.3

Participation of Civil Society in Evaluation

A recent blog article published by the IDR emphasises the importance of programme evaluations which could improve the effectiveness of public policies and drive more efficient use of public funds. The article mentions that India needs to strengthen its evaluation system for government programmes. The authors also question and identify the major challenges in front of the government in enhancing the evaluation ecosystem of India. The authors observed three major reasons for this: (1) Several ministries and departments continue to view evaluations conservatively, treating them as an exercise in accountability rather than learning. (2) India is yet to establish minimum standards of evaluation for public programmes. Their absence leads to a wide variation in the quality and frequency of evaluations and comparing the same programme across states and time becomes challenging. And (3) evaluations in India are predominantly used to validate the successes of programmes, rather than for policy and budgetary planning, and this needs to change (Mishra & Avinandan, 2020). Evaluation is indeed a critical issue for civil society organisations. However, among the smaller NGOs in India, evaluation is primarily a donor-driven activity whereas the international NGOs conduct internal evaluations of their programmes either by their own M&E team or by external evaluators. Care India conducted an ex-post sustainability evaluation of its programme (ECOI, 2018). Whereas Oxfam India uses tools like peer learning exercises for internal evaluations (Singh, 2018). NGOs often use evaluation findings for writing project proposals and programme designing. In recent years, evaluation has become a buzz word in the development discourse in India primarily as the result of the donordriven move supported by the international year of Evaluation campaign by the UN and the growing importance of achieving SDGs. However, in India, there are limited organised spaces for citizens to participate in the M&E of government programmes. Civil society has focused on “evidence- based advocacy, using social audits and public hearings to

5

EVALUATION IN INDIA

175

create spaces” (Posani & Aiyar, 2009). Accountability initiatives by the World Bank India Office (Malena et al., 2010) have played a key role in accountability. Kumar-Range (2014) cites Bangalore citizen report cards as one of the participatory and social accountability approaches from India. There is a growth in the ex-ante social accountability-led evaluations through measures like Right to Information (RTI) and Public Interest Litigation (PIL) (Nanda, 2020). In 2015, the Institute of Social Studies Trust (ISST), a not-for-profit research organisation from New Delhi, conducted a series of metaevaluations of the four government-run programmes in India with a gender transformative lens. The initiative was motivated by the recognition that in a large and diverse country, programme outcomes can vary widely, and therefore looking across a range of evaluations of the same programme in different contexts can offer valuable insights as far as the evaluation approach and frameworks are concerned (Sudarshan et al., 2015). 3.4

Demand for Evaluations by Civil Society Organisations

Development through centralised planning has been the key strategy in India since Independence. However, the need to make people and their concerns central to ‘planning’ was felt even as the idea of ‘plans’ was adopted. Civil society’s participation remained marginal—limited to giving inputs on an ad hoc basis, or to occasional ad hoc representation of a civil society member. Organised involvement of civil society and citizens as an institutionalised process in planning was rare in India. Significantly though, at the state level, the Kerala Sasthra Sahitya Parishad (KSSP) successfully led a campaign for a bottom-up planning process. It resulted in the State Planning Board decentralising the process of planning in Kerala. Though this happened on a much smaller scale, the ‘Kerala model’ became an important inspiration for civil society groups. It was in 2007 when a space for civil society was created with a National Voluntary Sector Policy 2007 that assures a formal or invited space for voluntary organisations. This too remained largely on paper. However, the ‘opening’ was seized by some civil society organisations and networks working on governance accountability—Wada Na Todo Abhiyan (WNTA), a national coalition of over 3000 organisations across states; the Centre for Budgeting and Accountability (CBGA), a think tank working towards bringing people-centred perspective on policies and government budgets;

176

R. AGRAWAL ET AL.

and the National Social Watch (NSW), a network of civil society organisations, communities, and citizens working on monitoring governance—to ‘claim’ space in the planning process. When the government began a midterm evaluation of the 11th Five Year Plan, these civil society organisations and their partners at the grassroots conducted a People’s Mid-Term Appraisal (PMTA) in 2009. An extensive Civil Society-Consultation (CSC) called “Ensuring transparency and accountability mechanisms for people’s participation in monitoring public services” was organised in 2015 to provide inputs for the Approach Paper to the 12th Five Year Plan with a representation of 149 civil society organisations.36 On the basis of the inputs from human rights activists and social action groups WNTA, Arghyam, and the National Campaign on Dalit Human Rights (NCDHR), twelve core challenges were identified by the Planning Commission, based on 340 cell matrix discussed and documented by officers of the Commission. These challenges were discussed with the CSO representatives to utilise their in-depth knowledge of the development issues. The rationale behind undertaking the civil society consultation, as provided was ‘Shaping India’s Future’. Before the above-referred exercise, CSOs were also engaged actively by WNTA in the Mid Term Appraisal of the 11th Five Year Plan and came out with a PMTA and produced two reports: (i) Voices of the People and (ii) A Review of Selected Sectors under the broad head ‘How inclusive is the Eleventh Five Year Plan. Since 2020, the DMEO of NITI Aayog started signing synergic partnerships with various organisations including evaluation think-tanks, evaluation consulting firms, VOPEs, academic institutes, government stakeholders and UN agencies. The key goals of the partnerships are to leverage existing capacities in the external ecosystem, meet M&E technical assistance needs of the Central, State Government and other stakeholders, jointly conduct high-quality evaluation and research, strengthen evidencebased policy-making, and foster a development data architecture for programme management. On the other hand, media is a constant source of demand, focusing the attention of bureaucrats and Parliament alike on the outputs/outcomes of public expenditure. The last two decades have seen a sharp increase in the

36 See https://ngodarpan.gov.in/assets/ngo_doc/booklet.pdf. February 2023.

Accessed

on

20

5

EVALUATION IN INDIA

177

number of TV and Internet-based media in India, as in most countries, as well as the rise of aggressive investigative journalism. The media may not be engaged in M&E per se but is a source of demand. The media have been proactively reporting the activities of a civil society movement, India Against Corruption, that has systematically worked to corner the government into enacting a bill to create an ombudsman (LokPal) in the central government and one for each state government (Lok-Ayukta). Another underlying source of demand for greater accountability is emerging from civil society activism around the RTI Act, passed by Parliament in 2005. A paper by Agrawal and Nair (2018) examines the evolution of the movement for transparency towards redressing grievances and holding public servants accountable to the people. It explains how three legislations—RTI, Rajasthan Guaranteed Delivery of Public Services Act of 2011, and the Right to Hearing (RTH) Act of 2012—form part of a continuum in the people’s struggle for transparency. The RTH may be traced back to the Jan Sunwai, which was a pivotal forum in the struggle for transparency because it functioned as a dialogical space between the people and the state, as well as a forum for social auditing and civic engagement. The output-related targets set by government departments for their own programmes are now available through the RTI—which gives fillip to the movement. Civil society organisations have often taken the initiative to assess the social impact of key development projects (for example, major hydroelectric projects and dams) or of a government policy (for example, programmes to address child malnutrition). Vasudha Foundation, a non-profit organisation that works for socially just and sustainable models of energy use, conducted a study of the India-Bhutan Energy Cooperation Agreements and the Implementation of Hydropower Projects in Bhutan.37 Such studies often influence government policies, both at the planning and execution stages. In the concluding paragraph, we like to highlight the fact that India has experienced a robust civil society movement in terms of voicing the issue of inclusion of marginalised sections of the citizens but unfortunately, there is a gap on their behalf in establishing the role of evidence in policy-making. Consequently, monitoring and evaluation could never 37 See http://www.vasudha-foundation.org/wp-content/uploads/Final-Bhutan-Rep ort_30th-Mar-2016.pdf. Accessed on 20 February 2023.

178

R. AGRAWAL ET AL.

take a central stage in the public development discourse neither by the civil society organisations nor by the media. However, the recent activities spearheaded by the UN agencies, impact investors, and the evaluation communities run by the evaluation practitioners created an urge for dialogues on the importance of evidence-based policy-making on one hand and monitoring and evaluation on the other. Subsequently, DMEO, the government-run evaluation thinktank of India, initiated a process of opening up a space of collaboration for learning, capacity-building, and enhancing the evaluation ecosystem in India.

4 System of Professions (Professionalisation of Evaluation) 4.1

Academic Study Courses and Training Practices

As of now, there are no full-fledged courses on monitoring and evaluation run by Indian universities. Young graduates from the disciplines like economics, social sciences, statistics, management, rural development, public policy, and other social sciences are hired by organisations to conduct programme evaluations. Some of them get an opportunity to learn and practice on the job under the supervision of matured professionals. A number of them, who are aware of such activities, get opportunities to attend national and international training workshops run by Voluntary Organisations of Professional Associations (VOPEs) like the ECOI. Very few of them also get an opportunity to attend somewhat longer and annually conducted courses like the International Program for Development Evaluation Training (IPDET) organised as a collaborative programme between the World Bank (Independent Evaluation Group), the Center for Continuing Education (ZUW) at the University of Bern, Switzerland, and the Center for Evaluation (CEval) in Saarbrücken, Germany. There are a few post-graduate programmes which include certain topics relating to M&E. For example, the Masters Programme on Development Communication and Extension run by the Lady Irwin College of Delhi University has a core course in Extension Systems and Programme

5

EVALUATION IN INDIA

179

Evaluation,38 which includes three Units on M&E, viz., Planning Development Programmes (which includes Theory of Change), Monitoring and Implementation of Development Programmes, and Programme Evaluation. A number of MBA courses in India have papers on certain specific topics like programme design and project cycle, sampling techniques, logic models, theory of change etc. Some organisations such as International Initiatives in Impact Evaluations (3ie), Sambodhi Research and Communications (webinar series of Evaluation Observatory), the Institute of Rural Management Anand (IRMA) (management development programmes), the ISST (online workshops and a 10-week self-paced online course on facilitating gender transformative evaluations), the JPAL, CLEAR South Asia (M&E webinar series), the ECOI (webinars, EvalFest events, and ad hoc programmes on request), Participatory Research in Asia (PRIA) (courses in Participatory Planning, Monitoring, Impact Evaluation), the International Institute for Health Management and Research (IIHMR) (as part of Master in Public Health programme) provide trainings on certain topics relating to evaluation from time to time. These are mostly short-term courses and vary from one to four weeks. Some of these are paid courses while some are free of cost. Shortterm programmes as well as mentorship programmes are conducted by IRMA. Webinars, workshops, and lectures on specific topics of evaluation are conducted by individual M&E specialists and/or evaluation associations in collaboration with other organisations. For instance, a series of webinars is organised by the Sri Lanka Evaluation Association (SLEvA) and the Asia Pacific Evaluation Association (APEA) in collaboration with ECOI, ISST, and other partners. Several institutions with courses in public health include M&E in their curriculum. For example, there is a three-month certificate course on M&E by the Public Health Foundation of India. It is an interactive distance learning programme.39 Prasanna School of Public Health in Manipal Academy of Higher Education teaches M&E as a part of its 2-year Master of Public Health programme.40 The International Institute 38 See Lady Irwin College. Development Communication and Extension (DCE) Master degree programme. https://ladyirwin.edu.in/courses-development-communication-extens ion-dce/#1619428657959-6ba1fbda-43fb8e28-bd70nsion. Accessed on 2 October 2021. 39 https://phfi.org/the-work/academic-programs/. Accessed on 11 July 2022. 40 https://manipal.edu/doph/program-list/mph-master-of-public-health.html.

Accessed on 02 October 2021.

180

R. AGRAWAL ET AL.

of Public Health Management, New Delhi, offers Project Management and Evaluation with 3 credits in its MPH programme.41 Tata Institute of Social Sciences, programme teaches monitoring and evaluation of health care programmes and projects in its 2-year MPH (Health Administration).42 The National Institute of Rural Development and Panchayat Raj, Hyderabad, attached to the Ministry of Rural Development has a Centre for Planning, Monitoring, and Evaluation (CPME) in its School of Public Policy and Governance. The CPME focuses mainly on the use of tools and techniques of micro planning, monitoring and evaluation including the developing of indicators, the Logic Framework Analysis for project formulation and management, M&E design, Participatory Rural Appraisal (PRA) and its tool, impact evaluation, and stakeholders’ analysis. Largely, CPME guides on planning, monitoring, and evaluation of rural development projects. Lal Bahadur Shastri Academy, which is the training ground for civil servants in India does have a short course on M&E which touches upon the very basic elements. There are also in-service add-on training programmes in M&E for civil servants. For example, the National Institute of Labour Economics Research and Development (NILERD) organises standardised M&E courses for officers of different State Governments as per their needs and demand. The International Perspective Participatory Monitoring and Evaluation (IPPME) is another three-month course by the PRIA. Its PRIA International Academy (PIA) offers advocacy programmes and training courses, among others, in participatory research and participatory monitoring and evaluation.43 NILERD, an organisation under NITI Aayog had a diploma course in M&E approved by the Ministry of External Affairs, Government of India. The course was of three months’ duration and was convened for international participants under Indian Technical and Economic Cooperation (ITEC)/Special Commonwealth Assistance for Africa (SCAAP) programmes. This face-to-face programme covered a wide range of topics relating to M&E, hands-on sessions and practical work, and 41 https://mph.iihmr.edu.in/academic/course-offerings. Accessed on 02 October 2021. 42 https://admissions.tiss.edu/view/10/admissions/ma-admissions/master-of-healthadministration-mha/. Accessed on 02 October 2021. 43 https://www.pria.org. Accessed on 17 March 2023.

5

EVALUATION IN INDIA

181

included project work. The course content comprises (a) Monitoring and Evaluation: An Overview of Concepts, Issues and Trends, (b) The Purpose and Building Blocks of Monitoring Processes: Models and Techniques, (c) Architecture of Evaluation: Theoretical Perspective, Rationale, Emerging Approaches, (d) Methodological Aspects: Sampling, Evaluation Models and Designs, Quantitative and Qualitative Approaches, and (e) Reporting, Presentation Skills and Dialogue. The course was extremely popular among international participants. A follow-up survey of alumni confirmed the utility of the course as several participants stated to have utilised the learning/got promotions/got involved in M&E activities back home. This diploma course was converted into a one-month certificate programme. At present it is discontinued due to the Covid-19 pandemic. NILERD also conducts week-long certificate programmes for State Government officials as per their needs. The contents generally included are (1) Overview of Monitoring and Evaluation, (2) Techniques of Monitoring, (3) Recent Approaches of Evaluation, (4) Impact Evaluation and Evidence based Policy Planning, (5) Evaluation Design, (6) Sampling, (7) Evaluation Models, and (8) Report Writing. In 2020, ECOI organised a basic course in M&E for undergraduate students of Delhi university which generated a lot of interest among the student community and faculty. About 90 students doing degrees and masters courses joined the programme from various colleges. The subjects covered included an overview of M&E concepts, types of evaluations, stakeholder analysis, approaches to evaluation, theory of change and logic model, the importance of indicators and their choice, techniques of evaluation, evaluation designs, development of data collection tools, evaluation report, and dissemination and ethics in evaluation. ECOI is now planning to do a survey for assessing demand for short- or long-term courses in M&E. On the basis of the established demand, online courses will be started for various stakeholders at different levels. ECOI has joined hands with Inter-Regional Initiative for Professionalization of Evaluation (IRIPE), a group of VOPEs from different countries/regions formed to work for professionalisation of evaluation established under the aegis of APEA in December 2020. This organisation has brought out country reports indicating the status of professionalisation in selected countries like India, the Philippines, Sri Lanka, Kenya etc. On the basis of these reports, countries are encouraged to work further on professionalisation and raise the standards of evaluations for enhancing its use. The IRIPE is working on the competencies of evaluators and is

182

R. AGRAWAL ET AL.

expected to have a knowledge product ready for use shortly. ECOI with IRIPE is also working on developing a repository of courses on M&E in India and neighbouring countries which will help in developing standardised training modules in due course. DMEO, NITI Aayog, has expressed interest in developing professionalism in evaluation and an MOU has been signed between ECOI and DMEO for working together. Since training and courses are not regular, and are generally geared to the needs of the training organisations, the content in various M&E training programmes is not standardised or uniform. Training imparting organisations choose topics for training or courses as per resource availability and expertise. However, some common basic topics are generally included in the training. Some workshops are specific to certain topics such as Impact evaluations, or Result Based Management, or statistical techniques and so on. Some organisations/universities in their inservice trainings include M&E-related topics. For instance, the National Academy for Agricultural Research Management (NAARM) under the ICAR, while providing in-service training to Krishi Vigyan Kendras (Agriculture Science and Extension Centres) established at the district level for providing extension services to farmers, include topics like Logic Models; impact evaluation etc. Similarly, Urdu University, Hyderabad, included various topics of M&E like evaluation designs, impact evaluation etc. in their teachers’ training programme. The ISST, New Delhi, has been conducting a ten-week-long selfpaced online course on gender transformative evaluation since 2018. The course covers modules on evaluation principles, approaches and frameworks, tools, methods and indicators, logic models and theory of change, participatory evaluation processes, etc.44 On the other hand, Sambodhi Research and Communications—an Indian research and communications organisation based out of Delhi NCR launched an eighteen-week online certificate course called Monitoring, Evaluation, and Learning for Development Professionals in 2022. The course is designed for early and mid-career professionals located in the global south.45

44 See the detailed course brochure at https://www.isstindia.org/pdf/projects/163308 8134_pdf_FGTE2_Course_Brochure_ISST_1.pdf. Accessed on 05 October 2021. 45 See the course brochure at https://sambodhi.co.in/capacity-building/course-detail. Accessed on 17 March 2023.

5

4.2

EVALUATION IN INDIA

183

Journals and Communication Platforms

No regular journals are brought out specific to M&E. Sambodhi, a research and communication organisation from Delhi started publishing the first academic journal on evaluation in India in 2012 titled “South Asian Journal of Evaluation in Practice” (SAJEP). However, the publication stopped after its third issue.46 Presently, ECOI brings out a quarterly newsletter providing information about various M&E activities, summaries of evaluations conducted, M&E resources, training organised etc. in the country as well as across borders.47 Research papers brought out by evaluators from various evaluations conducted by them get a place in reputed sectoral journals. For instance, research papers on evaluations and evaluative studies conducted on various sectors are published in Indian academic journals (Agrawal et al., 2013). Similar is the case with other sectors and policy themes like education, health and women’s economic empowerment, etc. (Pandey et al., 2020; Singh & Kumar, 2020; Singh et al., 2018, 2019). A number of papers dealing with the evaluation of impacts have also been published in the Journal of Community Mobilization and Sustainable Development from time to time. Sometimes articles on evaluation reports are published in daily newspapers. For instance, Times of India reported the achievements of States in India on various SDGs. It said that Andhra Pradesh ranked among the top three states in achieving SDGs (June 4, 2021). Andhra Pradesh scored 81 points in poverty alleviation while Kerala is the overall top performer with 75 points and Himachal Pradesh and Tamil Nadu followed with 74 points each.48 A few reports are discussed in digital media. While there is no formal policy on the dissemination of evaluation reports,49 most of the evaluation reports these days are in the public domain or on the respective websites of the ministries which can be accessed. For example, a report

46 https://www.sambodhi.co.in/south-asian-journal. Accessed on 2 October 2021. 47 http://www.ecoionline.com/newsletter/. Accessed on 2 October 2021. 48 https://timesofindia.indiatimes.com/city/amaravati/sustainable-developmentgoals-andhra-pradesh-ranked-among-top-3-states-in-country/articleshow/83222950.cms. Accessed on 11 July 2022. 49 There is, however, a national policy on dissemination of statistical data, vide http:/ /www.mospi.nic.in/sites/default/files/main_menu/data_discremination/national_policy_ data_dissemination_6jan1999.pdf. Accessed on 14 March 2023.

184

R. AGRAWAL ET AL.

on a mid-term study of the Performance of Mission Antyodaya Gram Panchayats is available on the Ministry of Rural Development’s website.50 Other communication platforms are discussions during webinars, conferences, lectures etc. that are convened from time to time. ECOI provides a good platform for such discussions during its biennial Evalfest.51 Individual evaluators also bring out books on the subject from time to time. Agrawal (Brahmachari et al., 2015) co-edited a book ‘Evaluation for Sustainable Development - Experience and Learning’. Various evaluation reports which do not have copyright are shared on various platforms,52 networks, among VOPEs etc. For example, the report on the Impact of the Prime Minister’s Awas Yojana (Housing scheme) is available on the Ministry of Rural Development’s website. In addition, in 2015, ECOI introduced a newsletter as another important evaluation-specific exchange platform. This now has 14 issues, only some of which are publicly available on the website (as of 2020).53 4.3

Professional Organisations

The ECOI came into being as an association of professional evaluators in 2015 in the wake of the celebrations of the Evaluation Year in 2015 globally and EvalWeek in India. Its motto is ‘Share and Learn’. ISST, which is a registered trust, is hosting ECOI for administrative and financial purposes. ECOI is recognised by the International Organization for Cooperation in Evaluation (IOCE). It has been growing since then in membership and range of activities. Its membership now stands at about 200 spread over all parts of the country. We also have members from other countries like the UK, Sri Lanka, African countries, South East Asian countries, and so on. ECOI represents different professional bodies—Non-Government Organisations (NGOs), academic and research organisations, government, students, and civil society. There are also two State Chapters of ECOI, one in Maharashtra and another in Gujarat. ECOI has organised two international conferences on selected themes 50 https://rural.nic.in/sites/default/files/MissionAntyodayaStudyReportByNIRDPR. pdf. Accessed on 20 February 2023. 51 http://www.ecoionline.com. Accessed on 20 February 2023. 52 https://rural.nic.in/sites/default/files/impact_of_pmay-g_on_icome_and_empoly

ment_report.pdf. Accessed on 02 October 2021. 53 http://www.ecoionline.com/newsletter/. Accessed on 12 July 2022.

5

EVALUATION IN INDIA

185

of evaluation, EvalFest 2018 and EvalFest 2020, both well-attended and well-received by the international community of evaluators. EvalFest 2022 is now taking shape as a joint event with APEA organising its 3rd conference together with EvalFest. These events in particular provide young evaluators with an opportunity to showcase their innovative approaches to evaluation and enable dissemination of the new advances in data collection, processing, and analysis. The governance of ECOI is unique and is in the hands of a core group consisting of about 9 to 10 members without any hierarchy attached to them. All decisions are taken collectively. ECOI believes in establishing networks and partnerships in order to work together. The number of such linkages is increasing over the years. ECOI signed MOUs with SLEvA, InDec, and the Afghan Evaluation Society and looks forward to forging such relationships with some more Evaluation Associations. It has also entered into a collaborative agreement with DMEO, and the two organisations have charted out annual action plans to work in collaboration for several evaluation activities. ECOI has been networking with UN agencies, educational institutions, NGOs, and community organisations. The association is currently working out an agreement with Sri Sri University in the state of Odisha for cooperation in evaluation capacity-building and awareness generation among students. ECOI has produced a number of knowledge products. These include books, resource packs, and quarterly newsletters (see Fig. 3). The newsletters, apart from featuring important developments and events relating to ECOI and evaluation, include a list of indicative reports, books, articles, guidance documents, etc., published anywhere. EvalYouth India has emerged as a part of ECOI to enable networking and to provide capacity-building opportunities to young and emerging evaluators (YEE) in the country and as a platform to connect interested YEEs with global networks and actions. The EvalYouth India chapter was established in November 2020, by the EvalYouth Asia chapter with grant support from the EvalYouth Global Network. EvalYouth India is the youth wing of the ECOI, the national VOPE in India, which is supporting it at every stage. It worked closely with APEA, SLEvA, ECOI, EY Asia, and other EY country chapters to conduct the first-ever Asia Pacific Virtual Winter School for YEEs 2021 in March 2021. As a special initiative, EY India has initiated the first-ever National Evaluation Case Competition in the Asia Pacific region to spread awareness about M&E among college students. As a part of this initiative,

186

R. AGRAWAL ET AL.

Fig. 3 Publications from ECOI and partner organisations (Source Various newsletters published by the Evaluation Community of India [ECOI])

capacity-building webinars have also been held. For this event, EY India is partnering with various agencies and academic institutions in the country to have a broader expert base and outreach. 44 teams of evaluators have participated in the competition and their entries are currently being evaluated. EY India has also organised one webinar with SLEvA and EY Sri Lanka on the Basics of Data Collection and recently hosted one gLOCAL Evaluation Week 2021 event with ECOI and ISST (Fig. 4). One of ECOI ‘s objectives is to develop capacities. Its members are invited as resource people at various platforms in person as well as through webinars, it also organises demand-based seminars and workshops/ trainings etc. on various relevant topics in evaluation. 4.4

Existence and Compliance with Standards and Quality Obligations

The country is generally using international guidance documents (UNEG, OECD-DAC, etc.) as a reference in formulating TORs for

5

EVALUATION IN INDIA

187

evaluation and in conducting evaluations. Sponsoring agencies prescribe quality and standards in TORs which vary from project to project. It is presumed that once an evaluation policy is en vogue and accepted, standardisation of evaluation will take place. There is no certification system in the country as of now. Ethical Standards in Evaluation—A guideline for Evaluators has been developed by members of ECOI in collaboration with IRIPE. The document makes an attempt to provide guidelines for the evaluators as well as commissioners and funders of evaluation, to ensure that all evaluations are conducted in the best ethical environment54 (Fig. 4). To sum up this section, the supply of evaluation professionals in India is not from academic institutions offering specific programmes in M&E, but

Fig. 4 Poster: launch of IRIPE in 2020 (Source Access the document at https:/ /twitter.com/IRIPE_1/status/1335446026652676096/photo/1. Accessed on 05 October 2021)

54 Access the document at https://drive.google.com/file/d/1NECvuNAJW9gT9h WRGZHxQWzZo3XF0nW_/view. Accessed on 20 February 2023.

188

R. AGRAWAL ET AL.

from professionals from a medley of disciplines with or without training in evaluation added-on, which is generally of short duration. Most evaluators pick up the tricks of the trade through on-the-job experience. This situation had gone unquestioned till a few years back, and to a major extent it continues today, as commissioners of evaluation have never faced any shortage of evaluators with background in other relevant fields. There is no national evaluation policy setting out professional evaluators’ qualifications and standards. The demand for and supply of evaluation professionals has been managed by Adam Smith’s ‘invisible hand’, and there is always a pool of qualified statisticians, economists, sociologists and the like, with some training in evaluation thrown in, to draw from. It is only after evaluators became more organised through associations, there is a growing awareness of the need for professionalism in the field. Advocacy for including evaluation as a discipline of study in higher academic institutions has been on the agenda of, for instance, ECOI. ECOI and its youth wing EvalYouth India have also been pressing for the adoption of a national evaluation policy establishing standards for professionalism in the field and also a systematic capacity development plan for evaluation commissioners, managers, and evaluators on the basis of felt needs. Unless a substantial need for evaluators is established ensuring adequate career opportunities for the youth in that field, and a curriculum is developed, it may be difficult to convince higher educational institutions of the utility of offering courses in evaluation. No separate needs survey has yet been done in India, but under the Developing Partnerships for Evaluation Capacity Building theme of the Asia Pacific Regional Strategy initiative, a limited survey is underway in countries of the region, including India. This survey is expected to be completed in 2021 itself.

5

Conclusions

The above brief survey of the M&E scene in India shows that even in the absence of constitutional or legislative mandates, or national executive orders that formally institutionalise evaluation, the evaluation profession and practice have flourished in the country over the past seven decades responding to the needs as and when they are visualised. The evaluations are primarily country-led and, in spite of the lack of national standards or guidelines, have been evolving methodologically drawing on international developments, and adopting a mix of quantitative and qualitative approaches. This trend has been strengthened as the implementation of

5

EVALUATION IN INDIA

189

evaluations commissioned by public organisations is increasingly being entrusted to civil service organisations and independent research institutions. However, this transfer of evaluation practice to independent agencies does not automatically ensure that the evaluations conducted are truly independent, as the commissioners of evaluations, because of their control over resources and final decision-making, still could exercise an extent of leverage and influence the findings and conclusions. There are also some questions unanswered about the quality of evaluations conducted and their eventual utilisation in decision-making. Not many decisive meta-studies have been conducted to assess the quality of evaluations and, in the absence of a tracking mechanism, on the extent of evaluation use. Some positive developments in recent years augur well for the future of M&E in India. The establishment of DMEO as an office attached to the NITI Aayog with the mandate of coordinating and systematising M&E functions in the government and ushering in a strong national evaluation ecosystem could eventually lead to the establishment of national quality standards and national policy of evaluation. The instructions issued by the national Ministry of Finance in recent times have also sharpened the M&E function. The emergence of ECOI in 2015 as a national VOPE in the non-government sphere and of its youth wing EvalYouth India in 2020, and their capacity-building and other promotional activities in partnership with DMEO have expanded the horizons of evaluation in the country. The adoption of SDGs globally and nationally in India has no doubt developed a sense of urgency in promoting M&E to gauge progress towards the goals and has led to a further strengthening of the already strong national statistical system to enable the monitoring of numerous parameters. However, in spite of prodding by international institutions to conduct and utilise the evidence from influential evaluations to enhance the quality of decision-making in development interventions, there is a need to collect and collate evidence to show that the country is willing to use evaluative evidence in reporting progress towards SDGs. This could be possible as evaluations are institutionalised and more reliable evidence flows from them. It is not only in the government sector that evaluation has been gaining ground. Civil society organisations in the development sector are increasingly adopting evaluation as a means of judging the efficacy of their

190

R. AGRAWAL ET AL.

own functioning and raising their performance. The media too report evaluative evidence and spur action based on such evidence. While there are, thus, some positive developments in the evaluation scene from the demand side, the supply side needs to be developed and strengthened considerably. There are as yet no formal educational streams devoted exclusively to evaluation, though the subject has definitely grown to such an extent that it deserves independent courses. Apart from a limited number of broader fields of study in higher academic institutions providing for M&E as a small part of them, knowledge and skill formation in M&E takes place mainly through short-term training conducted by national and international agencies, and acquisition of such skills on the job. In the absence of national standards, competencies and curriculum, the quality of evaluators thus raised is also varied. ECOI, the national VOPE is putting its efforts to promote professionalism in evaluation through advocacy for evaluation policy establishing standards and testing and certifying skills. DMEO is working on curriculum development and could bridge a vital gap.

References Agrawal, R., & Rao, B. V. L. N. (2011). Capacity building: The Indian experience. In R. C. Rist, M. H. Boily, & F. Martin (Eds.), Influencing change: Building evaluation capacity to strengthen governance (pp. 123–140). World Bank Group. Agrawal, R., Rao, D. R., & Joshi, G. P. (2013). Feminisation of Indian agriculture: Issues and challenges. Madhya Pradesh Journal of Social Sciences, 18(1), 27. Agrawal, V., & Nair, H. (2018). From Jan Sunwai to Rajasthan Right to Hearing Act 2012: Fostering transparency and accountability through citizen engagement. Studies in Indian Politics, 6(2), 282–296. Bhunia, A. (2021). India’s social sector has a data problem: Evidence-based policymaking in India will not improve until we make administrative data more reliable. India Development Review (IDR). https://idronline.org/indias-soc ial-sector-has-a-data-problem/. Accessed on 25 June 2021. Brahmachari, A., Agrawal, R., Ghosh, S., & Bohidar, N. (Eds.). (2015). Evaluations for sustainable development, OXFAM India. Daya Publishing House. B the Change (2016). CSR in India is now a law: A look into how things have changed one year on. https://bthechange.com/csr-in-india-is-now-a-law-250 2aa6d0daa. Accessed on 19 July 2021.

5

EVALUATION IN INDIA

191

CDC (2013). A guide to help ensure use of evaluation findings. US Dept of Health and Human Services. https://www.cdc.gov/dhdsp/docs/evaluation_ reporting_guide.pdf. Accessed on 21 July 2021. Comptroller and Auditor General of India (C&AG) (2014). Performance auditing guidelines 2014. Comptroller and Auditor General, Government of India. https://cag.gov.in/uploads/guidelines/PA-Guidelines2014-05de4f 757ca0964-06548165.pdf. Accessed on 17 July 2021. Diwakar, Y., De Mel, R., & Samarsinghe, H. (2021). A study on the status of national evaluation policies and systems in Asia Pacific region. APEA. https://asiapacificeval.org/programs/regional-evaluation-str ategy/promoting-national-evaluation-updates/. Accessed on 13 March 2023. Diwakar, Y., De Mel, R., Samarsinghe, H., Liyanagamage, M., Mahmodi, S., & Kalugampitiya, A. (2022). A study on the status of national evaluation policies and systems in the Asia Pacific region—2022. APEA. https://asiapacificeval. org/programs/regional-evaluation-strategy/promoting-national-evaluationupdates/. Accessed on 13 March 2023. Diwakar, Y., & Rao, B. (2023). Evolution of the national evaluation system in India. Asia Pacific Journal of Evaluation, 1(1), 47– 73. https://drive.google.com/file/d/1qwFszweGqvW8Wo02Rd09UMy1aE Tvl374/view. Accessed on 17 March 2023. ECOI (2018). EvalFest2018: Visibility voice and value. http://www.ecoionline. com/assets/uploads/2018/05/EvalFest-Report-2018.pdf. Accessed on 18 July 2021. Gupta, S. (2016). The role of civil society in shaping India’s development partnerships. https://www.e-ir.info/2016/01/07/the-role-of-civil-society-in-sha ping-indias-development-partnerships/. Accessed on 23 June 2021. IHD (2013). India: State of development evaluation report 2010. Institute of Human Development. https://idl-bnc-idrc.dspacedirect.org/bitstream/han dle/10625/52504/IDL-52504.pdf?sequence=1&isAllowed=y. Accessed on 21 June 2021. Jacob, S., & Mukhopadhyay, R. (2023). Critical questions for evaluation in the Indian context. Asia Pacific Journal of Evaluation, 1(1), 36–46. https://drive.google.com/file/d/1wMAC3h91p7LTE3CB2ztv d5iLS9yoT0VL/view. Accessed on 17 March 2023. Kumar-Range, S. (2014). Evaluation for development results: Implications of the governance context in South Asia. In K. E. Hay, & S. Kumar-Range (Eds.), Making evaluation matter: Writings from South Asia (pp. 15–38). Sage. Malena, C., Forster, R., & Singh, J. (2010). Social accountability: An introduction to the concept and emerging practice (English) (Social development papers, 76). World Bank Group. http://documents.worldbank.org/curated/ en/327691468779445304/Social-accountability-an-introduction-to-the-con cept-and-emerging-practice. Accessed on 17 July 2021.

192

R. AGRAWAL ET AL.

Mehrotra, S. (2013). The government monitoring and evaluation system in India: A work in progress (ECD Working Paper Series, 28). Independent Evaluation Group, The World Bank. https://ieg.worldbankgroup.org/sites/ default/files/Data/reports/ecd_wp28_india_me_0.pdf. Accessed on 23 June 2021. Ministry of Law and Justice. Government of India. (2020). The Constitution of India: Part IV—Directive Principles of State Policy—Articles. 39–43 (pp. 34– 35). Ministry of Law and Justice, Legislative Department. https://legislative. gov.in/sites/default/files/COI_1.pdf. Accessed on 17 July 2021. Mishra, A., & Avinandan, V. (2020). Evaluate schemes for better outcomes. India Development Review. https://idronline.org/monitoring-and-evaluation-pub lic-policies-rct-india/. Accessed on 25 June 2021. Nanda, R. B. (2020). Evaluation landscape in India with special reference to SDGs. In A. Brahmachari (Ed.), Emerging evaluation experiences: Way forward for achieving sustainable development goals (pp. 13–26). Daya Publishing House. Nichenametla, P. (2013). Flawed implementation, poor monitoring and evaluation affecting NREGA: CAG. Hindustan Times. https://www.hindustan times.com/delhi/flawed-implementation-poor-monitoring-and-evaluation-aff ecting-nrega-cag/story-Kum0mA0N2zvuRHmm2uu05O.html. Accessed on 25 June 2021. Pal, S. P. (2011). Development evaluation in India: An overview. http://www. desiindia.org. Accessed on 22 March 2016. Pal, S. P. (2012). Evidence of policy influence of evaluation studies: The Indian experience. http://www.desiindia.org. Accessed on 22 March 2016. Pandey, V., Singh, S., & Unni, J. (2020). Markets and spill over effects of political institutions in promoting women’s empowerment: Evidence from India. Feminist Economics, 26(4), 1–30. Pandit, A. (2021). Mizoram, Haryana & Uttarakhand among fast movers in SDG Index. Times of India. https://epaper.timesgroup.com/Olive/ODN/ TimesOfIndia/shared/ShowArticle.aspx?doc=TOIDEL/2021/06/07&ent ity=Ar00811&sk=640086F1&mode=text. Accessed on 21 July 2021. Patra, P., & Behar, A. (2015). Implications of corporate social responsibility on civil society in India. State of Civil Society Report 2015. https://ngosindia.com/ wp-content/uploads/2018/08/CSRIndia.pdf. Accessed on 21 June 2021. Patton, M. Q. (2008). Utilization-focused evaluation. Sage. Planning Commission, Government of India (1951). First Five Year Plan (1951– 1956). Government of India. https://niti.gov.in/planningcommission.gov.in/ docs/plans/planrel/fiveyr/1st/1planch6.html. Accessed on 17 July 2021. Planning Commission, Government of India. (2008). Eleventh Five Year Plan 2007–2012. Vol 1: Inclusive growth. https://niti.gov.in/planningcommission. gov.in/docs/plans/planrel/fiveyr/11th/11_v1/11th_vol1.pdf. Accessed on 17 July 2021.

5

EVALUATION IN INDIA

193

Planning Commission, Government of India. (2013). Twelfth Five Year Plan 2012–2017, Economic sectors: Vol 2. https://niti.gov.in/planningcommission. gov.in/docs/plans/planrel/fiveyr/12th/pdf/12fyp_vol2.pdf. Accessed on 17 July 2021. Posani, B., & Aiyar, Y. (2009). State of accountability: Emerging questions in public accountability in India (AI Working Paper No. 2). Centre for Policy Research. https://www.cprindia.org/research/papers/state-accountab ility-emerging-questions-india. Accessed on 17 July 2021. Prabhu, N. (2020). Bhagyalakshmi scheme changed attitude towards girl child among poor: Report. The Hindu. https://www.thehindu.com/news/nat ional/karnataka/bhagyalakshmi-scheme-changed-attitude-towards-girl-childamong-poor-report/article32933217.ece. Accessed on 25 June 2021. Programme Evaluation Organisation (PEO) (2004). Development evaluation in PEO and its impact. Planning Commission, Government of India. Rosenstein, B. (2015). Status of national evaluation policies: Global mapping report (2nd ed.). Parliamentarians Forum on Development Evaluation in South Asia jointly with EvalPartners. https://www.pfde.net/index.php/pub lications-resources/global-mapping-report-2015. Accessed on 2 March 2023. Rosenstein, B., & Kalugampitiya, A. (2021). Global mapping of the status of national evaluation policies (3rd ed.). Global Parliamentarians Forum for Evaluation. Sharma, Y. (2019). NITI Aayog invites proposals to evaluate 28 centrallysponsored schemes. Economic Times of India. https://economictimes.ind iatimes.com/news/economy/policy/niti-aayog-invites-proposals-to-evaluate28-centrally-sponsored-schemes/articleshow/69198709.cms?utm_source= contentofinterest&utm_medium=text&utm_campaign=cppst. Accessed on 21 July 2021. Shukla, A., Paul, T., & Cornish, F. (2016). Empty rituals? A qualitative study of users’ experience of monitoring & evaluation systems in HIV interventions in western India. Social Science & Medicine, 168, 7–15. Singh, A. (2018). Evaluation and learning by peers and fellow travellers—Access to natural resources and energy for forest dependent communities. In A. Brahmachari, & S. Ghosh (Eds.), New directions for evaluations: Visibility voice and value (pp. 127–136). Daya Publishing House. Singh, S., Holvoet, N., & Pandey, V. (2018). Bridging sustainability and corporate social responsibility: Culture of monitoring and evaluation of CSR initiatives in India. Sustainability, 10(7), 2353. Singh, S., & Kumar, Y. (2020). Civil society-mediated governance: Experiences of public hearings in Madhya Pradesh. Journal of Public Affairs, 22(2), e2477. Singh, S., Kumar, Y., & Arrawatia, R. (2019). Can disincentives bring accountability in governance? Experiences of Right to Public Services Guarantee Act in Madhya Pradesh State of India. Journal of Public Affairs, 19(4), e1966.

194

R. AGRAWAL ET AL.

Sudarshan, R., Murthy, R., & Chigateri, S. (2015). Introduction: Engendering meta-evaluations: Towards women’s empowerment. In R. Sudarshan, R. K. Murthy, & S. Chigateri (Eds.), Engendering meta evaluations: Towards women’s empowerments (pp. ix–x). Institute of Social Studies Trust. http:// www.isstindia.org/publications/meta_eval.pdf. Accessed on 17 July 2021. TNN (2019). Kerala: Implementation of schemes to be evaluated & monitored. The Times of India. http://timesofindia.indiatimes.com/articleshow/699 19105.cms?utm_source=contentofinterest&utm_medium=text&utm_cam paign=cppst. Accessed on 26 June 2021. UNICEF, Evalpartners, & IOCE (2013). Evaluation and civil society: Stakeholders’ perspectives on national evaluation capacity development. UNICEF, Evalpartners and IOCE. https://www.evalpartners.org/sites/default/files/ Evaluation%20and%20civil%20society_v9_final_web.pdf. Accessed on 21 June 2021. United Nations (2020). India: General information—Economic indicators. https://data.un.org/en/iso/in.html. Accessed on 19 June 2021.

CHAPTER 6

Evaluation in Japan Keiko Nishino, Yoko Ishida, and Kiyoshi Yamaya

1

General Country Overview

“Evaluation” is usually translated as “Hyoka” in Japanese. In Japan, if one says, “it is evaluated,” it means “it is positively evaluated.” On the other, if “it is negatively evaluated,” one tends to use the phrase “it is criticised.”

K. Nishino (B) School of Policy Studies, Kwansei Gakuin University, Nishinomiya, Japan e-mail: [email protected] United Nations and Foreign Affairs Studies, Kwansei Gakuin University, Nishinomiya, Japan Y. Ishida Center for the Study of International Cooperation in Education, Hiroshima University, Hiroshima, Japan e-mail: [email protected] Y. Ishida · K. Yamaya Japan Evaluation Society, Tokyo, Japan e-mail: [email protected] K. Yamaya Graduate School of Policy Sciences and Management, Doshisha University, Kyoto, Japan © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_6

195

196

K. NISHINO ET AL.

In Japan, this notion has been a barrier to promoting any type of evaluation for a long time. However, a breakthrough occurred in Japan in the 1990s following new approaches brought in by the Organisation for Economic Co-operation and Development (OECD) and the introduction of the concept and implementation of New Public Management (NPM). Around 1998, when the Management and Coordination Agency (Somu Cho: currently, the Ministry of Internal Affairs and Communications, or MIC) was engaged in the preparation of the National Policy Evaluation System, it dispatched a research mission to the United Kingdom, Australia, and New Zealand. This team studied the NPM performance measurement method. The team also visited the General Accounting Office (the current Government Accountability Office) and the Office of Management and Budget of the United States to review the status of performance measurement under the Government Performance and Results Act (GPRA) of 1993. At the turn of the century, several articles and books were published to discuss the introduction of a Japanese-style NPM. For example, Ohsumi (1999) introduced the concept, vision, and strategies of NPM, and the Policy Research Institute for Land, Infrastructure, Transport and Tourism (Kokudo Koutu Seisaku Kenkyusho: PRILIT) (2003) advocated the importance of applying NPM to Japanese government administration. For the development cooperation sector, when the OECD Development Assistance Committee (DAC) published the DAC Principles for Effective Aid in 1992, officials from the Ministry of Foreign Affairs (MOFA), staff of the Japan International Cooperation Agency (JICA), as well as aid practitioners and consultants, studied these newly minted evaluation principles. During the 1990s, JICA introduced the Project Cycle Management (PCM) method,1 a planning tool based on the logic model and logical framework. The evaluation concept and a set of evaluation criteria were regarded as the last piece in closing the project cycle. Although there were strong counterarguments and resistance within development cooperation circles, the JICA evaluation section managed to convince their counterparts and experts that the main purposes of evaluation were: (1) to improve future aid policy, programmes, and projects through feedback on lessons learned; and (2) to provide a basis for accountability, including the provision of information to the public 1 PCM method was originally produced by the Foundation for Advanced Studies on International Development (FASID).

6

EVALUATION IN JAPAN

197

(OECD/DAC, 2010). Evaluation was not intended to criticise poor performance or the unmet purposes of projects. MOFA disseminated the first edition of the Official Development Assistance (ODA) Evaluation Guidelines in 2003, and since then, MOFA has continuously reviewed and revised these guidelines. The latest version was published in June 2021 and incorporates the new criterion of “coherence.” In the development cooperation sector, evaluation practice based on the DAC guiding principles and the “Plan-Do-Check-Act” (PDCA) cycle has gradually been accepted among the stakeholders. The second component of evaluation history in Japan is related to national policy and administration evaluation. For this aspect, Yamaya (2002a) describes two fundamental concurrent streams of development. The first was performance measurement-type evaluation and public works project reviews voluntarily conducted by some progressive prefectures. The other stream was initiated by the central government to experiment with a policy review system. Both approaches took place in the late 1990s, and discussions to improve public sector management resulted in the establishment of the Administrative Reform Council (ARC) in 1996. ARC issued a final report in 1997 that led to the passing of the Central Machinery Reform Bill in 1998 and the Government Information Disclosure Act in 1999. These consecutive actions to improve administrative performance, however, resulted in the marginalisation of the policy evaluation system in Japan (Koike et al., 2006). Both Yamaya and Koike pointed out that, within such a short period, many different evaluation concepts, mythologies, and systems sprouted in Japan without deep scrutiny or consensus among stakeholders. In Japan, there was also a tendency to identify policy evaluation as a performance measurement (Yamaya, 2002a). Even with these different schools of thought and interests, evaluation culture was born in Japan in the 1990s, and the Japan Evaluation Society (JES) was established in 2000 by interdisciplinary scholars.

2 Institutional Structures and Processes (Political System) Japan, the only non-western G7 nation, has a representative democracy system with its National Diet (Kokkai) as the highest organ of state power. The National Diet is divided into two chambers: the lower chamber called the House of Representatives, and the upper chamber referred to as the

198

K. NISHINO ET AL.

House of Councillors. The National Diet members are directly elected by citizens over 18 years of age. As the sole law-making organ, the Diet is responsible for approving the national budget and ratifying international treaties. The Cabinet (Naikaku) is the decision-making and executive branch of government headed by the Prime Minister (MOFA, n.d.). The Japanese administrative system consists of three tiers, namely national, prefectural, and municipality (cities, towns, and villages) levels.2 At the national level, there are the Cabinet Office (CAO) and 12 ministries.3 2.1

Evaluation Regulations

As stated above, the Japanese public policy reform process started with the establishment of ARC in 1996 under the Hashimoto administration. Among several recommendations made by ARC, it proposed the establishment of a policy evaluation system to ensure the transparency and accountability of public administration. In 2001, the government of Japan reorganised its central machinery, and the National Diet passed the “Government Policy Evaluation Act,” as shown in Table 1. An overview of the Government Policy Evaluation Act (No. 86) has been set out in Box 1. The main objectives for governmental agencies are: 1. to achieve accountability to the public and enhance transparency, 2. to seek a shift to outcome-oriented administration that incorporates the perspective of the public, 3. to realise efficient and high-quality administration for the public. As stated in the previous section, Japan underwent major administrative reforms in the late 1990s. The reform process was conducted within four main thematic areas: (1) it should be politics-oriented, (2) it needed to reduce siloing in its approaches, (3) it should promote transparency and accountability, and (4) and it should lead to the downsizing of government administration. The arguments for transparency and accountability provided a strong foundation for introducing the Government Policy

2 As of August 2021, there are 47 prefectures, 792 cities, 743 towns, and 183 villages in Japan (MIC HP https://www.soumu.go.jp/kouiki/kouiki.html. Accessed on 13 January 2023). 3 For more information, please refer to Neary (2019).

6

Table 1

EVALUATION IN JAPAN

199

Historical development of policy evaluation in Japan

Year

Month

Event

1997 2001 2001

December January June

2001 2002

December April

2005 2007 2010

December October May

2015

April

Final report of ARC submitted to Hashimoto Government Policy Evaluation System introduced by Mori Government Government policy evaluation bill approved by the National Diet under Koizumi government Koizumi cabinet approved the Basic Policy on Policy Evaluation The Government Policy Evaluation Act enforced (Law No. 86 of 2001) The Basic Policy on Policy Evaluation revised Ex-Ante Evaluation on Regulations introduced Policy Evaluation for the Special Measures Concerning Taxation introduced Policy Evaluation Council established

Source MIC (2017)

Evaluation Act (PRILIT 2003). In the Act, policy evaluation became enforceable by law and all sectoral ministries have since been obliged to introduce “policy evaluation.” As shown in Table 1, Guidelines for ExAnte Evaluation of Regulations and for the Special Measures concerning Taxation have also been implemented (MIC, 2017, 2020). Box 1: Outline of Government Policy Evaluations Act (Act No. 86 of 2001) 1. It is the purpose of this Act to promote the objective and rigorous implementation of Policy Evaluation, to reflect the results of this evaluation in the planning and development of Policy, and to publicise information on Policy Evaluation by providing for basic matters regarding Policy Evaluation carried out by Administrative Organs. This is designed to promote effective and efficient administration and ensuring the Government’s proper discharge of its responsibility to remain accountable to the public in its operations. 2. The term “Policy” used in this Act shall mean the policies, measures, and the like, with respect to a set of activities planned and developed by an Administrative Organ to achieve a certain objective within its duties or affairs under the jurisdiction. 3. An Administrative Organ shall study and acquire information on the effects of the Policies under its jurisdiction at the appropriate

200

K. NISHINO ET AL.

4.

5.

6.

7. 8.

9.

time, and based on such study, evaluate its own Policies from the standpoints of necessity, efficiency, effectiveness, or other standpoints that special characteristics of each Policy may require. It will appropriately reflect the results of such evaluation in the planning and development of the said Policy. The Government shall endeavour to appropriately utilise the results of Policy Evaluation for the preparation of budgets and the planning and development of policies relating to the jurisdiction of two or more Administrative Organs and requiring their comprehensive promotion. The Government shall establish Basic Guidelines for implementing Policy Evaluation for the systematic and steady implementation of Policy Evaluation. The Minister of Internal Affairs and Communications shall publicise the Basic Guidelines without delay. The Head of an Administrative Organ shall draw up, based upon the Basic Guidelines, a basic plan for Policy Evaluation for a term of longer than three years but not longer than five years for policies under the jurisdiction of the said Administrative Organ. The Head of an Administrative Organ shall decide on the plan for the conduct of Ex-post Evaluation for each year. An Administrative Organ shall carry out the Ex-ante Evaluation concerning affairs under their jurisdiction when they intend to make any decisions on Policies pertaining to an individual project of research and development, public works, or ODA, or any other Policy to be specified by Cabinet Order. When The Head of an Administrative Organ has conducted the Policy Evaluation, he/she shall prepare a report containing the following items: (i) The Policy subjected to evaluation, (ii) The department or organisation that carried out the evaluation and the period that evaluation was carried out for, (iii) Standpoints adopted of the evaluation, (iv) Methods employed and results of studying and acquiring information on the Effects of Policy, (v) Matters related to acquisition and utilisation of findings of persons with relevant knowledge and experience, (vi) Matters related to materials and other information used in the course of conducting the evaluation, and (vii) Results of the evaluation. When the Head of an Administrative Organ has prepared a report on the evaluation pursuant to the provision of the preceding paragraph, he/she shall promptly forward the report to the Minister of Internal Affairs and Communications and publicise the report and its summary.

6

EVALUATION IN JAPAN

201

10. The Head of an Administrative Organ shall inform the Minister of Internal Affairs and Communications of how the results of an evaluation have been reflected in the Policy planning and development in the said Administrative Organ and make such information public at least once a year. 11. The MIC shall carry out the evaluation of a Government Policy with a view to ensuring its coherent and/or comprehensive implementation with regard to a Policy that is commonly adopted by the two or more Administrative Organs and its evaluation is deemed necessary for the purpose of ensuring their government-wide coherence, or a Policy relating to the jurisdiction of two or more Administrative Organs and its evaluation is deemed necessary for the purpose of its comprehensive promotion. 12. The Government shall prepare a report each year on the status of Policy Evaluation and on how the results of such evaluations have been reflected in policy planning and development, submit the report to the Diet, and publicise it. 13. The Government shall promote studies on methods of Policy Evaluation, etc., and take necessary measures, including training to ensure the acquisition of capable personnel as Policy Evaluation staff and the enhancement of staff capabilities. Source Adapted from MIC (2017) and an official translation in compliance with the Standard Bilingual Dictionary (March 2006 edition). https://www.soumu.go.jp/english/kansatu/evaluation/evaluation_09. pdf. Accessed on 13 January 2023.

2.2

Organisational Structure

Currently, twelve ministries, five agencies, and five commissions (shown in Table 2) conduct their own policy/programme evaluations independently, unless a particular policy area is covered by multiple ministries or agencies. In cases where there is multiple coverage, MIC collates the evaluations conducted by each individual organisation (Act No. 86). Each organisation listed in Table 2 implements its own PDCA cycle annually based on the Government Policy Evaluations Act and other guidelines. In October 2020, the Administrative Evaluation Bureau of

202

K. NISHINO ET AL.

Table 2

Organisations conducting policy evaluations

Ministries

Agencies

Commissions

Cabinet Office

Imperial Household Agency

Ministry of Internal Affairs and Communications

Consumer Affairs Agency

Ministry of Justice

Reconstruction Agency

Ministry of Finance

National Police Agency

Ministry of Foreign Affairs

Financial Services Agency

Japan Fair Trade Commission Japan Casino Regulatory Commission Environmental Disputes Coordination Commission Personal Information Protection Commission Nuclear Regulation Authority

Ministry of Health, Labour and Welfare Ministry of Land, Infrastructure, Transport and Tourism Ministry of Defense Ministry of the Environment Ministry of Agriculture, Forestry and Fisheries Ministry of Education, Culture, Sports, Science and Technology Ministry of Economy, Trade and Industry Source MIC HP Policy Evaluation Portal Site (https://www.soumu.go.jp/main_sosiki/hyouka/sei saku_n/portal/index.html#pb21). Accessed on 20 October 2021

MIC published “Reference Materials on Policy Evaluation.”4 According to this instruction manual, each organisation prepares policy system tables (logic models) for its policies, programmes, and projects, and conducts evaluations not only for the policy but also for programmes and projects. These massive evaluation exercises are, in total, called “policy evaluations” and are performed by the government officers responsible for the policies,

4 https://iss.ndl.go.jp/books/R100000002-I030086297-00?ar=4e1f&locale=en. Accessed on 21 August 2021.

6

EVALUATION IN JAPAN

203

Fig. 1 Government administrative organisations’ policy system (logic model) (Source MIC HP Policy Evaluation Portal Site. https://www.soumu.go.jp/main_ sosiki/hyouka/seisaku_n/000065209.html#label4. Accessed on 25 October 2021. Author’s own illustration)

programmes, and projects. In addition, the organisation’s head is responsible for preparing a basic plan for policy evaluation every 3–5 years and submitting the plan to the Minister of MIC (Act No. 80 of 20185 ; see Fig. 1 for an overview). As mentioned above, in Japan, “policy evaluation” is used as a synonym for the “performance measurement” of programmes and projects performed by administrative institutions (Yamaya, 2002a). This is because it is much easier for ordinary government officials to measure

5 The latest version of the Government Policy Evaluation Act was enacted in 2018.

204

K. NISHINO ET AL.

their performance rather than to evaluate outcomes.6 Ordinary government officials fill in ex-ante and ex-post evaluation sheets to monitor and measure their performances. In most cases, these performance evaluation sheets are then consolidated in the evaluation unit and forwarded to MIC. Under the overall supervision of the Minister (MIC) and Deputy Director-General for Policy Evaluation, the Administrative Evaluation Bureau and Policy Evaluation and Public Relations Division are responsible and accountable for reviewing the “evaluation reports” submitted from 22 parties and publicising them in the Policy Evaluation Portal Site. To foster government officials’ evaluation knowledge and skills, MIC organises regular trainings on policy evaluation for central and local government staff. An E-learning course is also available for government employees and some training materials have been published on the MIC website.7 Figure 2 shows the basic flow of evaluation results from the Ministries/ Agencies/Commissions to MIC, then on to the National Diet and civil society. As described above, there are no independent national M&E systems or research institutions specialising in policy evaluation. Some ministries and agencies have established their own advisory boards to review their performance and evaluation results as necessary. 2.3

Parliamentary and National Audit Structures

As shown in Fig. 2, policy evaluation is conducted within each ministry that is responsible for planning, implementing, monitoring, and evaluating its own policies and performances. The MIC consolidates evaluation results submitted by each ministry and reports to the National Diet. The MIC reviews whether the contents and method of self-evaluation conducted by each ministry are appropriate and evaluates policies that cut across multiple ministries. Just like other countries’ supreme audit agencies, the Board of Audit of Japan performs a “3E” audit, namely Economy, Efficiency, and Effectiveness. To stay in line with international trends, the Board of Audit 6 This is due to the Japanese recruitment and placement system. Most government officials are not recruited as specialists, such as evaluators. They are generalists and become responsible for evaluating their own programmes and projects. 7 https://www.soumu.go.jp/main_sosiki/hyouka/seisaku_n/portal/index.html. Accessed on 1 November 2021.

6

EVALUATION IN JAPAN

205

Fig. 2 Flow of government policy evaluation in Japan (Source MIC [2017]. Author’s own illustration)

of Japan revised the Board of Audit Act in 1997 to strengthen effectiveness audits.8 However, within the National Diet, there is no official audit system currently available for evaluating results. Besides the Board of Audit of Japan, the Government Policy Evaluation Council (Seisaku Hyoka Shingi Kai) is accountable for the quality of policy evaluations. 2.4

Evaluation Practice and Use

On the MIC website,9 the policy evaluation portal site is easily accessible and includes the icons of the CAO, twelve ministries, five agencies, and five committees (Table 1) that are obliged to submit evaluation reports to MIC. It is also easy to find the Basic Plan and Implementation Plan

8 Board of Audit of Japan, https://www.jbaudit.go.jp/english/exchange/govern.html. Accessed on 31 October 2021. 9 MIC, https://www.soumu.go.jp/main_sosiki/hyouka/seisaku_n/portal/index.html. Accessed on 31 October 2021.

206

K. NISHINO ET AL.

of concerned ministries, agencies, and committees, as well as the evaluation results. In this sense, the government of Japan has fulfilled its responsibility to implement policy evaluation and disclose the results to the public. However, as stated above, all evaluation reports use a standardised format that is introduced in the guidelines, with the emphasis placed on performance measurement. As stated in Sect. 1, Japan’s policy evaluation was encouraged and stimulated by its ODA evaluation sector. Since MOFA needs to fulfil both ODA and Policy Evaluation requirements, MOFA has a unique evaluation structure. According to Articles 6 and 7 of the Government Policy Evaluation Act, internal evaluation is conducted by relevant departments within MOFA based on the Basic Plan and the Implementation Plan. The Basic Plan is formulated every three to five years, and the Implementation Plan is formulated annually. The policy evaluation and administrative review division, management and coordination division, financial affairs division, and other relevant divisions in the minister’s secretariat work together to review and consolidate the results. To secure its objectivity, MOFA established an advisory group consisting of outside experts in 2003. The most recent MOFA policy evaluation was conducted in 2020 utilising the Basic Plan (2018–2020) and Annual Implementation Plan (April 2020–March 2021). MOFA draws up a policy system (Seisaku Taikei) table that sets out Policy (impacts)—Programme/measures (outcomes)— Project (outputs), in a theory of change diagram. MOFA applies a five-level rating system (from S as the highest score, then A, B, C, and finally D as the lowest) to evaluate its programmes (MOFA, 2020). Most MOFA programmes were given a rating of B, a middle level that means “objectives are being achieved” in the 2020 report. This 418-page report contains a full overview of Japan’s foreign policies, means, indicators, results, and comments from the Advisory Group. The report is easily accessed from the MOFA webpage. By reviewing the report, however, one can conclude that the usage of this report may well be limited to a few researchers, although it retains value as a long-term record. This is a good example of an evaluation that is carried out and is accessible, but in terms of accountability, its impact may be minimal. On the other hand, ODA evaluation derives from a different stream based on the DAC guiding principles and exists separately from government policy evaluation. In addition, Japanese ODA is implemented not only by MOFA but also by the CAO, eleven other ministries, and JICA.

6

EVALUATION IN JAPAN

207

The budget and implementation system are very complex, so MOFA conducts evaluations only at the policy level, while other stakeholders evaluate their own jurisdictions. This ODA evaluation is conducted using ODA Evaluation Guidelines that are separate from those used for government policy evaluations. The latest ODA Evaluation Guidelines (MOFA, 2021a) and ODA Evaluation Handbook (MOFA, 2021b) were published in June and July 2021, respectively, reflecting the addition of new DAC evaluation criteria. Unlike the policy evaluation report explained above, the Annual Report on Japan’s ODA Evaluation is colourful and userfriendly, with easily understandable information for Japanese and foreign audiences. This is most likely because MOFA is obliged to raise public awareness and support for maintaining the ODA expenditure level under severe budget limitations based on government debt. MOFA evaluates its ODA policies based on (1) relevance of policies, (2) effectiveness of results, (3) appropriateness of processes, (4) diplomatic importance, and (5) diplomatic impact. MOFA contracts out to consultants and entrusts academics with conducting evaluations. It reviews primary evaluation results internally before consolidating the results into an annual report. MOFA also conducts country- and theme-focused evaluations using a rating system where appropriate. While reviewing the evaluation, MOFA officials of the ODA Evaluation Division use the recommendations and lessons in further discussions with concerned parties. The other major stakeholder in ODA evaluations is JICA. It is responsible for evaluating ODA-assisted programmes and projects. JICA evaluates all projects based on its project cycle using formative and summative, ex-ante and ex-post evaluations. DAC criteria are used throughout its project cycle, and a rating scale is applied for the summative and ex-post evaluations.10 These evaluations are primarily conducted by consultants who are conversant with evaluation methodologies and OECD/DAC guidelines. JICA occasionally conducts meta-analyses of past evaluation reports as well as thematic evaluations for improving their performances.11 10 Please see the JICA’s project evaluation website for further details. https://www. jica.go.jp/english/our_work/evaluation/c8h0vm000001rdg1-att/evaluations_01.pdf. Accessed on 16 October 2021. 11 Please note that in addition to the cited references, the following sources were also used in the review for this article: CAO (2021), MOFA (2021c), and Yamaya (2002b, 2016).

208

K. NISHINO ET AL.

3 Societal Dissemination/ Acceptance (Social System) Non-governmental organisations (NGO) and non-profit organisations (NPO) are the main civil society organisations in Japan. These organisations are considered to be separate from both the public and private sectors. In Japan, there is a tendency to refer to NGOs as groups that devote themselves to international cooperation activities, while NPOs are those whose activities are focused on Japan (MOFA, 2021). Based on this distinction, evaluations conducted by NGOs and NPOs are discussed in the following sub-section. 3.1

Use of Evaluations by Civil Society

From the 1970s, NGOs have been regarded as groups of people who engage in socio-economic development, disaster management, or human rights protection in developing countries. Oxfam Japan and Save the Children Japan are good examples of local branches of international NGOs, and their performance evaluations are done within the framework of their parental organisations. On the other hand, it is not so easy for small Japanese NGOs to conduct regular performance evaluations due to financial and human constraints unless they are required to do so by donors. In Japan, the year 1995 is often referred to as “the first year of NPOs.” This was the year when many volunteers participated in rescue and recovery operations following the Hanshin-Awaji earthquake disaster (Hanshin-Awaji Daishinsai) (Tanaka, 2005). Based on the positive recognition and the demand from civil society, the Law Concerning the Promotion of Specific Non-Profit Organisation Activities was enacted in December 1998. Since then, the number of NPOs has continued to increase, with more than 5,000 NPOs being registered annually between 2003 and 2007 (Iwabuchi, 2011). As of March 2021, there are 50,896 NPOs currently certified by CAO. However, the organisational system of most NPOs is fragile. CAO conducted a public opinion survey on NPOs in 2005 and 2018. According to the 2018 survey, about 40% of NPOs have an annual income of less than 10 million Japanese yen.12 They

12 Exchange rate is 110 Japanese yen = 1 US dollar (August 26, 2021).

6

EVALUATION IN JAPAN

209

also suffer from shortages of staff, face insufficient training opportunities, and require income diversification (CAO, 2018b). NPOs are obliged to submit an annual report containing activities and a financial summary to the authority such as the Ministry of Interior or a local government. This annual practice serves as a way of monitoring NPOs’ annual activities, outputs, and financial status. Besides this annual report, not many NPOs have sufficient financial or human capabilities necessary to conduct self-evaluations. Even when their annual reports are shared through their webpages, they are seldom used for public debates. Only a limited number of stakeholders are interested in NPO evaluation results. Overall, NGOs and NPOs for international cooperation realise real term and concept of evaluation much more than the NGOs and NPOs working mainly in Japan, since most of them are trained to follow the OECD guidelines and concepts for reviewing and evaluating the outcomes and impacts. 3.2

Public Perception and Discourses on Evaluation and Evaluation Results

According to the CAO 2018 survey, 89.2% of the Japanese population responded with “I am familiar with the idea of NPOs,” and 71.5% as “I trust NPOs” (CAO, 2018a). In the previous CAO survey in 2005, 85.2% of the population responded with “I am familiar with the idea of NPOs.” However, only 30.6% answered, “I trust NPOs,” 15.7% said, “I cannot trust NPOs,” while 40.7% responded with “I do not know” (CAO, 2005). This distrust of NPOs is derived from several incidents reported in the mass media, including the misuse of grant aid received from MOFA and increasing claims against fraudulent NPOs. Differences in public opinion surveys arose from the increasingly favourable recognition of roles played by NGOs and NPOs. The Japanese public, including NGOs and NPOs themselves, observed NPOs’ contributions in assisting those affected after the Great East Japan Earthquake, providing nursing care for the elderly, and delivering support to homeless people. In addition, the umbrella organisations for NGOs/NPOs jointly made efforts to recover public trust by creating a self-evaluation checklist for improving NGOs/NPOs’ performance. These specific actions led to an increased perception of the reliability of NGOs/NPOs among the Japanese public. It should be noted that public perception has been formed based on the NGOs/NPOs’ evaluation results or based on public discourses.

210

K. NISHINO ET AL.

Most public perception developed or has been influenced by the images provided in the mass media. The NGOs/NPOs know that they need to conduct evaluations and use the results; however, this process has not yet been fully institutionalised. Most NGOs/NPOs still do not conduct evaluations. At the same time, the importance or usefulness of evaluation has not been properly understood by the stakeholders or by the public. Japanese media including newspapers, TV, and websites inform the public by using the term “evaluation,” however, a very general use of the term generally prevails. In addition, environmental impact is often discussed in the media. A recent example is the case of releasing radioactive contaminated water into the Pacific Ocean. Only in this very broad sense the public is interested in evaluation. While accordingly, the Japanese public is not familiar with the instrument of evaluation, it is only known in some NGOs and NPOs, especially in the context of international cooperation. 3.3

Participation of Civil Society in Evaluations

In Japan, the institutionalisation of evaluations has been led by the central government, as described in Sect. 2. To follow the government guidelines superficially, civil society organisations, such as NGOs/NPOs, attempt to practice evaluations, but there are no formal participation procedures that are included in the standard tools for evaluations. For NGOs/NPOs, evaluation refers to not only programme or project evaluations but also covers a wide range of concepts, such as financial and accounting checks and strategic planning (Tanaka, 2011). To measure the outcomes and impacts produced from their financial inputs, government agencies, and foundations request that NGOs/NPOs conduct programme/project evaluations (Tanaka, 2011). This means that evaluations are carried out within the evaluation frameworks specified by government agencies or other donors such as MOFA, JICA, or the Nippon Foundation. Using the funds from these organisations, experimental or quasi-experimental evaluations are conducted by NGOs/ NPOs. NGOs/NPOs are therefore implementing evaluation; however, in most cases, they tend to play the roles of information providers and not evaluators. On the other hand, in 2015, CAO established a Social Impact Evaluation Working Group to promote social impact evaluation in civil society (CAO, 2017). Although CAO might intend to accelerate the use of

6

EVALUATION IN JAPAN

211

evaluation in civil society, the actual target group is currently limited to academic and research institutions and a small number of NPOs. For most NPOs, conducting annual self-checks of their activities, diagnosing their management and financial situation, and reflecting the results in their planning for the following year seems to be the most appropriate and realistic evaluation style. As mentioned above, most NGOs/NPOs conduct evaluations to meet government guidelines. The evaluations are not designed to be used by stakeholders or the public. Consequently, the use of participatory approaches in the evaluation and monitoring system by institutions remains limited in Japan. 3.4

Demand for Evaluation Within Civil Society

For most NGOs/NPOs, who face financial constraints and understaffing, government financial support and public donations provide essential income while volunteers provide a labour force. Therefore, they tend to quickly settle any potentially embarrassing matters that arise before they come to public attention. As noted above, most NGOs/NPOs in Japan do not have sufficient capacity to conduct such evaluations and are unfortunately unable to provide sufficient opportunities for the public to participate in their monitoring or evaluation (Yamaoka et al., 2011). NGOs/NPOs could strengthen their social action programmes by improving their management and increasing opportunities for participation from society. To achieve this goal, it is essential for NGOs/NPOs to learn from the results of their basic self-evaluation. The Japan NPO Network currently promotes the use of the “Seven Necessary Conditions for Reliable Non-profit Organisations.” This includes monitoring performance and improving their financial situation but does not include anything on conducting evaluations. The Japan NGO Centre for International Cooperation (JANIC) developed a self-monitoring tool in 2008 and has revised and disseminated it among NGOs as an “Accountability Self-check 2021” (JANIC, 2021).

212

K. NISHINO ET AL.

4 Professionalisation of Evaluation (System of Professions) 4.1

Academic Study Courses and Training Practices

There are no universities or graduate schools that specialise in evaluation in Japan. However, the Graduate Schools of Public Policy at the University of Tokyo, Kyoto University, and Waseda University offer practical courses to develop highly professional human resources for planning, implementing, and evaluating policies. Other graduate schools, such as the Graduate School of Humanities and Social Sciences at Hiroshima University, and the Graduate School of International Cooperation Studies at Takushoku University, offer courses in programme evaluation. For undergraduates, faculties focused on policy studies such as the Faculty of Policy Studies at Doshisha University, the School of Policy Studies at Kwansei Gakuin University, and the Faculty of Regional Policy at Takasaki City University of Economics, teach policy evaluation subjects. It is hoped that in the future more universities will offer relevant courses and subjects to further advance the recognition of the importance of policy and programme evaluation. Evaluation-related training courses are provided as part of the academic or practical training of the subject areas such as health sciences, social welfare, and business management etc. by the quasi-government or the private organisations. The JES offers a special course to train professional evaluators twice a year. This six-day unique programme consists of (1) basic evaluation theories, (2) programme evaluation and semantic components, (3) performance measurement, (4) analytical framework, (5) reporting and use of evaluation results, and (6) specialised components, such as policy evaluation or ODA evaluation. Participants who pass the written examination will receive certification from JES as a “certified evaluator.” Central government officers, along with JICA and other public offices, regard this certificate as proof of their professionalism in the field of evaluation. JES has concluded a Memorandum of Understanding with Doshisha University and the Takushoku University to certify students who have completed the specified subjects as “Young and Emerging Evaluators (YEE).” These subjects comprise the policy evaluation subject taught by the Faculty of Policy Studies (undergraduate) at Doshisha University and the programme evaluation course offered by the Graduate School of International Cooperation at Takushoku University. For national and

6

EVALUATION IN JAPAN

213

local government officials, MIC implements regular training courses on policy evaluation to promote understanding of policy evaluation among officials. MIC also organises forums, lectures, and workshops, especially for local government officials, on the importance and necessity of policy evaluations. Besides government agencies, private business schools or consulting companies also provide practical training related to monitoring and evaluation. 4.2

Journals and Communication Platforms

The Japanese Journal of Evaluation Studies,13 published by JES, and the Evaluation Quarterly,14 published by the Institute of Administrative Management (IAM), are the two regularly published evaluation-focused journals in Japan. The JES journal was first published in 2001, and the latest publication, volume 21, number 1 was published in March 2021. Each issue volume contains a special feature proposed by the members. All articles are critically peer reviewed by the assigned members and the journal is published twice a year. Most of the articles are in Japanese. Although English articles are also accepted, there have only been a few articles by researchers from other countries so far. The Evaluation Quarterly of IAM was first published in April 2007 and is especially aimed towards policy evaluation scholars and practitioners. The latest issue, published in July 2021, contains the special feature “Twenty Years of Policy Evaluation.” The Journal of International Development Studies sometimes publish evaluation articles on development cooperation. The annual conference on evaluation organised by JES has become an essential communication platform for both academia and practitioners. The JES also organises subcommittees for its members, such as the Subcommittee on Social Experimentation, on Ethics and Standards, and on School Evaluation. Members of these subcommittees hold frequent sessions to study and exchange relevant subjects.

13 Japanese Journal of Evaluation Studies, https://www.jstage.jst.go.jp/browse/jjoes. Accessed on 16 October 2021. 14 Evaluation Quarterly (only in Japanese), http://www.iam.or.jp/quarterly_ev.html. Accessed on 16 October 2021.

214

K. NISHINO ET AL.

4.3

Professional Organisations

As stated above, JES was established in September 2000 with the aim of promoting evaluation through the systematising of evaluation and reviewing evaluation methodologies. JES covers all evaluation-related academic fields, publishes academic journals, implements training courses, honours contributors, and organises annual conferences, symposiums, seminars, and subcommittee study groups. As of August 2021, there are over 500 members, with 30% from academic institutions and 70% from government offices, JICA, and consulting firms. There are also several other professional societies, namely the Association for University Evaluation, the Japanese Society for the Study of Educational Objectives and Evaluation, the Japan Academy of Nursing Evaluation, the Japan Society for International Development, and IAM. These societies extend research into specific and relevant fields of evaluation; however, there are no formal interactions between these organisations and societies. 4.4

Compliance with Quality Standards and Obligations

The JES has published the “Ethics Guidelines for Evaluators” on its webpage to encourage compliance, and to promote the quality and professional attitude of evaluators. These guidelines are applied on a voluntary basis in both internal and external evaluations by contributing to the healthy development of organisations and societies. The JES guidelines list four Principles of Ethics (Responsibilities for general and public welfare, Integrity, Respect for people, and Independence) as well as three Principles of Ethics (Systematic Inquiry, Unity, and Competence). However, the guidelines do not provide penalties for violating rules. The JES has tried to disseminate the guidelines to the members and to the relevant organisations; however, the guidelines are not yet well-known.

5

Conclusions

Twenty years have passed since Japan enacted the Government Policy Evaluation Act. Since then, where required, central and local government offices have made great efforts to institutionalise the use of policy and performance evaluations. The two-decade anniversary provides a good opportunity to assess the outcome of this progress. As featured in the latest volume of the Evaluation Quarterly, evaluation and discussions on

6

EVALUATION IN JAPAN

215

policy evaluation among the scholars and practitioners remain heated in Japan. Most evaluation reports are written using official language and are difficult to read and understand, as shown in the case of MOFA. In the past twenty years, policy evaluation has become a routine task, is losing its substance, and may not be regarded seriously even among government officials. In 2020, a book titled Why Policies Cannot Be Evaluated was published (Nishide, 2020). In the book, Nishide raised three reasons why the government policy evaluation system is not functioning as intended. The first reason is related to the organisational structures, such as the lack of a professional evaluator in the organisation. Moreover, evaluation posts are not very attractive for career development, and any findings critical of the organisations can easily be edited out due to the bottom-up reportmaking processes of many organisations. The second reason is related to the nature of the evaluation method itself. In other words, there is no perfect evaluation that will satisfy all users. Japan’s internal evaluation system is also a root cause for losing substance in the results. The third reason is that, in the current policy evaluation system, the connection between evaluation and planning is not well institutionalised. This means that evaluation results are not well reflected in planning (Nishide, 2020). Currently, many government officials are forced to devote themselves to policy evaluation. They may receive on-the-job training; however, it is often easier for them to follow their predecessors’ evaluation report format without engaging in any deep thought about the users and the feedback. As discussed in Sect. 4, some universities offer academic training on evaluation. However, since policy evaluation is conducted through internal evaluations and is done by government officials, there are few opportunities for YEE to conduct policy evaluations. The social dimension of evaluation is still unfavourable. The public demands an evaluation of government policies and programmes. However, their interest remains centred on the output of current policies and programmes and not in the direction of future policies or programmes. For civil society, including NGOs/NPOs, it is not easy to conduct regular self-evaluation due to financial and human resource constraints. To improve the situation, it will be necessary to create an independent evaluation office and introduce an external audit system. However, “is it worth doing?” is the question being asked among the stakeholders, including the academic community.

216

K. NISHINO ET AL.

References Cabinet Office (CAO) (2005). NPO houjin nikansuru yoron chosa [Public opinion survey on NPOs]. Cabinet Office, Government of Japan. https:/ /survey.gov-online.go.jp/h17/h17-npo/index.html. Accessed on 25 August 2021. Cabinet Office (CAO) (2017). Shakaiteki Inpakuto Hyoka no Fukyu Sokushin nikakaru Chosa [Study report on the promotion and dissemination of the social impact evaluation]. Cabinet Office, Government of Japan. https://www.npo-homepage.go.jp/uploads/h28-social-impact-sok ushin-chousa-01.pdf. Accessed on 25 August 2021. Cabinet Office (CAO) (2018a). Tokutei Hieiri Katsudou Houjin ni kansuru Jittai Chosa [Fact-finding survey report on non-profit organisations]. Cabinet Office, Government of Japan. https://www.npo-homepage.go.jp/toukei/npo jittai-chousa/2017npojittai-chousa. Accessed on 25 August 2021. Cabinet Office (CAO) (2018b). NPO Houjin nikansuru Yoron Chosa [Public opinion survey on NPOs]. Cabinet Office, Government of Japan. https://sur vey.gov-online.go.jp/h30/index-h30.html. Accessed on 25 August 2021. Cabinet Office (CAO) (2021). Cabinet Office NPO homepage. https:// www.npo-homepage.go.jp/about/toukei-info/ninshou-bunyabetsu. Accessed on 25 August 2021. Iwabuchi, K. (2011). Seisaku no Hyoka to NPO – Chiiki Keiei wo Shiza ni [The role of NPOs in the evaluation of local governance—From the perspective of regional management]. Japanese Journal of Evaluation Studies, 11(1), 105– 119. Japan NGO Center for International Cooperation (JANIC) (2021). Jiko Shinsa Gaido [Accountability Self-check (ASO) guide 2021]. https://www.janic. org/wp-content/uploads/2021/03/asc2021_guide-1.pdf. Accessed on 25 January 2023. Koike, O., Hori, M., & Kabashima, H. (2006). The Japanese government reform of 2001 and policy evaluation system: Efforts, results and limitations. Paper presented at the international conference of the international political science association at Fukuoka city, Japan. https://www.researchgate.net/pro file/Masaharu-Hori/publication/237701627_The_Japanese_Government_ Reform_of_2001_and_Policy_Evaluation_System_Efforts_Results_and_Lim itations/links/57c548d408ae496e42109ea4/The-Japanese-Government-Ref orm-of-2001-and-Policy-Evaluation-System-Efforts-Results-and-Limitations. pdf. Accessed on 20 August 2021. Ministry of Foreign Affairs (MOFA) (2020). Annual report on Japan’s ODA evaluation 2020. https://www.mofa.go.jp/policy/oda/evaluation/pdfs/oda report2020_en_single.pdf. Accessed on 2 November 2021.

6

EVALUATION IN JAPAN

217

Ministry of Foreign Affairs (MOFA) (2021a). ODA evaluation guidelines. https://www.mofa.go.jp/mofaj/gaiko/oda/files/100205689.pdf. Accessed on 2 November 2021. Ministry of Foreign Affairs (MOFA) (2021b). ODA evaluation handbook. https:/ /www.mofa.go.jp/mofaj/gaiko/oda/files/100205690.pdf. Accessed on 2 November 2021. Ministry of Foreign Affairs (MOFA) (2021c). Seisaku Hyoka Houkokusho [Policy evaluation report]. https://www.mofa.go.jp/mofaj/files/100097965. pdf. Accessed on 22 August 2021. Ministry of Foreign Affairs (MOFA). (n.d.) Japan Fact Sheet. Web Japan. https:/ /web-japan.org/factsheet/. Accessed on 22 August 2021. Ministry of Foreign Affairs (MOFA). (n.d.). ODA homepage: What is ODA? International cooperation and NGO FAQ (Frequently Asked Questions). https://www.mofa.go.jp/mofaj/gaiko/oda/shimin/oda_ngo/kyoumi/ faq01.html. Accessed on 25 August 2021. Ministry of Internal Affairs and Communications (MIC). (2003). Government Policy Evaluations Act (Act No. 86 of 2001), unofficial translation by MIC. https://www.soumu.go.jp/english/kansatu/evaluation/evaluation_09. pdf. Accessed on 24 August 2021. Ministry of Internal Affairs and Communications (MIC). (2017). Seisaku hyoka Q & A [Q & A for policy evaluation]. https://www.soumu.go.jp/main_cont ent/000359598.pdf. Accessed on 21 August 2021. Ministry of Internal Affairs and Communications (MIC). (2020). Seisaku hyoka ni kansuru shiryoushu [Reference for policy evaluation]. https://www.soumu. go.jp/main_content/000716072.pdf. Accessed on 21 August 2021. Nishide, J. (2020). Seiaku wa Naze Kenshodekinai noka [Why policy cannot be evaluated]. Keisoshobo. Neary, I. (2019). The state and politics in Japan (2nd ed.). Wiley. OECD/DAC (2010). Evaluating development cooperation: Summary of key norms and standards (2nd ed.). https://www.oecd.org/dac/evaluation/41612905. pdf. Accessed on 23 August 2021 Ohsumi, S. (1999). New Public Management, Rinen, Bijon, Senryaku [New public management: Concept, vision and strategies]. Nihon Hyoron Sha. Policy Research Institute for Land, Infrastructure, Transport and Tourism. (2003). Wagakuni ni okeru NPMgata Gyosei Kaikaku no Torikumi to Soshikinaibu no Manejimento [NPM-style administrative reform and internal management in Japan]. Kokudo Kotsu Seisaku Kenkyu 17 . Tanaka, Y. (2005). NPO to Shakai wo Tsunagu [To connect NPOs with society]. University of Tokyo Press. Tanaka, Y. (2011). Ekuserento NPO Kijun – Kadai Kaiketsu toshiteno Hyoka [Assessment system for excellence of non-profit—Assessment for problem solution]. Japanese Journal of Evaluation Studies, 11(1), 3–19.

218

K. NISHINO ET AL.

Yamaoka, Y., Tajiri, Y., & Jodoi, A. (2011). Shinrai-sareru NPO no 7-tsu no Joken [Seven necessary conditions for reliable non-profit organisations]. Japanese Journal of Evaluation Studies, 11(1), 21–30. Yamaya, K. (2002a). The review of policy evaluation in Japan—1996 to 2002. Japanese Journal of Evaluation Studies, 2(2), 3–15. Yamaya, K. (2002b). The art of policy evaluation in Japan. In J. Furubo, R. Rist, & R. Sandahl (Eds.), International atlas of evaluation (pp. 337–355). Transaction. Yamaya, K. (2016). Seisaku Hyoka no Riron to Tenkai [Theory and practice of policy evaluation]. Kouyou Shobo.

CHAPTER 7

Evaluation in Korea Nowook Park

1

General Country Overview

The Republic of Korea is located in North-East Asia. The Korean peninsula was divided into the Republic of Korea (South Korea) and the People’s Republic of Korea (North Korea) after the Second World War. 44.6% of the Korean peninsula belongs to the Republic of Korea (hereafter Korea). Its population is about 51 million, while the combined population of the Koreans is about 73 million. Its land size is close to the size of Portugal or Hungary. Korea is a member of the OECD and G20. Its economy ranks as the world’s tenth-largest by nominal GDP as of 2020. The Korean political system is a presidential system with 5-year single term and has unicameral legislature. President, National Assembly members, governors of local governments, and local Assembly members are elected by direct vote. Like other countries with a presidential system, the Korean government has three independent branches, which are executive, judiciary, and legislative branch. The central executive offices consist of Supreme Audit Office (Board of Audit and Inspection, BAI), Prime Minister’s Office (PMO),

N. Park (B) Korea Institute of Public Finance, Sejong, South Korea e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_7

219

220

N. PARK

10 commissions, 18 ministries, and 22 agencies (5 small ministries and 17 sub-ministries). Its local governments include 8 metropolitan cities and 9 provinces. Korea established its policy evaluation systems in 1961 with the beginning of its economic development plan (Hur 2013). The first evaluation system, known as the Policy and Program Assessment System (PPAS), was designed to monitor and evaluate the progress of the Five-Year Economic Development Plan. The Prime Minister’s Planning and Coordination Office was in charge of operating the evaluation system. This regime lasted until 1981. With the inauguration of the new government in 1981, the Economic Planning Board (EPB) took over the role of programme assessment from the PMO. In 1990, the new type of policy evaluation system was introduced with a focus on various social areas such as cultural affairs, social welfare, and environmental protection. Due to the broadened scope of policy evaluation, Korea had two tracks of policy evaluation systems: one was focused on the economic development programmes, while the other was concentrated on each ministry’s key public policies. With the consolidation of EPB and the Ministry of Economy and Finance into a single ministry in 1994, the PMO became the primary management agency of the programme assessment. In 2001, the Government Performance and Evaluation Act was enacted. Under this Act, each ministry submits one or two key public policies for the PMO’s assessment. The assessment was conducted by the Policy Evaluation Committee that ranked each ministry’s policy execution efforts based on the assessment results of key public policies and citizen satisfaction on each ministry. This policy evaluation system was more focused on the evaluations on an individual institution rather than on policies. In 2006, the Government Performance Evaluation Act (GPEA) was enacted. It introduced two types of policy evaluation for central ministries: self-assessment and top-down assessment. This framework intends to encompass all the evaluation activities in the executive branch, albeit with a limited degree of success. Currently, the self-assessment covers the assessment of major policies, budgetary programmes, and administrative capacity. These assessments are done by each ministry’s self-evaluation committee which consists of 10 to 30 external experts and internal members. The top-down assessment is led by the PMO and focuses on 100 national agenda, regulation reforms, government innovation, and communications on policy with citizens. The top-down assessment is close

7

EVALUATION IN KOREA

221

to periodic monitoring and assessment of each ministry’s efforts on the incumbent government’s agenda. Within this policy evaluation framework, the Ministry of Economy and Finance (MoEF) operates its own performance management and evaluation policy for budgetary programmes since 2006. It established and implemented its own comprehensive evaluation systems. The following explanation is mostly based on these two sets of evaluation systems run by the PMO and MoEF with additional reference to evaluation systems run by sectoral ministries.

2 Institutional Structures and Processes (Political System) 2.1

Evaluation Regulations

While there is no reference to evaluation policy in the constitution, the GPEA establishes the government-wide evaluation policy framework. This law applies to the ministries and agencies in the central government, local governments and public enterprises, and quasi-governmental organisations. It defines two types of policy evaluation for central ministries: selfassessment and top-down specific assessment. It mandates the ministries in the central government to carry out self-assessment processes. It also imposes a top-down assessment focused on the presidential agenda and institutional capability. The self-assessment was conducted for the major policies, budgetary programmes, administrative capability of 46 central ministries and agencies. The top-down assessment was focused on the presidential agenda, regulation and government reforms efforts, and policy communication efforts. The GPEA consists of six chapters: general provisions, provision of the government performance evaluation systems, types of government performance evaluation and procedures, support to building infrastructure of government performance evaluation, use of evaluation results, and supplementary provisions. The general provisions provide definitions of key concepts, principles of government performance evaluation, and ministries’ responsibility about strategic plans and annual performance plans. In the chapter on the provision of the government performance evaluation systems, the following are stipulated: (1) PMO’s responsibility about preparing the medium-term plan of government performance evaluation, (2) mandates of the committee for government performance

222

N. PARK

evaluation, (3) composition and operation of the committee, and (4) obligation of data provision by ministries and establishment of information systems for the evaluation. The chapter on the types of government performance evaluation and procedures provides regulations on the selfassessment and top-down specific assessment. The chapter on the support to building infrastructure of government performance evaluation stipulates that the government can provide necessary support to ministries for building capacity of government performance evaluation including budgets and staff. The chapter on the use of evaluation results requires the evaluation results should be open to the public and used for budgeting and personnel management. It also mandates that the head of ministries should take corrective actions for identified issues during the evaluation process. While the PMO sets the government-wide evaluation policy framework with the GPEA, the MoEF establishes its evaluation policy on government budgetary programmes with the National Finance Act (NFA). The NFA was enacted to provide legal foundations for the public financial management reforms introduced during mid-2000s (Kim & Park, 2007; Park & Choi, 2013). The comprehensive reform measures included medium-term expenditure framework, top-down budgeting, Progamme budgeting, performance-informed budgeting, integrated financial information management systems, and accrual accounting (Park, 2013; Park et al., 2015). There are four layers of the performance management and evaluation of budgetary Progammes (Park, 2018). For the annual performance monitoring, ministries are required to submit an annual performance plan and report which includes performance indicators and targets of their budgetary Progammes. For the in-year monitoring and performance management, a focused quarterly monitoring has been conducted to identify and tackle issues in Progamme implementation. For the annual self-assessment of budgetary Progammes, checklist-based self-assessment has been done by ministries, of which results are finalised by the MoEF. Lastly, in-depth evaluation has been conducted for the selected crosscutting Progammes. Every year about 8–10 Progamme evaluations are conducted with the purpose of evaluating Progammes’ effectiveness and efficiency (Fig.1). While these performance management and evaluation systems were stipulated in the NFA and its enforcement decree, the recent revision of the NFA consolidated them under a separate chapter (Park, 2019). The

7

EVALUATION IN KOREA

223

Annual Monitoring Annual performance plan & report

Performance indicators & targets

In-year Quarterly Monitoring & Performance Management Focus on about 80 sub-programmes

Quarterly monitoring + Annual Assessment

Budgetary Programme Annual Self-Assessment (BPA) Line ministries’ self assessment

Checklist based assessment

In-depth Evaluation Focus on Crosscutting Programmes

Programme evaluation + Spending Review

Fig. 1 The four layers of performance evaluation & management systems of budgetary Progammes (Source Park [2018])

revised NFA also added other elements to strengthen accountability and capacity-building efforts into the evaluation systems. The newly added chapter of the NFA includes the following: (1) the principles of performance management of budgetary Progammes, (2) requirements of the medium-term performance management plan, (3) ministries’ responsibilities of performance management, (4) elements of performance management and evaluation, (5) use of performance information, (6) support for evaluation capacity building, and (7) requirement for open data. While the GPEA and the NFA provide framework for the performance management and evaluation in the government, practical guidance is developed for the actual implementation of the evaluation. For example, there are manuals for annual performance plan and report, self-assessment of budgetary Progammes, and in-depth evaluation. For the performance management and evaluation of budgetary Progammes, evaluation committees consisted of experts who provide feedback on the quality of performance information and finalise evaluation results. For the indepth evaluation, a working party consisted of external experts do actual evaluation work, while evaluation agenda and results are reported to the committee chaired by the vice minister of the MoEF. All the evaluation

224

N. PARK

activities conducted by the MoEF is supported by the secretariat function in the Korea Institute of Public Finance (KIPF), which is a public thinktank. There are many sectoral evaluation activities in the area of Research and Development (R&D) Progammes, employment Progammes, small and medium-sized business Progammes, schools, universities, social service providers, and so on. Almost all the evaluation activities have their own legal foundation and guidance. For example, the public R&D sector and the small and medium-sized enterprise Progammes have their own evaluation systems with legal foundation. The evaluation policy of the public R&D Progammes is stipulated in the Performance Evaluation and Management Act of the National R&D Progammes, which was enacted in 2005. The evaluation of small and medium-sized enterprise Progammes is stipulated in the Framework Act of the Small and Medium-sized enterprise. Usually and across different sectors, a group of external experts or public thinktank conducts evaluations, while the results are often finalised by evaluation committees when they are formed. 2.2

Parliamentarian and National Audit Structures

The National Assembly has their own supporting institution for budget analysis and Progamme evaluation, which is the National Assembly Budget Office (NABO). The NABO was established to support the National Assembly for budgetary issues. As of 2021, it consists of 138 staff members and conducts budget analysis, revenue forecasts, cost estimates, and Progamme evaluations. The National Assembly can commission the NABO, the Board of Audit and Inspection (BAI), the external experts, and research institutes to conduct an evaluation. It can request the NABO to conduct a particular evaluation but the process is internal rather than openly tendered. The selection of specific evaluation topics is ultimately at the discretion of the NABO, which may refuse the National Assembly’s requests in principle, but usually accommodates them. Evaluation information is used in the budget approval process on an ad hoc basis. The National Assembly members have access to various kinds of evaluation and performance information. First of all, the annual performance plans and reports from ministries and agencies are submitted to the National Assembly members. They include information on the budgeted

7

EVALUATION IN KOREA

225

amount, Progamme logic, targeted groups, performance indicators and targets of spending Progammes, and a Progamme context. Since they are mandatory documents, they are available to the National Assembly members without request. The National Assembly members can request other evaluation information to ministries and public organisations. The BAI (Board of Audit and Inspection), the supreme audit office, conducts performance audit in addition to the traditional compliance audit. The official tasks for the BAI consist of audit and inspection. The inspection is more like internal audit in many countries, which focuses on compliance issues in the public organisations. However, the BAI has been trying to extend performance audit as one of its official mandates. Although it is not explicitly stipulated in the regulation of the BAI, the BAI incorporated evaluation approach into its audit and inspection. 2.3

Organisational Structure

One of the PMO’s main function is to conduct government performance assessment at an organisational level. The office of government performance assessment is responsible for managing assessment activities stipulated in the GPEA. It consists of three divisions, of which primary task is the management of the evaluation rather than its implementation. Actual assessment is conducted by the evaluation committees consisting of external experts and academics. The number of staff is about fifty as of 2021. In the MoEF, the Bureau of Fiscal Management is mainly responsible for the performance management and evaluation of budgetary Progammes. Like the evaluation activities in the PMO, the MoEF’s evaluation activities are conducted by the evaluation committees consisting of external experts. There are other evaluation systems run by the sectoral ministries, and their organisational structure is similar to that of the PMO and MoEF in that a unit in the ministry runs the system and actual evaluation is conducted by the external evaluation committee. Usually, the term of the evaluation committee members lasts two to three years and the unit in the ministry plays a steering role, in that they set the scope and directions of the evaluation. There are other important actors in running the evaluation systems in the Korean government, which are public research institutes. They usually play a secretariat role by providing research, consulting, training, and administrative services to the public officials and evaluation

226

N. PARK

committee members. For example, the Korea Institute of Public Finance (KIPF) plays the secretariat role for the evaluation systems of budgetary Progammes. In addition to the secretariat role, researchers at the KIPF conduct evaluations by themselves. 2.4

Evaluation Practice and Use

The PMO and the MoEF conduct evaluations on crosscutting issues and major policies every year. The evaluation system managed by the PMO and MoEF is comprehensive in terms of scope and implemented every year. In addition, there are many sectoral evaluation systems. The Ministry of Science and ICT institutes evaluation systems for R&D Progammes, and the Ministry of Employment and Labor runs evaluation system systems for job creation and employment Progammes. Sectoral committees also establish evaluation systems for sectoral policies and Progammes. For example, the Social Security Council develops a fouryear medium-term plan and annual implementation plan. To assess the progress of these plans, it conducts an assessment of annual implementation plan and selective in-depth policy evaluations. The Social Security Council manages the evaluation process and the evaluation is outsourced to external experts and public research institutes. Evaluation activities are embedded in almost every sector and evaluation results are used to justify policies or identify problems and improvement plans. Common evaluation types vary and many can be found in practice. Evaluations conducted by the PMO and MoEF are focused on impact evaluations to examine whether policies and progammes meet their intended goals, while evaluations done by the sectoral ministries are close to formative evaluation in that they aim for developing Progamme improvement plans. Evaluations are usually outsourced to public research institutes and external experts and their budgets are between USD 30,000 and USD 200,000 depending on the evaluation coverage and depth. Evaluations consisted of impact evaluation and field study cost more than desktop evaluation. The duration of evaluations is between 3 to 12 months. Impact evaluations and evaluations on crosscutting Progammes take longer and are more expensive. The PMO conducts evaluations to improve organisational performance and policy effectiveness by giving recognition and incentives to ministries and civil servants. The MoEF evaluates budgetary Progammes’ effectiveness and efficiency to use the evaluation results for budget formulation

7

EVALUATION IN KOREA

227

(Park & Jang, 2015). Sectoral ministries’ evaluations are used for budget formulation and streamlining sectoral policies. Sectoral ministries’ internal evaluations are sometimes used to justify their Progammes. Outside of the Administration, the BAI and NABO also conduct evaluations. The BAI’s evaluation is more about compliance issues based on field inspection, and the NABO’s evaluations are close to desktop evaluations by collecting data from line ministries. Their evaluations are usually notified to ministries with a requirement to report back with their rectifying plans. The quality of evaluations is guarded by the commissioning ministries and other stakeholders, because evaluations are usually outsourced. The commissioning ministries manage the evaluation process and decide to endorse the evaluation results. Due to other stakeholders’ potential challenges, the quality of evaluations is crosschecked by the commissioning ministries. The public research institutes are quite often consulted to assess the quality of evaluations by the ministries. Sometimes meta-analysis is conducted for controversial policies, but it is done on a case-bycase basis. Since Korea has many qualified researchers and strong public research institutes, the qualification of evaluators is usually guaranteed. The active use of evaluation results in policy-making and budget process greatly helps to improve the quality of evaluations, since it creates demand for quality evaluation.

3 Societal Dissemination/ Acceptance (Social System) 3.1

Use of Evaluations by Civil Society

Korea has many civil society organisations, but their role as producers of evaluations is limited. Since there is abundance of evaluation activities in the Administration, the National Assembly and the BAI, the civil society organisations are usually consumers of evaluations rather than their producers. They sometimes do evaluations by their own initiative for controversial issues, but they are the exception rather than the rule. If they conduct evaluations by their own initiative, they are for public debate and raise issues for government policies. The government usually does not commission evaluations to the civil society organisations, but the ministries invite them as advisory committee members and to the public hearings to solicit their feedback.

228

N. PARK

There are no significant hindering factors in the use of evaluations as tool for evidence-informed decision-making within civil society organisations—where it is used. For this purpose, the civil society organisations use many available evaluations results and sometimes they commission them. 3.2

Public Perception and Discourse on Evaluation and Evaluation Results

Almost all the evaluation results are open to the public, although there is a possibility that the official version of evaluation results might be watered down. Previously, evaluation results are not always open to the public and often summarised results are publicised. Nowadays, evaluation reports are posted on the ministries’ website or to the government-wide website for the commissioned research projects. If ministries commission evaluations with their budget, they are required to post the reports on the designated website except for legitimate reasons, like national security. It is fair to say that many evaluation results are available to the public, although some sensitive issues might not be addressed in a timely manner. The challenges to improve public discourse based on evaluation results are how to enhance the public awareness and trust on the evaluation. Despite the abundance of evaluations, the quality and objectivity of evaluation results are quite often questioned. Polarisation of the public opinion makes it hard to earn people’s trust in government and evaluation results. Quality evaluation results are relatively well accepted within the government and the National Assembly, but it is harder to make people accept them when people’s trust in government is low. For controversial projects and policies, evaluation results are cited by the media. For example, the evaluation results of the local currency Progammes, promoted by a presidential candidate, received much attention from media in the recent elections.1

1 There are lots of media articles on this issue. An analytical report from KIPF received lots of attention from media and the National Assembly. For example, an opinion piece from the Seoul Newspaper cites recent evaluation studies on the “local currency Progammes” and calls for further evaluations: https://www.seoul.co.kr/news/newsView. php?id=20200918500129. Accessed on 9 March 2023.

7

3.3

EVALUATION IN KOREA

229

Participation of Civil Society in Evaluations

The representatives of the civil society organisations are quite often invited to the advisory committees of government evaluations. Since many leaders of civil society organisations are professors, they sometimes play a role of evaluator. To facilitate citizen’s participation in the budget process, participatory budget practice was institutionalised in local government since 2011, and in the central government since 2018. While initially soliciting proposals of budgetary progammes was the main focus of the participatory budget, the focus has been shifted to the deliberation of complex policy issues and, to set the agenda, evaluation results were shared with citizens who participated in the participatory budget process. For the public policy-making, related civil society organisations are usually invited to the public hearings and the advisory committees to get their feedback. Their participation in the evaluation process is as follows: their opinions are solicited as one of the stakeholders, and representatives of the civil society organisations sometimes play a role of evaluator when they are qualified. Citizen’s surveys are quite often used as one of the common evaluation tools. However, evaluation itself is conducted by the experts and sometimes, citizen or stakeholders join the advisory panel for evaluations. 3.4

Demand for Evaluation Within Civil Society

Individual citizens, civil society organisations, unions, and private enterprises can request audit of the government and public organisations. Through the Citizen Audit Request System, individual citizens can request audit against the public organisations.2 Through the Public Interest Audit Request System, civil society organisations and local assembly members can request audit against illegal behavior of the Administration. The Citizen Audit Request System was introduced in 2002 and the Public Interest Audit Request System in 1996. Although they are more focused on compliance and fraud issues, they often include evaluation components.

2 The Overview and process of Citizen Audit Request System is explained under the following link at the Board of Audit and Inspection of Korea: https://www.bai.go.kr/bai/ audit/auditDemand/auditDemandStep/tab01. Accessed on 9 March 2023. The number of received requests is the following: 27 cases in 2019, 27 cases in 2020 and 28 cases in 2021, and 25 in 2022.

230

N. PARK

While these two channels provide citizens and civil society organisations to request evaluations to the BAI, they are often subject to criticism in that only 11% of the citizen audit requests and 21% of the public interest audit requests are approved for the BAI’s actual investigation during past 10 years (Lee, 2020). The Civil Society organisations argue that conditions for audit approval should be relaxed and the requesters should be given opportunities to appeal their case during the approval process. To conclude, one should note that there is no formal mechanism for civil society organisations or citizens to request evaluations in the narrow sense, while they can request audit to the Supreme Audit Institution, which regularly entails evaluative evidence or certain elements of evaluations. In addition, they can request an evaluation through the online portal of the president’s office, called the Citizens’ Proposal,3 since they can report or propose anything relevant for government intervention. Citizens can use this tool to request a certain action from government or propose a certain policy. By using this participatory governance tool, it is also possible to request an evaluation or evaluative evidence.4

4 Professionalisation of Evaluation (System of Professions) 4.1

Academic Study Courses and Training Practices

There are not dedicated degree Progammes for evaluators at the universities, but academic departments in social sciences offer Progammes for evaluation techniques, such as statistics, econometrics, survey methodology, policy evaluations, and so on. The academic Progammes closely related to evaluations are economics and public policy studies. In Korea, almost every university has economics and public policy study Progammes for undergraduates and graduates.5 In addition to the general Progammes 3 https://www.epeople.go.kr/nep/withpeople/presIntro.npaid. Accessed on 9 March 2023. 4 Finally, for a more deliberative process, there is another participatory channel which is the Citizens’ Deliberation. Citizens or institutions can make a proposal which will be subject to deliberation process among citizens, experts, and civil servants. 5 For example, the Graduate School of Public Administration at the Seoul National University offers courses on cost-benefit analysis and policy evaluation (http://gspa.snu. ac.kr/en/curriculum/school/curriculum. Accessed on 19 January 2023). The Graduate

7

EVALUATION IN KOREA

231

related to evaluations, sectoral disciplines also teach evaluation techniques, such as social welfare, public health, environment policy, labor policy, consumer policy, education, and so on. Non-academic institutions also offer training Progammes to civil servants, employees in the public organisations, and individual citizens. The public research institutes usually provide short workshops for civil servants and public employees before the evaluation starts. Some of them offer introductory Progammes on evaluation systems and techniques on a regular basis. The private consultancies also provide Progammes on evaluation to civil servants and the general public. These training Progammes are usually for civil servants and public employees who are responsible for running the evaluation systems and managing public policy and Progammes. Evaluators are mostly experts and professors who received PhD from social science Progammes. The qualification of evaluators is assessed by their academic degrees and affiliated institutions rather than certificates from training Progammes. 4.2

Journals and Communication Platforms

Policy evaluations are published not only in specialised evaluation journals but also in the journals of social sciences. There is not a journal dedicated to the evaluation itself. Considering from their main contents, there are some journals whose focus is on evaluation: International Journal of Policy Evaluation and Management (IJPEM),6 Journal of Audit and Inspection,7 and The Korean Association of Policy Studies.8 There are other journals dedicated to the evaluation of specific sectoral policies and Progammes, namely the Journal of Educational Evaluation for Health Professions 9 and the Journal of Environmental Impact Evaluation.10 The journals in the other social science also publish issues of policy evaluation School of Public Health at the Seoul National University offers courses on the health policy analysis and evaluation (https://health.snu.ac.kr/en/node/139. Accessed on 19 January 2023). Likewise, many other universities offer courses on policy evaluation and sectoral policy evaluation. 6 http://www.kapae.kr/article/. Accessed on 19 January 2023. 7 https://bai-eri.jams.or.kr/co/main/jmMain.kci. Accessed on 4 February 2023. 8 https://kaps.or.kr/. Accessed on 4 February 2023. 9 https://jeehp.org/. Accessed on 19 January 2023. 10 http://journal.kci.go.kr/kseia?language=ko. Accessed on 19 January 2023.

232

N. PARK

and evaluation methodologies. To name a few, those are the following: Journal of Public Administration, Journal of Economics, Journal of Public Finance, Journal of Fiscal Policy, Journal of Social Welfare Policy, Journal of Environmental Policy, and so on. These journals are available online and offline. Academic conferences dealing with evaluation topics are held regularly. In addition, irregular policy forums are frequently held by academic associations. 4.3

Professional Organisations

There are no volunteer organisations for professional evaluation, which may reflect that the field of evaluation itself is not professionalised and subject to other scientific disciplines. Policy evaluations are major contents of journals in other scientific disciplines and all the journals are managed by the academic associations. For example, in the fields of economics, public policy studies, education, environmental studies, social welfare studies, and public health studies, there are multiple academic organisations that publish academic journals dealing with public policy evaluations and evaluation methodologies. A small number of academic associations are dedicated to the policy evaluations. They are the association of public policy evaluation, environmental policy evaluation, and educational evaluation. The main source of evaluation services is public research institutes. For instance, the Korea Institute of Public Finance (KIPF)11 hosts a secretariat function for the evaluation systems run by the MoEF.12 Researchers from the KIPF conduct Progamme evaluations and provide research services to MoEF. Other public research institutes also provide evaluation services for their sectoral ministries. In addition, university professors do evaluations for the government. Sometimes private consultancies participate in the evaluation process, but their engagement is relatively limited compared to that of the public research institutes and academicians. The market for well-trained freelancers is limited due to well-established public research institutes and universities, but some retirees from public research institutes and universities provide evaluation services as freelancers.

11 https://www.kipf.re.kr/eng/index.do. Accessed on 19 January 2023. 12 This centre is the Center for Performance Evaluation and Management. https://

www.kipf.re.kr/cpemeng/index.do. Accessed on 4 February 2023.

7

4.4

EVALUATION IN KOREA

233

Compliance with Quality Standards and Obligations

While there are not general standards or guiding principles for evaluators in Korea, evaluators are generally required to sign a confidentiality agreement and to disclose a conflict of interest before starting a particular evaluation. When ministries commission an evaluation, they usually demand evaluators of having PhD with some professional experiences. Except aforementioned confidentiality agreement and a conflict of interest disclosure, no explicit code of conduct or standards are imposed on the evaluators. There is a certificate of policy analysis and evaluation which is issued by a private association. When ministries and public agencies commission evaluations, they do not require and give a special consideration to the certificate. As a result, the cetificate does not have much efficacy. For evaluations commissioned by the government, a PhD degree is required for the primary evaluators.

5

Conclusions

Evaluation activities in Korea are heavily influenced by the central government policies. The Korean government established extensive governmentwide evaluation systems. The central ministries, the PMO and MoEF manage comprehensive sets of performance evaluation and management systems for government policies and budgetary Progammes. Sectoral ministries and special committees run their own evaluation systems for sectoral policies and Progammes. Almost all the evaluation systems have legal bases. The NABO, a research institute in the National Assembly and the BAI, a supreme audit office, also conduct evaluations on public policies and Progammes. Evaluation services are usually provided by the public research institute and university professors, while civil society organisations are often invited to the advisory evaluation committees. Since there are many PhD researchers and professors in social sciences in Korea, the field of evaluation is not well-established as its own, and is developed as a part of other social scientific disciplines. Despite the abundance of evaluation activities and evaluators, there are challenges to produce quality evaluation results which can contribute to the Progamme improvement and resource allocation. Limited data availability, capacity of civil servants, and politicians’ support are major hurdles to produce quality evaluation results and facilitate their use for policy

234

N. PARK

making. There has been some efforts to improve the availability administrative data, strengthen responsibility and capacity of civil servants, and enhance online accessbility of evaluation results (Hwang, 2016; Park, 2019).

References Hur, M. H. (2013). Korea’s government performance evaluation system and operating experience knowledge sharing program. Ministry of Strategy and Finance. Hwang, H. S. & Cho, M. S. (2016). A study on government performance evaluation systems and its effectiveness in Korean. Korea Institute of Public Administration. Kim, M. J., & Park, N. (2007). Performance budgeting in Korea. OECD Journal on Budgeting, 7 (4), 1–11. Park, N., & Choi, J. (2013). Making Performance Budgeting Reform Work, Policy Research Working Paper 6353. World Bank. Park, N. (2013). Peformance budgeting in the republic of Korea. PREM Notes, World Bank. Park, N., & Jang, J. (2015). Performance budgeting in Korea: Overview and assessment. OECD Journal on Budgeting, 14(3), 2023. https://doi.org/10. 1787/budget-14-5jrtl4wnm1r5.Accessedon18January Park, N., Oh, Y., & Weon, J. (2015). Assessment of the performance management systems of budgetary programs in Korea, Korea Institute of Public Finance. Park, N. (2018). M&E Function in the Korean government: Overview and lessons, Presentation at the International Seminar on Governance, Policy Evaluation and Regulatory Impact, Organized by the Brazilian Government and the World Bank, Annex I Autidotirum of the Planalto Palace, Brazilia, Brazil, 25 May 2018. Park, N. (2019). Policy proposals to improve public financial management in Korean. Ministry of Economy and Finance. Lee, S. (2020, October 15). Meaningless citizen audit request, good morning today. http://www.gmitoday.com/news/articleView.html?idxno= 2109. Accessed on 12 January 2022.

CHAPTER 8

Evaluation in Nepal Asela Kalugampitiya and Gana Pati Ojha

1

General Country Overview

Nepal lies in Asia in the foothill of the Himalayas between China in the north and India on three other sides. The country occupies 147,516 square kilometres (DHM, 2015) of land and lies between approximately 28° North and 84° East. Nepal is ranked as the 93rd largest country in the world by land size (World Bank, n.d.) and 49th by population with a little less than 30 million inhabitants (Worldometer, n.d.). Nepal has a great variation in elevation ranging from 60 metres from the mean sea level to 8848 metres, all within a distance of about 150 km, resulting in climatic conditions varying from sub-tropical to arctic. Nepal has eight out of the 14 highest peaks in the world exceeding 8000 metres including Mount Everest. The population comprises of 125 distinct ethnicities/castes who speak 123 diverse languages and observe different

A. Kalugampitiya (B) Centre for Evaluation, University of Sri Jayewardenepura, Columbo, Sri Lanka e-mail: [email protected] G. P. Ojha Community of Evaluators Nepal, Agriculture and Forestry University, Bharatpur, Nepal © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_8

235

236

A. KALUGAMPITIYA AND G. P. OJHA

religions and cultures (Dahal, 2014; Yadava, 2014). It is a land-linked/ locked country, not touching any side of any sea or ocean. However, it has several rivers and tributaries with a high potential of generating electricity. Nepal has never been a colony of any foreign power but lost some parts of it to the British power in India. The present-day Nepal was formed by uniting several princely states and was ruled directly by a king for several years and by the prime ministers of one family (Rana) for 104 years till 1951. There was again direct rule of the monarch till 1990 with some exception from 1951 to 1960. From 1991 to 2006, Nepal exercised democracy under a constitutional monarchy. Thereafter, Nepal has become a republic with federal system (NLC, 2020). With the abolition of the kingship and introduction of the republic, a new constitution has been developed, which considers the rights of diverse populations and provides guidelines for their use. The constitutional commitments include adoption of democratic norms and values, including peoples’ competitive multi-party democratic governance system, civil liberty, fundamental rights, human rights, adult franchise, periodic elections, complete press freedom and an independent, impartial and competent judiciary, and the concept of rule of law. The constitution also pays special attention to gender and social inclusion. Accordingly, gender and social issues are given high priority in Nepal’s development as it is made mandatory in the constitution of Nepal and in its directive principles, policies, and state structure (NLC, 2020). To enhance implementation of constitutional provision on social inclusion, the Constitution has further made provisions for the National Inclusion Commission. The proportional representation of ethnic groups in social and political organs is guaranteed. In case of gender equality, there must be at least one-third representation reserved for women in all spheres of social and political agenda (NLC, 2020). This has been taken into consideration in implementing the development programmes and conducting evaluations, in most of the cases. 1.1

Governance System

Monitoring and evaluation (M&E), as a political agenda, is largely influenced by the type of governance system that a country practices. The history of M&E, therefore, does align with the history of a country’s governance system. Hence, we have provided a brief overview of the governance system in Nepal below. Before 1951, Nepal had allround

8

EVALUATION IN NEPAL

237

family rulers for 104 years, where all high-ranking positions, be it civil or military, were occupied by a single family which is called the Rana family and governance was limited to the collection of taxes and punishment for those not paying the tax. The clan governing governance system was overthrown in 1951 and a shared government between the main political party and the king emerged (Dhakal, 2007). But this resulted in power struggle between the palace and the political parties so the king took over all powers by restricting political activities and banning political parties in 1960. During this period, Nepal entered into the planned development with the preparation of the first five-year plan in 1956 (HMGN, 1956; NPC, n.d. a). From this time to 1990, Nepal exercised an absolute monarchy, but planned development continued. There was some limited role of M&E which was practised mostly in the supervision and monitoring of major development projects. It was largely monitoring of activities and inputs with limited monitoring of output and outcome (NPC, 2013). The people’s movement brought the king under constitution in 1991 and the government exercised the liberalised economic policy. The role of M&E was strengthened during this period. The second people’s movement established the federal republican structure of the governance and proportional representation of ethnic groups, and the one-third quota of women was made mandatory in political and social institutions at each level of governance. It has many clauses that provide directive principles, policies, and obligations to pay special attention to the development of vulnerable and marginalised people including women, ethnic, and religious minorities, Dalits, and differently-abled people. To ensure the implementation of the clauses, the Constitution has special provision for the monitoring and evaluation Clause 54 (NLC, 2020). This encouraged M&E to pay special attention to equity, rights, and gender equality. The M&E guidelines prepared thereafter and the Monitoring and Evaluation Management Bill is accordingly equity-focused and gender-responsive (NPC, 2018).

2 Institutional Structures and Processes (Political System) The history of monitoring and evaluation (M&E) in Nepal goes back to the year 2000 B.C. It was influenced largely by the judgement criteria used by then scholars on the Indian sub-continent. It has also been inspired by the teachings of Buddha since 600 B.C. (Dahal & Timsina,

238

A. KALUGAMPITIYA AND G. P. OJHA

2007). The modern-day M&E started in Nepal with the downfall of the Rana family regime in 1951. Nepal experienced three major political systems thereafter, such as the pre-democratic era with an absolute monarchy, the democratic era with a constitutional monarchy and the republican democratic era. Below, the M&E situation under these three eras is given. M&E During the Pre-Democratic Era (1951–1990) Nepal started planned development in 1956 and monitoring and evaluation (M&E) of projects has been an integral part since then (NPC, 2013). The role of M&E, however, was limited to the review of annual and periodic plans until 1975. Actual evaluation of the development projects/ programmes commenced with the start of the fifth Five Year Plan 1975/ 76. The National Planning Commission as well as sectoral ministries started conducting M&E. During the period of the fifth and sixth periodic plans, a weightage system of monitoring was brought into practice which was based largely on the expenses. In the seventh five-year plan (1985–90), M&E was conducted based on sectoral performance indicators (NPC, 2018). During this period, the monarchy was in absolute power and M&E had a limited role. M&E During the Democratic Monarchy Era (1991–2006) With the advent of the democratic system with the constitutional monarchy in 1990, government introduced the economic liberalisation policy. Together with this, the government of Nepal took initiatives to further strengthen the M&E system, wherein a high-level political commitment was ensured for performance and result-based evaluation (ADB, 2014). Evaluation was one of the 10 priority areas in the development plans. Also, a New M&E system was introduced to achieve expected outcomes through regular, effective, and efficient M&E mechanisms (ADB, 2014). An arrangement was made to have M&E present from central to local levels. The Local Self-Governance Act (LSGA) was introduced where M&E of local projects was made mandatory (HMGN, 1999). Similarly, there were some institutional arrangements and procedural reforms made, including the creation of action committees such as the National Development Action Committee (NDAC) under the chairmanship of the Prime Minister and the Ministerial Development Action Committee (MDAC) under the chairmanship of the respective

8

EVALUATION IN NEPAL

239

minister. These were formed to resolve problems brought up through M&E systems (NPC, 2013). M&E During the Federal Republican Democratic System (2007–Present) Further institutional strengthening of M&E systems has been taking place since 2007 to the present day. During this period, the constitution of the Federal Republic of Nepal was prepared which has given more impetus to the M&E by providing two clauses for it. During the constitution formation period from 2006 to 2015, some initiatives related to an M&E enabling environment were simultaneously taking place including, but not limited to, the issuance of result-based monitoring and evaluation (RBM&E) guidelines, development of policy and results matrices for programmes and policies, initiation of the impact evaluation of programmes and projects, introduction of the public expenditure tracking system, for analysing the status of public service delivery, and tracking the time spent reaching out to the people (ADB, 2014; NPC, 2018). Some pluralistic initiatives were carried out with the involvement of M&E stakeholders from multiple sectors. These included the formation of Voluntary Organisations for Professional Evaluation (VOPEs), the formation of Parliamentarian Forum for Evaluation, the conduct of two international evaluation conclaves and the organisation of the culmination of International Evaluation Year 2015, conduct of annual EvalNet meetings,1 M&E human resources development through national and international training, development of a separate M&E system for projects implemented by civil society organisations (CSOs), and studies in evaluation use.2 After the promulgation of the constitution which coincides with the Sustainable Development Goal era, there have been other important activities related to evaluation in Nepal. These include the formation of an Integrated National Evaluation Action Plan (2016–2020), the preparation of the National Evaluation Guidelines-2018, the development of M&E model guidelines for newly formed provinces-2018, and 1 EvalNet meetings were organised by NPC while inviting evaluators and its stakeholders to present papers and discuss evaluation issues based on which proceedings are prepared and published. 2 One of the authors was heavily involved in the evaluation promotion in Nepal and this is written from his experience.

240

A. KALUGAMPITIYA AND G. P. OJHA

the preparation of SDG-friendly indicators for M&E (NPC, 2017). Equally important is also the formation of the Monitoring and Evaluation Management bill which is prepared by the government with inputs from multi-stakeholders of evaluation including VOPEs. The bill is in the National Assembly waiting for parliamentary approval.3 2.1

Evolution of Evaluation in Nepal

Nepal’s evaluation system has been aligned with the contemporary global evaluation systems. There has been a good influence of the donor-funded development projects implemented by the government as well as nongovernment sectors. A significant share of the development budget in Nepal comes from the donor agencies either in the form of grant or loan projects. Following the 2005 Paris Declaration on Aid Effectiveness and the 2008 Accra Agenda for Action of 2008 the development projects have laid emphasis on the results measurement and the role of M&E has been prominent (OECD, 2005, 2008). These international instruments contributed significantly to strengthen M&E systems in Nepal. Before these, M&E emphasis was on monitoring; whereas these instruments influenced the M&E system to gradually shift to the evaluation.4 The evolution of evaluation in Nepal is given below in a diagram (Fig. 1). 2.2

Evaluation Regulations

Nepal has achieved several accomplishments regarding the institutionalisation of evaluation in the political structures. The Constitution of Nepal, various acts, regulations, and policies are integrated in M&E. The M&E bill has been drafted in 2016 to provide the legal base for strengthening the M&E system. Nepal is the only country in the Asia Pacific region and one of the few countries around the world to have evaluation embedded in the national constitution. Other examples are Morocco, Ivory Coast, and Switzerland. The following two articles: #54 and #293 in the constitution are directly related to evaluation: 3 One of the authors was heavily involved in the evaluation promotion in Nepal and this is written from his memory. 4 One of the authors was heavily involved in the evaluation promotion in Nepal and this is written from his memory.

8

EVALUATION IN NEPAL

1st to 4th plan (1956-75)

• • •

5th to 7th plan (1975–1990)

• Extension of M&E function to ministries and agencies • Introduction of output and outcome to measure achievements against periodic plan’s goals and targets • Introduction of project monitoring based on expenditure weightage system was initiated

8th and 9th plan (1992-2002)

• M&E was one of the plan’s 10 priorities • New M&E introduced from central to project levels • Institutional arrangements and procedural reforms initiated (Establishment of NDAC and MDAC, Creation of NPC M&E division and sectoral divisions, Creation of M&E section in each ministry, Capacity building)

10th plan (2002-2007)

241

Review of plan, evaluate progress, set priority Aggregate achievement of programme at the central level Economic analysis

• Mandatory use of logical framework approach in all central and externally funded development projects • Programme and project M&E at the implementation and impact levels were introduced • A poverty monitoring and analysis system, institutionalised at both district and national levels

11th and 12th plan (2007-2013)

• • • •

Result-based monitoring and evaluation (RBME) guidelines issued Policy and results matrices for programmes and policies developed NPC initiated the impact evaluations of programmes and projects A public expenditure tracking system introduced

13th plan (2013-2016)

• • • • •

M&E used as a tool for evidence-based policy making M&E policy guidelines (2013) issued M&E Draft Bill prepared Piloting of IT in M&E Three internal and 10 third-party evaluations carried out

14th plan (2016-2019)

• National M&E guidelines 2018 which is equity focused and gender responsive prepared • Prepared SDG friendly indicators for M&E • Provincial M&E model guidelines 2018 prepared

15th plan (2019-2024)

• • • • •

Institutionalisation of the result-based M&E system at federal, provincial, and local levels Plan for developing M&E human resources Collaboration with national and international M&E agencies including VOPES Promulgation of M&E Act Introduction of Project Performance Information System

Fig. 1 Evolution of the evaluation system in Nepal as per periodic plans (Source ADB (2014); NPC (2018); Author’s own illustration)

242

A. KALUGAMPITIYA AND G. P. OJHA

54. Provisions relating to monitoring: There shall be a committee, in accordance with law, in the Federal Parliament in order to monitor and evaluate whether the 42 directive principles, policies and obligations of the State set forth in this Part have been implemented progressively or not. 293. Monitoring of functioning of Constitutional Bodies: The chiefs and officials of the Constitutional Bodies must be accountable and C responsible to the Federal Parliament. The committees of the House of Representatives may monitor and evaluate the functioning, including reports, of the Constitutional Bodies, other than the National Human Rights Commission, and give necessary direction or advice.

Source: Article 54 and 293 of the Constitution of Nepal. Moreover, there are three other sub-clauses where evaluation is included in the constitution:

51. Policies of the State: The State shall pursue the following policies: (j) Policies relating to social justice and inclusion: (4) to evaluate economically the work and contribution such as maintenance of children and care of families, 253. Functions, duties and powers of National Women Commission: (1) The functions, duties and powers of the National Women Commission shall be as follows: (c) in order to have the women included in the mainstream of national development and ensure proportional participation in all organs of the State, to assess, monitor and evaluate the existing policies and programs, and make recommendation to the Government of Nepal for their effective implementation, 256. Functions, duties and powers of National Dalit Commission: (1) The functions, duties and powers of the National Dalit Commission shall be as follows: (e) in order to have the Dalit community included in the mainstream of national development and ensure proportional participation in all organs of the State, to assess, monitor and evaluate the existing policies and programs, and make recommendation to the Government of Nepal for their effective implementation,

8

EVALUATION IN NEPAL

243

In addition, there are seven other sub-clauses and two schedules that emphasise monitoring. In this way, Nepal has the most progressive constitution in terms of monitoring and evaluation. The provisions included regarding M&E are comprehensive enough for the country to promote practice and use of evaluations across the programmes. To translate the constitutional provision into action, Nepal drafted a National Monitoring and Evaluation Bill. The process was started in 2015 and it provides a comprehensive policy guidance to evaluation in the country in addition to the provisions given in the constitution. The process of drafting the Bill was led by the National Planning Commission with support from the National Parliamentarians Forum for Development Evaluation Policy in Nepal, Community of Evaluators—Nepal, United Nations agencies, and other stakeholders. It was a challenging process to take the draft bill to the parliament. The process took over four years and finally reached the Cabinet of Ministers which has approved the bill for submission to the parliament. Currently, the bill is at the National Assembly and once approved by them, it will be forwarded to the House of Representatives for approval. In Nepal, there are some sector-wise evaluation regulations and provisions. The Education Rule 2004, has M&E roles and responsibilities for officials and management committees at various levels. The rule states that the Director General will make arrangements for M&E of educational programmes implemented by the Education Department. Likewise, Local Self-Governance Act 1999 (LSGA) has provisions for evaluating all projects at the local level. The Forest Regulation 1995 has provision for evaluation of the possible impact on the environment, monitoring and evaluating the implementation of the Forest Work plan, for example. In addition, there are National Environment Assessment Guidelines 1993 which put emphasis on integration of environmental concerns into development programmes and projects (Bhatt & Khanal, 2009), especially the larger projects like hydropower, hotels, irrigation, road, agriculture, and forestry. The constitutional provisions for evaluation have provided an opportunity for the regulations above to refine them and other sectors to design equity-focused and gender-responsive M&E along the line of the constitutional motive and spirit.

244

A. KALUGAMPITIYA AND G. P. OJHA

2.3

Parliamentarian and National Audit Structures

Parliament Mechanisms There are 12 parliamentary committees. Closely related to evaluation is the Implementation, Monitoring, and Evaluation Committee on Directive Principles, Policies, and Obligations of the State. The committee is formed based on the Constitutional article 54. One of the scopes of the Committee is to monitor and evaluate progress on the implementation of directive principles, policies, and obligations of the state at the federal, provincial, and local levels. As per the website of the parliament of Nepal, the committee has a Monitoring and Evaluation Sub-Committee to monitor and evaluate progress. In the process, the sub-committee has provision of inviting evaluators and subject specialists, and/or related organisations to conduct M&E. The sub-committee parliamentary members, specialists/evaluators, and other stakeholders can discuss the progress on the implementation of the directive principles, policies, and obligations. The sub-committee has decided to prepare a roster of specialists/evaluators/organisations. The sub-committee organises a joint meeting with the NPC semi-annually. The team visit implementation sites, monitor activities, and provides reports to the sub-committee. The information thus gathered is used for decision-making (Parliament of Nepal, n.d.). The results of evaluations are to be discussed in the plenary session of the Parliament, depending on the nature of the project, which is of greater national interest, but this rarely happens in practice. National Parliamentarian’s Forum for Development Evaluation Policy (NPFDEP) Evaluation used to be considered as an area of specialists limiting evaluation only to the projects and programmes. Policies were seldom evaluated. Even if they were evaluated, politicians were rarely involved in the evaluation process. To bring change to this process, parliamentarians increasingly have been included as new stakeholders in the evaluation arena. In the second Evaluation Conclave organised jointly by the Community of Evaluators South Asia (COE-SA) and the Community of Evaluators Nepal (COE-Nepal), three parliamentarians, each from Bangladesh, Nepal, and Sri Lanka, created a panel and expressed their views on the importance of parliamentarians’ engagement in evaluation (COEN, 2013). Thereafter, the Global Parliamentarian Forum for Evaluation (GPFE) emerged in different countries. Nepal was the first country to

8

EVALUATION IN NEPAL

245

form a national evaluation forum—the National Parliamentarians’ Forum for Development Evaluation Policy. The forum included 15 members from five major political parties in the parliament of Nepal in 2013. The parliamentarians in the forum played an important role in convincing other parliamentarians to have an M&E agenda in the constitution of Nepal which was in the process of preparation. They also organised an evaluation training for parliamentarians in 2015. The forum was highly influential in organising the culmination of the International Evaluation Year 2015 in the legislature parliament of Nepal which declared EvalAgenda 2020 and different evaluation networks such as EVALSDGs, EvalGender + , EvalYouth, EvalIndigenous, and Global Parliamentarians Forum (EvalPartners, n.d.). The NPFDEP was also one of the active partners together with COE-Nepal, the government, and UN systems, to organise a workshop to prepare the Integrated National Evaluation Action Plan (2016–2020). It has also been involved in other several national and international evaluation events. The role of the Office of the Auditor General is to audit the accounts of all Federal and State Government Offices, in accordance with law, having regard to the regularity, economy, efficiency, effectiveness, and the propriety. It audits the wholly owned corporate bodies. It is also consulted while carrying out the audit of a corporate body of which the Government of Nepal or State Government owns more than fifty per cent of the shares or assets. The Office is centrally located with 647 staff who are deputed to carry out audits at federal, provincial, and local levels. Its role is limited to financial audit and programme audit is not carried out. It annually prepares a comprehensive financial audit report and submits it to the President. The Office publicises its reports on its website which is accessible to all (OAG, 2018). 2.4

Organisational Structure

To have a more systematic way of evaluation, M&E divisions and sections are established in the Office of the Prime Minister and Council of Ministers (OPMCM), the NPC, line ministries, and departments. The Planning and Monitoring Section of the OPMCM is assigned with a number of functions including monitoring and evaluation of annual, medium-term, and long-term policies of the government (NPC, 2013). Its other major tasks are:

246

A. KALUGAMPITIYA AND G. P. OJHA

• develop indicators of government policies and programmes, • develop and implement standard operating procedures, • performing policy auditing in coordination and collaboration with the NPC and concerned ministries and conducting impact studies of policy implementation, • submission of reports based on the monitoring and evaluation of annual and periodic plans, programmes, and projects, • monitor the implementation status of policies and programmes approved by the cabinet, • give priorities to government’s priority programmes and projects in monitoring and evaluation. It can be observed from the above that the PM’s office and cabinet are directly involved in monitoring, evaluation, and implementation of priority programmes with opportunity for evidence-based policy planning and implementation in the country. The National Planning Commission is the specialised and apex advisory body of the Government of Nepal for formulating a national vision, development policy, periodic plans, and sectoral policies for the overall development of Nepal. The NPC assesses resource needs, identifies sources of funding, and allocates a budget for socio-economic development. It serves as a central agency for monitoring and evaluating development policies, plans, and programmes (NPC, 2013). NPC is formulating development plans and policies of the country under the National Development Council’s (NDC) directive. It allocates resources for economic development and works as a central agency for monitoring and evaluating development plans, policies, and programmes. Moreover, it provides a platform for the exchange of ideas, discussions, and consultations pertaining to the economic development of the country. It also serves as an institution for analysing and finding solutions to the problems of civil society, non-governmental organisations, and the private sector in the country (NPC, 2013). NPC is currently functioning by the Executive Order of March 12, 2018, issued by the cabinet. As per the Executive Order, the following are the assigned roles by the Government of Nepal for the NPC:

8

EVALUATION IN NEPAL

247

• to formulate basic development policies and prepare periodic development plans, accordingly within the framework of a long-term development perspective, • to explore internal and external resources along as well as indigenous and foreign technology and to recommend suggestions to GoN to accelerate the pace of development, • to explore innovative approaches for sustainable development based on the economic situation of the country, • to formulate annual programmes and assist GoN in their implementation, • to advise GoN for the institutional development of monitoring and evaluation system, to monitor the implementation of programmes and projects according to plan target and outlay, and to carry out an impact evaluation of plans and projects, • to provide guidelines, advice, and suggestions to sectoral ministries, departments, other agencies, and local bodies and assist them in the formulation of plans and projects, • to provide guidelines to collect data and to carry out action-oriented research necessary for the evaluation of new policies and for the refinement of the planning process.

2.5

Evaluation Practice and Use

Evaluation Practice In practice, the National Planning Commission M&E Division manages the overall M&E system (including information systems and capacities) and undertakes evaluations of programmes and projects. As Dhakal observed (2014), as an apex planning and M&E agency, the National Planning Commission facilitates evaluations, engaging third parties hired through competitive processes. Each year, some programmes or projects are selected for evaluation using specific criteria received from line ministries. Steering committees, formed for each evaluation to facilitate the process, approve the terms of reference, select the evaluators, facilitate the evaluation processes, and maintain the quality of evaluations and reports (Dhakal, 2014). Sectoral divisions of the National Planning Commission Secretariat (NPCS) conduct field inspections of programmes and projects and undertake policy, programme, and project-related studies and evaluation reports of the sectoral ministry/agency.

248

A. KALUGAMPITIYA AND G. P. OJHA

Several government departments are tasked with M&E functions. The evaluations are conducted mostly of the projects that are related to the development sector including irrigation, road, agriculture, health, education, and forest. Almost all government ministries have M&E Divisions/ Sections which are responsible for preparing M&E plans, monitoring and evaluating projects implemented by the ministry and conducting regular evaluations as prescribed by the National Planning Commission. Those public institutions where there is no separate M&E division, have M&E focal points. It is observed that mid-term reviews/evaluations are common in the public sector but final evaluations are rare. Therefore, evaluations are rather process evaluations than impact/outcome evaluations. The practice of special evaluations like meta-evaluations in Nepal is very rare.5 Generally, government institutions do not have sufficient capacity to conduct evaluations. An illustration of the M&E subsystem in the ministry of Agriculture and Livestock Development (MOALD) is provided below. MOALD has seven Divisions including administration, agriculture development, food security and food technology, planning and development assistance coordination, livestock and poultry business promotion, poultry and fisheries, and livestock health. Each Division has several sections. For example, the planning and development assistance coordination division has a policy coordination section, a budget and programme section, a development assistance coordination section, a human resource, a gender development and inclusion section, a monitoring and evaluation section, and a statistics and analysis section. Each section has specific roles. The M&E section’s works include the preparation of programme M&E policy, measurement criteria and procedure, M&E of federal programmes and projects, preparation of periodic progress reviews and reports, carry out federal parliamentary works related to the ministry (MOALD, n.d.). The new draft M&E Management Bill has provisions to make evaluation mandatory for all programmes and projects (Government of Nepal, 2016). According to the national M&E guidelines, each programme/project should have a Theory of Change (ToC) or a Logical Framework Analysis (LFA). However, in practice, ToC or LFA is developed mainly for 5 One was commissioned by the UN Women Country Office Nepal. Kate Butcher in 2017 named “meta-analysis of six Women’s Peace and Security (WPS) programme evaluations”. https://gate.unwomen.org/Evaluation/Details?evaluationId=11212. Accessed on 14 February 2023.

8

EVALUATION IN NEPAL

249

donor-funded projects at the public level. All the major programmes have ToC/LFA but projects under programmes do not necessarily have ToC/ LFA, only some of the projects include this. Once the ToC/LFA are developed and available, they are used to develop annual work plans. In all projects funded by the national budget, monitoring and evaluation is compulsory. However, in practice, monitoring has become compulsory but not the evaluation. Evaluations are conducted mainly for mega projects and donor-funded projects. In a total of NPR 16.47 billion budget, the budget allocated for M&E was NPR 1.49 million, around 0.09%, for the fiscal year 2021/22 (MOF, 2021). The M&E budget for this year is slightly higher than the budget in the years 2020–21 which was 1.39 million. This indicates that the government has been allocating a budget for M&E tasks in its annual budget plan, though the size of the amount is not highly encouraging. The NPC website has listed 46 evaluations conducted by the government. Out of these, 36 are third-party evaluations and 10 are internal evaluations. These evaluations cover many development sectors including education, irrigation, agriculture, road, health, forestry, watershed management, cooperatives, forestry, women’s development, energy, employment, social security, poverty alleviation, tourism, communication, and drinking water. The external evaluation also covers the six impact evaluations in which three projects were related to poverty alleviation, two to agriculture, and one to irrigation (NPC, n.d. b). The information available suggests that there is no typical evaluation, but rather very diverse types of evaluations that are carried out in Nepal (See Table 1 below). The conducted evaluations are publicly available at www.npc.gov.np. The low number of evaluations was conducted due to the limited budget (around 0.09%) allocated for monitoring and evaluations through NPC and the limited capacities of evaluators. Preparing, implementing, and monitoring the Management Response Plans (MRPs) has been made mandatory to facilitate the implementation of evaluation recommendations. However, it has not been well practised (COE-Nepal, 2014). Both the demand and supply sides of evaluation have a limited capacity to facilitate, conduct, and use outcomes of rigorous evaluations. Sub-national entities have a limited capacity and outcome data that could impede the rolling out of the M&E of SDGs. Despite some efforts from the government side, there was inadequate networking and collaborative work among the evaluation actors. This was partly due to inadequate capacity within the government

250

A. KALUGAMPITIYA AND G. P. OJHA

Table 1 Evaluation conducted by third parties in Nepal as of 2020

Irrigation Road Education Agriculture Health Forest Institutional development Women Watershed Poverty Cooperation Childhood development Social security Employment Tourism MF Depressed community Power

8 4 4 4 3 3 2 2 2 2 2 2 2 2 1 1 1 1

Source NPC (n.d. b). See NPC website: https://npc.gov.np/en/ category/third_party_evaluation_reports. Accessed on 14 February 2023

systems, lack of clear scope of the stakeholders on promoting evaluation, and inadequate collaborative working. The Evaluation Practice at Decentralised Level The institutional arrangement of M&E at a local level was established with the introduction of the Local Self-Governance Rules in 1999 which has the provision of having a Supervision and Monitoring Committee at local level. With the introduction of the federal system in 2015, Nepal is exercising a three-tier political system with each tier having rights to form acts, rules, policies, guidelines and plans. Recently, some provincial and local level M&E guidelines were introduced. One such guideline is prepared by the Gandaki Provincial Government. The institutional arrangement as given in the guidelines include the Problem Action Committees at provincial and local level as well as M&E Committees at provincial, district, municipality, and ward level. The guidelines give special importance to SDGs and have made provision for an SDG M&E Committee at a provincial and municipal (local) level (GOGP, 2021). The guidelines encourage to have evaluation before the start of the project, during the project, and

8

EVALUATION IN NEPAL

251

after the project and suggest using mixed methods, blending qualitative and quantitative approaches. Internal vs. External Evaluation Practices Evaluations are conducted by the internal staff of government departments and by independent evaluators under contracts (Dhakal, 2015). For evaluations of the majority of donor-funded projects, external evaluators are the main source of experts. In most cases of evaluation of donor-funded projects, international evaluators are hired as team leaders and team members. This is due to the lack of expertise of local evaluators on the subject. Local external evaluators are taken as team members for certain evaluations depending on expertise and budgetary constraints. Local evaluators work as a team with international evaluators to match the knowledge of the context. As evaluations of donor-funded projects through the government are led by donors, the selection of evaluation teams is also driven by the donors. The staff at NPC and M&E units of the line ministries have general knowledge on evaluation. However, they still need further capacity building in conducting evaluations. With more capacity building, internal staff at public institutions will be able to conduct evaluations by themselves. This is also well reflected in the Provincial and Local level M&E guidelines (GOGP, 2021). 2.6

Use of Evaluations

It is necessary to ensure that development policies are based on evidence obtained through evaluation findings. It has been observed in most of the countries and also in Nepal that the use of evaluations for policy planning and implementation is not adequate. Evaluation reports tend to focus on the results of processes and delivery of outputs rather than on outcomes and impacts. Studies are generally quantitative, and proper triangulations are not usually done on the tools, their design, or data analysis. “Even when an evaluation was methodologically sound and captured many facts, if its recommendations were insufficiently based on rigorous analysis, its overall quality and use declined” (Dhakal, 2014, p. 139). The National M&E Guidelines are a mechanism to improve and systematise the M&E process in Nepal, making it “more scientific, practical and useful” (NPC 2013, p. i). However, its impact is yet to be ascertained. The National M&E Guidelines clearly indicate that monitoring and evaluation reports should be prepared at different intervals to measure the

252

A. KALUGAMPITIYA AND G. P. OJHA

implementation status of programmes and projects as well as to ensure that a project is completed and results achieved within the stipulated timeframe. It is indicated that periodic evaluations should be carried out focusing on relevance, effectiveness, efficiency, impact, and sustainability of programmes and projects. The reports should then be prepared and submitted to the NPC. The Guidelines include arrangements to implement appropriate findings and recommendations of evaluation reports. Also, the summary of recommendations obtained through the M&E reports is used by the management at relevant levels to improve policy design and decisionmaking processes. The National M&E Guidelines highlight the various situations to use reports and recommendations. The Guidelines also mention that M&E reports should be disseminated by the following procedures: • by posting on the website of the concerned ministry or agency, • by organising meetings, interaction programmes, or workshops to deliberate on monitoring and evaluation reports, and to inform all concerned stakeholders, • by submitting important accounts, data, information, or reports to the decision-making agencies such as the NPC, the OPMCM, and the MOF. All recommendations and findings of evaluations may not be implementable as they are. Some of them may require immediate action. Accordingly, to implement the recommendations of an evaluation, the concerned agency should prepare a management response and action plan and regularly keep track of the implementation. The above-mentioned guidelines show the intent of the government to utilise monitoring and evaluation results by institutions at various levels. But there is negligible evidence that there is an institutional mechanism to use evaluation findings. There are established practices to disseminate evaluation findings to policymakers from line ministries and relevant partners and to publish evaluation reports on websites (Dhakal, 2014). Evaluation reports are made public and are easily available on government websites. The National Planning Commission Guidelines determine provisions ranging from parliamentary committees to public hearing committees and also an

8

EVALUATION IN NEPAL

253

M&E Committee in order to ensure information is provided to stakeholders on programmes’ activities and outputs, and in order to promote transparency, social responsibility, and accountability (NPC, 2013). Annual programme and project budgets also discuss evaluation findings. However, they are not conducted systematically or with clearly defined purposes, such as for specific policy needs. “A lack of clear evaluation objectives makes it difficult to frame evaluation questions that will generate evidence in areas of interest to policymakers. There is lack of a clear and coherent evaluation policy that drives systematic selection, conduct and use of evaluations. There are also weak capacities to demand, facilitate and conduct impact evaluations, which results in low-quality studies of limited use” (Dhakal, 2014, p. 140). It is evident that the use of evaluation is very low due to various factors. Sometimes there are political compulsions to continue certain programmes despite the recommendations from the evaluations otherwise while sometimes the quality of evaluations is not adequate. Recommendations are biased towards the donors’ agenda and therefore are not based on the results of consultative processes in the true sense. It was reported that there is a need for professionalisation of evaluations. Thus, the utilisation part of the evaluation has been weak. In many cases, evaluations are carried out to fulfil the donors’ interest but they are hardly used for improvement, accountability, and learning. Involving stakeholders in overall evaluation planning and management is encouraged recently but there is still room to improve by providing a wider space to contribute and take the lead.

3 Societal Dissemination/ Acceptance (Social System) The social system herein covers the institutionalised use of evaluation by civil society and the demand for evaluation by civil society, public perception of evaluation of civil society, and the civil society’s demand for evaluation. In the subchapter on the institutionalised use of civil society, an overview of civil society, the general role of evaluation in civil society, and evaluation use in civil society is presented.

254

A. KALUGAMPITIYA AND G. P. OJHA

3.1

Institutionalised Use of Evaluations by Civil Society

Overview of Civil Societies in Nepal Nepal has a long history of civil society organisations (CSOs) dating back to around 2000 BC. These were committees in communities elected/ selected by people to make decisions about community well-being and settling the local disputes. In addition, there have been some indigenous organisations including, but not limited to, Gurukul—voluntary residential school, Guthi—trusts, and Parma—voluntary contribution and exchange of labour (Dahal & Timilsina, 2006). CSOs’ activities, however, were limited to the interest of the authoritarian regimes which ruled Nepal for several centuries. Even after the overthrow of the Rana regime in 1951, a conducive environment for the civil society to engage in economic, social, and political advocacy was almost restricted during the absolute monarchy period till 1990 (Dhakal, 2007). With the advent of democracy with a constitutional monarchy, the CSOs grew tremendously in number, mostly in the form of non-governmental organisations (NGOs). The democratic government developed and practised liberal policies that were favourable to the private sector and CSOs (British Council, 2019). An enabling environment for civil society growth is in place there, as the constitution of Nepal ensures the freedom of opinion and expression, the right to assemble peacefully without arms, to form unions and associations, and the freedom of mobility within the country, as well as the liberty to engage in a profession of their choice (Dipendra & Lorsuwannarat, n.d.). Currently, there are about 50,000 CSOs in Nepal (International centre for Not-for-Profit Law—ICNL, SWC, 2020) affiliated to the Social Welfare Council. There is a greater number of CSOs which are either registered with other entities or not registered anywhere, but functioning as Community Based Organisations (CBO) or Groups such as 19,000 forest users’ groups, 33,000 community groups, and 300,000 mothers’ clubs (British Council, 2019). 3.2

Evaluation Use in CSOs

The demand side agencies such as the government, UN bodies, NGOs/ INGOs, and donors are the users of evaluation. They use evaluation at varied levels. A study conducted in Nepal in 2014–2015 revealed that about 14% of the evaluations conducted were used in decision-making. This value differed largely by stakeholder type. The Government ranked

8

EVALUATION IN NEPAL

255

last in using evaluation in decision-making (4%), and INGOs at first place (67%), whereas, the UN systems used 47% and donors 48% (CoE-Nepal, 2015). This indicates that the use of evaluation in decision-making by political decision-makers is not encouraged. The study shows that the NGO/ INGO sector, which can be considered a part of CSO, does make use of two-thirds of the evaluations conducted. They largely use the recommendations of mid-term evaluations rather than final ones. One of the reasons for largely using the recommendations of the mid-term evaluation is that they are mostly of the nature that they are implemented within the project period. In the case of final evaluations, most of the recommendations are applicable after the project period where the interest of their use by CSOs is limited (COE, 2015). In our discussion with some renowned CSOs in Nepal, it has become apparent that the process evaluation recommendations are reviewed in the final evaluation and reported accordingly. The reporting of implementation of the mid-term evaluation recommendations in the final evaluation encourages the projects to use such recommendations. 3.3

Evaluation Database/repository and More

As a regulatory body, the SWC is also responsible for monitoring and evaluation of the projects implemented by CSOs. Within the SWC, there is a monitoring and evaluation department which monitors, supervises, evaluates, and prepares reports of CSO activities (SWC, 2015). Though the SWC has been in the evaluation business since 1992, it has prepared and is practising the Monitoring, Supervision and Evaluation Guidelines since 2015, which have been revised in 2020 (SWC 2015, 2020). This is meant for monitoring, supervising, and evaluating the activities and results of the national and international CSOs operating in Nepal. The SWC charges the projects for their M&E, facilitation, and supervision. The charge is fixed at the time of project approval based on the norms it has developed. The charge ranges from 0.1% for large projects to 0.5% for smaller projects of their total costs (SWC, 2020). They have also a common ToR for evaluation and team formation method. However, in the evaluation team, the guideline has not made evaluation specialists mandatory, whereas an economist is rather a must. This indicates that the M&E guideline of the SWC requires further revision, especially in the team formation aspect.

256

A. KALUGAMPITIYA AND G. P. OJHA

Though the SWC has a separate M&E department, other CSOs rarely have a separate M&E unit. In most of the cases, the M&E is included in their planning section. The well-established CSOs which also have good funding size and sources have a separate M&E Unit with a size of 1–3 staff, sometimes just a focal person for M&E, sometimes with a management information system (MIS) looking after the M&E; others do M&E activities through a Planning section. In the market, there are some independent firms which also carry out evaluation activities together with other activities. The CSOs that are specific to the development of evaluation are three voluntary organisations for professional evaluators (VOPEs) as of May 2021. The SWC has an extensive database of the projects implemented by the affiliated CSOs, it has not yet stored evaluation reports in its database (see https://swc.org.np). The conducted evaluations are supposed to be placed in the SWC’s website, but the SWC does not have any evaluation reports displayed on its website. Not only the SWC but CSOs also rarely publish evaluation reports on their website. However, the websites include monitoring data which remains in the management information systems as long as the project runs. This data is not stored beyond the project period, in most of the cases. One of the VOPEs—Community of Evaluators Nepal (COE-Nepal)— has created an online repository of evaluation for exploring, learning, and sharing evaluation options, approaches, methods, designs, tools, and techniques. In addition, it has a code of conduct for evaluators.6 The repository is open for public access. One of the objectives of the repository is to place evaluation reports conducted in Nepal in its dedicated file. However, it has not done anything yet to avoid the duplication of activities with the NPC, the government’s main organ for evaluation. The NPC also has a similar activity in the Integrated National Evaluation Action Plan 2016–20. The NPC has uploaded evaluations conducted by it to its website (see https://www.npc.gov.np), not the evaluation reports carried out by other stakeholders. After a dialogue with the NPC, the upload of evaluation reports either to the website of the NPC or in the COE-Nepal repository is likely to be realised.

6 See http://coe-nepal.org.np/resources.html, Newsletter 6. Accessed on 14 February 2023.

8

3.4

EVALUATION IN NEPAL

257

Public Perception and Discourse on Evaluation and Evaluation Results

Nepal has an abundant number of donor-funded projects. The projects are spread in different parts of the country except the far-flung areas, not yet covered by road network and beyond the influence of leadership. Those involved in the donor-funded projects have some sort of dialogue with the evaluation teams. It is a common practice in most of the evaluation cases to consult beneficiaries of the project at various levels. It is also included in the evaluation methodology that the preliminary findings are discussed at district and central levels in most of the cases. Representatives of the main stakeholder including CSOs are invited to the discussion forums. Despite the fact that they take part in the evaluation, the public cares little about the evaluation findings. The reasons being that actions on the evaluation findings that relate to them are rarely taken and thus the influence of evaluation on them is very low, if at all. Another reason is limited awareness about the importance of evaluation. A third point is that a large segment of the population thinks that evaluations are for project people for their upward accountability, and these have little to do with the public.7 The point is that very few CSOs share evaluation findings with the public. A study conducted in three South Asian countries including Nepal shows that sharing evaluation with the public is about 13% (COE-Nepal, 2015). They don’t even keep evaluation reports on their website. Therefore, limited access to evaluation findings is also an important factor that limits the public’s positive perception of evaluation findings. The same study shows that sharing action plans preparing to implement the evaluation recommendations is still low (6.25%). 3.5

Participation of Civil Society in Evaluations

CSOs rarely have a separate M&E unit but the M&E is included in their planning section in most of the cases, as informed by some CSOs. The well-established CSOs, which have also good funding size and sources, have a separate M&E unit, for example, the Nepal Red Cross Society; others do M&E activities through their Planning section. The SWC which monitors, supervises, and evaluates CSO activities has a separate M&E division with full-fledged staff (SWC, 2020), but not necessarily staff 7 This is from the personal experience of one of the authors of this case study.

258

A. KALUGAMPITIYA AND G. P. OJHA

that is well-versed in M&E. On the market, there are some independent firms that also carry out evaluation activities. They conduct not only the evaluations but also other research activities. In most of the cases, CSOs in Nepal have been implementing interventions funded by the external donors. They are project-based interventions. For project-based interventions funded by external donors, evaluation is included in the project document, especially since the Paris Declaration on Aid Effectiveness (2005) and the Accra Agenda for Action (2008). The projects with a duration of three years and longer have both midterm and final evaluations, whereas those with less than three years have only one final evaluation (SWC, 2020). Evaluations of donor-funded projects implemented by CSOs is a mandatory arrangement. Therefore, it is a common practice that the projects implemented by CSOs are evaluated. The Social Welfare Council (SWC) which is the regulatory body of CSOs, and international NGOs conducts evaluations of each project funded by donors. It conducts around 100 CSO evaluations annually. The donor-funded projects to be implemented by CSOs require approval from the SWC and projects not having the provision for evaluation are not approved indicating that evaluation is common in CSO projects. Almost all projects implemented by CSOs conduct final evaluations, whereas mid-term evaluations are carried out only for those projects that have a lifespan of three or more years. However, these are project-based evaluations. The institutionalisation of evaluation in the CSOs is in a primitive stage. Some CSOs, like the Nepal Red Cross Society (NRCS) and Nepal Participatory Action Network (NEPAN) have an evaluation framework institutionalised, as informed by them, but this is not common in every CSO. Rather than evaluation, these CSOs conduct internal annual reviews of the projects based on which the plan for the following year is revised.8 Many CSOs conduct internal reviews largely with their staff using informal techniques, such as staff workshops and drawing lessons from the projects implemented by them. Some of the CSOs also collect lessons from the external evaluations. They use lessons from both internal reviews and external evaluations for preparing annual plans and periodic strategies.

8 Interview with some CSOs in the context of this case study.

8

EVALUATION IN NEPAL

259

This is a common practice with some CSOs like the Nepal Red Cross Society and NEPAN, for example.9 In almost all cases, the evaluations are carried out largely because of requirements either by the SWC or by donors. Very few CSOs have commissioned evaluations on their own initiative. They are also in the form of review rather than evaluation, which they carry out before the preparation of a five-year strategy. However, many CSOs carry out lessons learned exercises with their own staff and/or board to feed into/include in the annual reports and strategic plans.10 Civil society participation is ensured in the M&E network meetings held by the NPC. They also participate in consultations when finalising guidelines, drafting policies, etc. State-centred evaluations include CSOs and citizens as respondents in data collection, and in some cases even the participation in evaluation reference groups. Citizens participate when they are beneficiaries of a particular intervention which is under evaluation. Other than that, citizens are not considered to participate as general citizens. 3.6

Demand for Evaluation Within Civil Society

Evaluation demand by CSOs is quite low in Nepal. The demand of evaluation depends on the capacity to use evaluation and ownership. Despite that, the efforts to engage stakeholders in evaluation are continuing, they give low value to it, and rarely use in decision-making. Discussions with some CSOs11 reveal that some CSOs are thinking of demanding evaluation for decision-making. Some of them take reference from other projects, and some are partially doing it at the annual planning and strategy development. Some do not practise this at all. Those who demand evaluation for decision-making do it because it is made mandatory by donors and required for strategy development. They also mentioned that it is useful to make strong proposals to convince donors of project funding. It also convinces stakeholders of its capacities. There are some hindering factors for evaluation demand as revealed by the CSOs.

9 Interview with some CSOs in the context of this case study. 10 Interview with some CSOs in the context of this case study. 11 There were five well-known CSOs interviewed in the context of this case study.

260

A. KALUGAMPITIYA AND G. P. OJHA

These include a lack of awareness about evaluation, importance of evaluation is not well understood, law of land does not require the demand for evaluation, and there is no robust system for evaluation demand. Other factors, as revealed by the interviewees working in CSOs, are resource constraints (both human and financial), irrelevant recommendations made by evaluation teams, frequent turnover of decision-makers, and an inadequate enabling environment. 3.7

Access to Open Data

CSOs rarely make evaluation reports public by putting them on their websites or any other media. Some share evaluation drafts with a limited number of concerned agencies as disclosed by CSO personnel. Some studies on evaluation are also made public again through websites. For example, COE-Nepal has placed a study on management response to evaluation on its website which is openly available to the public (see http:// coe-nepal.org.np/programs.html). The online repository, which has evaluation approaches, themes, and options in its content area, is free to all. The repository intends to store evaluation reports from different agencies in future. In general, a culture of making the evaluation a subject for public debate, sharing evaluation publicly, and results discussed publicly in media is yet to be developed.12

4 4.1

Professionalisation

Academic Study Courses and Training Practices

Some universities have M&E courses under certain faculties at the graduate level. There are, at least, three universities offering M&E under education and agricultural extension faculties. The School of Education, Kathmandu University, runs a 3-hour credit course on “Planning, Monitoring and Evaluation of Development Programmes/Projects”. It has a module on “monitoring and evaluation” under the MPhil in Education (Development Studies). The course is designed to impart theoretical and practical aspects of monitoring and evaluation of the development

12 One of the authors is actively involved in the activities of COE-Nepal and this is written on the basis of his experience.

8

EVALUATION IN NEPAL

261

projects and programmes. The content of the course includes conceptualising, planning, monitoring and evaluation, project design and development, result-based monitoring and evaluation, log frame approach, linking project/programme plan and developing a monitoring plan, evaluation, gathering, processing, and disseminating M&E information, report writing and dissemination, developing an M&E system. Under “evaluation”, it has types of evaluation, mid-term and final (ex-post evaluation), evaluative research design, project/programme evaluation methods (tools/techniques), outcome and impact evaluation, programme evaluation in education, and reporting and feedback in evaluation (Dahal et al., 2016). Similarly, Tribhuvan University has included a monitoring and evaluation unit in the Project Management course. The course aims at developing the skills of students in project monitoring and evaluation, linking project design and its results, and monitoring and evaluation of development projects in the Nepalese context. Monitoring and evaluation have three units in the course. The Faculty of Education has evaluation as a compulsory component of the Bachelor of Education focused on students’ performance evaluation and an optional specialisation in the Master of Education degree. Likewise, M&E is offered at the Agriculture and Forestry University’s Extension Education programme at its graduate level. It is a two-credit course. The contents of the course include an introduction to evaluation, concept of evaluation and monitoring, principles and functions of evaluation, monitoring and evaluation, evaluation and extension, evaluation and research, evaluation and measurement, types of evaluation, role of objectives in programme evaluation, basic terminology, levels and types, taxonomy of educational objectives, criteria for selecting appropriate objectives, methods of stating objectives, techniques, and procedure of quantitative and qualitative evaluation, different techniques and procedures, concept and processes used in CIPP (Context, Inputs, Process, and Product) model, and a comparative analysis of other extension programme planning evaluation models. Here, a practicum is offered in the design of a conceptual framework, develop objectives and indicators, conduct of a focus group, pre-test of a communication, processing of service statistics, use of participatory evaluation techniques, and writing an

262

A. KALUGAMPITIYA AND G. P. OJHA

evaluation plan.13 These courses, however, are general and do not cover contemporary issues related to the rights-based approach, utilisationfocused evaluation, strength-based evaluation, transformative evaluation, or equity and gender equality. 4.2

Non-Academic Training

Many development ministries have training centres which offer M&E courses with short durations and focus on monitoring. The non-gazetted officer course is more about filling up formats, ticking marking checklists, and preparing reports. The gazetted officer’s course deals with basic theoretical aspects such as concepts, importance, design, tools as well as analysis of monthly target achievement, identification of areas for improvement, and provision of feedback. The course differs by institution. For example, the Local Development Training Academy and Nepal Administrative Staff College (NASC), each has 3–4 hours of M&E course as a five-week training and the National Health Training Centre has 4– 5 hours M&E section in a six-month course. The course content of the NASC include conceptualisation and definitions of monitoring and evaluation, timing for M&E, and type and method of M&E. Regarding the evaluation, it deals with types of evaluation, level of evaluation, evaluation methods, use of evaluation, and M&E and follow up system (Uprety, 2008). However, these courses are offered on demand basis, and are therefore not regular. Some of the training courses are offered by private training centres like the Empowering Learning Development (ELD).14 It offers a five-day M&E course covering philosophy of participation, types of M&E, collection and analysis of information, and tools for M&E (including participatory rural appraisal, survey, logical framework approach, SWOT analysis, and group facilitation). It offers both live and online courses (ELD, n.d.). Occasionally, VOPEs offer evaluation training on a demand basis. The Community of Evaluators Nepal offered general evaluation courses to young and emerging evaluators. It also offered impact evaluation training to the evaluators.15 It conducted 13 Information obtained from an Assistant Professor conducting the M&E course at the Agriculture and Forestry University. 14 ELD is an international training institution covering M&E courses in many countries. 15 See http://coe-nepal.org.np/resources.html, Newsletter 7. Accessed on 14 February

2023.

8

EVALUATION IN NEPAL

263

online training on the updated evaluation criteria. It has also carried out a three-day online evaluation training on evaluation in a crisis-situation to multisectoral agencies, and a four-day online course for developing a monitoring and evaluation plan for the Hospital & Rehabilitation Centre for Disabled Children. Likewise, the Nepal Evaluation Society organises trainings and seminars to its members and others. One such training was on project monitoring and evaluation, a three-day training.16 The National Planning Commission (NPC) provides M&E training with the following approach: The NPC provides capacity building training first to its staff, followed by ministry staff of the respective M&E Divisions and district staff engaged in M&E activities. The approach also includes the training of more staff members to retain trained staff, linking training with on-the-job training, involving young M&E staff in evaluations together with senior M&E evaluators. 4.3

Journals and Communication Platforms

There is no academic journal of evaluation published in Nepal so far. It is also the case that journals from other disciplines have not regularly dealt with evaluation. However, there are online newsletters published by VOPEs, specifically by the COE-Nepal. COE-Nepal has issued several volumes of the newsletter but it is not a regular publication. In general, the Newsletter covers periodic events with respect to evaluation. The newsletter includes updates about the COE-Nepal activities, including training and updates from other countries or global events. In some cases, special issues covering articles from evaluators on contemporary evaluation issues, methodologies, approaches, and policies are entertained. The platform for sharing the evaluation related issues is email lists such as COE-Nepal members ([email protected]), Nepal Evaluation Society ([email protected]), and Society of Monitoring and Evaluation Nepal ([email protected]). The VOPEs communicate evaluation news with their members using their respective email. The list serves, usually, to disseminate training opportunities, work opportunities, and events organised by various organisations. Evaluation conferences, webinars, and events are another popular and well-used form of a knowledge sharing opportunity. Many such events are attended by speakers/

16 This comes from the experience of one of the authors of this case study.

264

A. KALUGAMPITIYA AND G. P. OJHA

presenters from Nepal where knowledge and experience are shared. Evaluators from Nepal also have the opportunity to attend and learn from these opportunities. Nepal already hosted regional evaluation conclaves (similar to evaluation conferences) in 2013 and 2015, where many international participants attended (COE-Nepal, 2013). 4.4

Professional Organisations

Nepal currently has three VOPEs at the time of preparation of this chapter. These are: the Society of Monitoring and Evaluation (SOME), the Nepal Evaluation Society (NES), and the Community of Evaluators Nepal (COE-Nepal). All the three VOPEs are registered with the concerned agency in the government of Nepal. All of them are members of the International Organisation for Cooperation in Evaluation (IOCE). While the NES is a member of the Asia Pacific Evaluation Association, COE-Nepal is a member of the Community of Evaluators South Asia. They are active in both the national and international arena. They have developed a code of conduct for evaluators and contributed significantly to the preparation of equity-focused and gender-responsive evaluation, providing inputs to the M&E Bill and in the preparation of the Integrated National Evaluation Action Plan (2016–2020). They have participated in EvalNet workshops organised by the NPC in 2013 and 2014.17 While working together, they have also distinct areas of engagement. One of the distinguishing features is that SOME and NES conduct project evaluation, whereas COE-Nepal does not. COE-Nepal, on the other hand, is engaged in organising evaluation conclaves and conferences, seminars and workshops, organising online courses, conducting research on evaluation, creating an enabling evaluation environment by motivating parliamentarians to create evaluative culture, publishing a newsletter, and making evaluation tools available through the online repository. It was COE-N that played a pivotal role in organising two international evaluation conclaves in 2013 and 2015 in partnership with the Community of Evaluation South Asia (COE SA). The COE-Nepal was also was significantly engaged in hosting the culmination of the International Evaluation Year 2015 in Nepal in collaboration with the International Organisation for Cooperation in Evaluation, EvalPartners, and COE SA, as well 17 One of the authors has been involved actively in promoting evaluative culture in Nepal and this is written based on his experience.

8

EVALUATION IN NEPAL

265

as other national partners like the government of Nepal and the Parliamentary Forum on Development Evaluation Policy, Nepal. Such activities have contributed to the professionalisation of evaluation.18 Having a greater number of VOPEs is good for evaluation professionalisation but equally challenging. Ego management is highly challenging as is working in a coordinated manner. Getting consensus takes time. 4.5

Compliance with Quality Standards and Obligations

There are no explicit guiding principles and standards for evaluators developed yet. But the commissioning agencies’ terms of reference have some competency criteria required for evaluators. To have a quality evaluation, they look for education and experience. In addition, most of the agencies ask evaluators to use the OECD/DAC criteria which are relevance, efficiency, effectiveness, impact, and sustainability depending on the nature of evaluation. The M&E Management Bill, which is yet to be approved, has also included coherence criteria in addition to the five above. The VOPEs have developed a code of conduct for evaluators (COE-Nepal, 2019).19 The main features of the code of conduct are maintaining integrity, confidentiality, honesty, quality assurance and respecting participants rights, putting the beneficiaries at the centre, maintaining confidentiality and ensuring a fair representation at all stages of evaluation, adopting to not harming principles, paying special attention to vulnerable people, discriminated groups, deprived genders, equity, and the human rights of beneficiaries, and those not being able to express their voice to be given special consideration in the project/programme, selecting participants to ensure that both women and men in relatively powerless, “hidden”, or otherwise excluded groups are represented, and avoiding conflict of interest (COE-Nepal, 2019). Professional organisations such as VOPEs encourage their members to follow the code of conduct. Some members do follow the code and encourage others to do so. However, applying these codes of conduct strictly by all evaluators is yet to be realised. Nepal has yet to work towards

18 One of the authors has been involved actively in promoting evaluative culture in Nepal and this is written based on his experience. 19 See http://coe-nepal.org.np/files/CoC%20final%20Sept%202019.pdf. Accessed on 14 February 2023.

266

A. KALUGAMPITIYA AND G. P. OJHA

regularly issuing an evaluation journal, developing a certification system, an arbitration board, and a professorship.20 4.6

Partnership

One of the peculiarities in the current evaluation development in Nepal is that there exists a culture of working together by different stakeholders of evaluation since 2013. Developing a national evaluation capacity enhancement plan using the recommendations of three meetings participated by different stakeholders in EvalNet meetings is the first step towards working together. The culture of working together was further enhanced by the hosting of the culmination of the International Year of Evaluation 2015 in the parliament of Nepal, where active participation of parliamentarians including the speaker of the parliament, the government including the prime minister, civil society organisations, private firms, academic institutions, and development partners was pronounced. This followed the organisation of the national workshop on EVALSDG and EvalAgenda in 2016 which was also participated by concerned high-ranking government officials, parliamentarians from different political parties, UN systems, private firms, CSOs, VOPEs, and academia. This workshop laid a foundation for developing an integrated national evaluation action plan. They identified main issues and based on the issues, developed the action plan which has roles for every type of stakeholder. They also developed a process of reviewing the action plan. The action plan is completely owned by the government. Accordingly, the stakeholders have worked towards implementing the planned activities. The stakeholders are in the process of developing another action plan 2021–2025 to work together. The government is making other stakeholders as partners in the evaluation development. It has taken suggestions from VOPEs and other stakeholders of evaluation in shaping the M&E Management Bill. Similarly, government is taking technical support of VOPEs in the internal evaluation of 10 selected projects. These are some of the examples of partnership and different stakeholders working together.

20 One of the authors has been involved actively in promoting evaluative culture in Nepal and this is written based on his experience.

8

5

EVALUATION IN NEPAL

267

Conclusions

Nepal has comprehensive regulations for evaluation through the constitutional provisions, being the only country in the Asia Pacific region having evaluation directly embedded in the national constitution. Nepal seems to be aiming to further strengthen the regulations for evaluation as the national evaluation bill is in the process of approval. The National Planning Commission, as the designated public institution for evaluation, is leading the evaluation function in the country. The NPC, through national evaluation guidelines, has introduced a system for all the public institutions to follow with clear instructions. The M&E units established at each public institution make the NPC’s role in evaluation easier and doable. After introducing the new decentralised admin system in the country, the NPC has developed new national guidelines for M&E to suit the new system. This shows NPC’s intention to improve M&E practices in changing situations. However, the capacity of evaluators to conduct quality evaluations still seems to be a challenge in Nepal. It is evident that the year 2015 and champions in evaluation in Nepal have been vital in the development of the evaluation field in the country. Evaluation practice at the public sector and the non-governmental sector is happening although that is not the expected situation. The public sector seems to focus more on monitoring compared to evaluation. However, NPC, ministries and other departments conduct evaluations as per the national guidelines. The NPC has been working at the federal level, but sub-national governments (provincial and municipal) have just initiated to institutionalise M&E systems. These structures need to be strengthened to ensure that evaluation is used in planning and management of development initiatives and the results are useful, help to assess the performance, impact, and effectiveness of their programmes, and learnings are used for decision-making processes. Evaluation has been partly decentralised across the line ministries to facilitate this work, but they are not well linked with the major national development agenda and in some cases, they contradict each other. All public institutions need annual fiscal evaluation plans and implemented properly. It is evident that non-governmental organisations are more active in conducting evaluations compared to the public sector, as evaluation is included in many donor-funded projects and it is a requirement of the SWC. However, the use of evaluation needs to be strengthened.

268

A. KALUGAMPITIYA AND G. P. OJHA

The role of parliamentarians is very critical and they seem positive to contribute towards this end but it was difficult to see the notable actions from their side. Still, the use of evaluation in the Parliament is not clearly evident and it needs to be thought of. It is true, for governmental institutions beyond the Parliament too, as the use of evaluation is still in need of attention. This may require more advocacy and showcasing of how the evaluations can actually help in better management of development objectives. Although Nepal has many essential elements for the evaluation eco-system, institutional and individual capacities on evaluation still face challenges. Changes in these two aspects will help to excel the evaluation practice and use.

References Asian Development Bank (ADB). (2014). Nepal: The national monitoring and evaluation system and the SREP investment plan. Mandaluyong City, Philippines: Asian Development Bank. Bhatt, R. P., & Khanal, S. N. (2009). Environmental impact assessment system in Nepal—An overview of policy, legal instruments and process. Journal of Science, Engineering and Technology, 5(2), 160–170. British Council. (2019). Civil society in federal Nepal: A landscape study. https://www.britishcouncil.org.np/sites/default/files/nepal_cso_landsape_ study_final_report.pdf. Accessed on 6 February 2023. Caltech Edu (n.d.). Geography. http://web.gps.caltech.edu/~avouac/nepal_ trip/geography.htm Accessed on 27 December 2021. Community of Evaluators Nepal (COE-Nepal). (2013). Evaluation conclave 2013: Evaluation for development. https://communityofevaluators.org/sites/ default/files/2020-01/Report-Evaluation%20Conclave-2013.pdf. Accessed on 27 December 2021. Community of Evaluators Nepal (COE-Nepal). (2015). Management response to evaluation. http://www.coe-nepal.org.np/files/MREexe.pdf. Accessed on 27 December 2021. Community of Evaluators Nepal (COE-Nepal). (2019). Code of conducts for evaluators. http://coe-nepal.org.np/files/CoC%20final%20Sept%202019. pdf. Accessed on 27 December 2021. Dahal, B. P., Uprety, U., & Gautam, P. (2016). Planning, monitoring and evaluation (PME) of development programs/projects. Kathmandu University. Dahal, D. R. (2014). Social composition of the population: Caste/ethnicity and religion in Nepal. In CBS, Population monograph of Nepal, Volume II. Central Bureau of Statistics (CBS).

8

EVALUATION IN NEPAL

269

Dahal, D. R., & Timsina, T. P. (2006). Civil society in Nepal: Searching for a viable role. Civicus Civil Society Index Report for Nepal. https://www.civ icus.org/media/CSI_Nepal_Country_Report.pdf. Accessed on 27 December 2021. Department of Hydrology and Meteorology (DHM). (2015). Study of climate and climatic variation in Nepal. Kathmandu: DHM, Ministry of Science, Technology and Environment. https://www.dhm.gov.np/uploads/climatic/ 47171194Climate%20and%20Climatic%20variability%20of%20Nepal-2015. pdf. Accessed on 27 December 2021. Dhakal, P. (2007). A study on the evolution of governance in Nepal. Proceedings of Unfolding Futures: Nepalese Economy, Society, and Politics. http://cffn.ca/ 2008/01/a-study-of-the-evolution-of-governance-in-nepal/. Accessed on 27 December 2021. Dhakal, T. (2014). Nepal—Institutionalization and use of evaluations in the public sector in Nepal. Proceedings from the Third International Conference on National Evaluation Capacities, UNDP IEO, New York, June 2014. Dhakal, T. (2015). NEC conference 2015 Blending evaluation principles with development practices to change people’s lives. UNDP IEO. Dipendra, K. C., & Lorsuwannarat, T. (2019). Mapping the Nepali NGO Sector: Understanding the evolutionary patterns. Research Nepal Journal of Development Studies, 2(2), 72–89. https://www.researchgate.net/publication/341 921209_. Accessed on 27 December 2021. ELD (n.d.). Getting development results. https://www.eldtraining.com/. Accessed on 6 February 2023. EvalPartners (n.d.). Global Evaluation Agenda 2020 Review Survey. https:// www.evalpartners.org/global-evaluation-agenda. Accessed on 27 December 2021. Government of Gandaki Province (GOGP). (2021). Province and local level monitoring and evaluation guidelines. Province Government, Policy and Planning Commission, Gandaki. His Majesty’s government of Nepal (HMGN). (1995). Forest regulation, 2051 (1995). Kathmandu: GoN. http://www.forestaction.org/app/webroot/js/ tinymce/editor/plugins/filemanager/files/Forest_Regulation_1995%20_2_. pdf. Accessed on 27 December 2021. His majesty’s government of Nepal (HMGN). (1999). Local self-governance act 1999. Kathmandu: GoN. https://www.lawcommission.gov.np/en/wp-con tent/uploads/2018/10/local-self-governance-act-2055-1999.pdf. Accessed on 27 December 2021. His majesty’s government of Nepal (HMGN). (2004). Education (Eighth Amendment) Ordinance 2004. Kathmandu: GoN. ICNL (n.d.). Civic monitor Nepal. https://www.icnl.org/resources/civic-fre edom-monitor/nepal. Accessed on 27 December 2021b.

270

A. KALUGAMPITIYA AND G. P. OJHA

Ministry of Agriculture and Livestock Development (MoALD). (n.d.). Planning and development cooperation coordination division. https://moald.gov.np/ department/planning-and-development-cooperation-coordination-division/. Accessed on 14 February 2023. Ministry of Finance (MOF). (2021). Public announcement of income-expenditure details of fiscal year 2021/22. Government of Nepal, Ministry of finance, Kathmandu, Nepal. National Planning Commission of Nepal (NPC). (2010). Results based monitoring and evaluation guidelines. Nepal. National Planning Commission of Nepal (NPC). (2013). National monitoring and evaluation guidelines. Nepal. National Planning Commission of Nepal (NPC). (2015). Sustainable development Goals 2016–2020: National [Preliminary] Report. Government of Nepal, National Planning Commission. Kathmandu, Nepal. National Planning Commission of Nepal (NPC). (2016). Integrated National Evaluation Action Plan 2016–2020. National Planning Commission, Nepal. https://theredddesk.org/countries/actors/national-planning-commis sion-nepal. Accessed on 27 December 2021. National Planning Commission, Nepal (NPC). (2017). Nepal: Sustainable development goals. Status and roadmap: 2016–2030. National Planning Commission of Nepal (NPC). (2018). National monitoring and evaluation guidelines (In Nepali language). National Planning Commission, Government of Nepal. National Planning Commission of Nepal (NPC). (2020). The fifteenth plan (Fiscal Year 2019/20 – 2023/24). Kathmandu, Nepal: National Planning Commission, Government of Nepal. National Planning Commission of Nepal (NPC). (2023). Introduction & history. https://www.npc.gov.np/en/page/introduction_history. Accessed on 6 February 2023. National Planning Commission of Nepal (NPC). (n.d.) Draft Monitoring and Evaluation Management Bill 2019. National Planning Commission of Nepal (NPC) (n.d. a). First five-year plan of Nepal 1956. https://www.npc.gov.np/images/category/FirrstPlan_Eng.pdf. Accessed on 27 December 2021. National Planning Commission of Nepal (NPC). (n.d. b). Third party evaluation reports. https://npc.gov.np/en/category/third_party_evaluation_ reports. Accessed on 27 December 2021. Nepal Law Commission (NLC). (2020). The Constitution of Nepal 2015. Nepal Law Commission. Kathmandu, Nepal. OECD. (2005). Paris declaration on aid effectiveness. OECD Publishing, Paris, 2023. https://doi.org/10.1787/9789264098084-en.Lastaccessedon1 4February

8

EVALUATION IN NEPAL

271

OECD. (2008). Accra agenda for action. OECD Publishing, Paris, 2023,. https://doi.org/10.1787/9789264098107-en.Lastaccessedon14February Office of the Auditor General (OAG). (2018). Stakeholder engagement strategy (2019–2923). Kathmandu, Nepal: Office of the Auditor General. Ojha G. P., Nath, B., Kezang, & Tuladhar, R. (2015). Enhancing demand for and use of evaluation in development through management response to evaluation: experiences and learning from South Asia. http://www.coe-nepal.org. np/files/MREReport.pdf. Accessed on 6 February 2023. Parliament of Nepal (n.d.). Committees. https://hr.parliament.gov.np/np/ committees/State-Direction-Principle-Rules-and-Responsibility-Committee. Accessed on 27 December 2021. Social Welfare Council (SWC). (2015). Samajik Sangha Sansthako anugaman, suparivekshan ra mulyankan nirdeshika 2071. [Monitoring, supervision, and evaluation guidelines of civil society organisations 2015]. Kathmandu, Nepal: Social Welfare Council. Social Welfare Council (SWC). (2020). Monitoring, supervision and evaluation of civil society organizations. Kathmandu, Nepal: Social Welfare Council. Uprety, S. R. (2008). Fundamentals of monitoring and evaluation. Kathmandu, Nepal: Bina Uprety. World Bank (n.d.). Land area (sq. km) Nepal. https://data.worldbank.org/ind icator/AG.LND.TOTL.K2?locations=NP. Accessed on 27 December 2021. Worldometer (n.d.). World population/Nepal population. https://www.worldo meters.info/world-population/nepal-population/. Accessed on 27 December 2021 Yadava, Y. P. (2014). Language use in Nepal. In CBS, Population monograph of Nepal, Volume II. Kathmandu: Central Bureau of Statistics (CBS).

CHAPTER 9

Evaluation in Pakistan Amna Aaqil, Amna Mahmood, Anam Shoaib, and Sehr Jamil

1

General Country Overview

Pakistan, officially The Islamic Republic of Pakistan, is the fifth most populous country in the world, with a population of 226 million people (World Population Review, 2021). Located in South Asia, Pakistan was created as a Muslim-majority country in 1947 during the partition of the Indian subcontinent (Nations Online, 2021). To better understand the role of evaluation in Pakistan, it is important to establish the general context for the country and understand the country’s political and economic systems.

A. Aaqil (B) Marketing & Partnerships, CERP, Lahore, Pakistan e-mail: [email protected] A. Mahmood CERP, Lahore, Pakistan A. Shoaib · S. Jamil Learning Hub, CERP, Lahore, Pakistan e-mail: [email protected] S. Jamil e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_9

273

274

A. AAQIL ET AL.

Pakistan’s political system acts as a federal parliamentary republic in which the chief of state is the President and the head of government is the Prime Minister. Modelled after the Westminster system of government, Pakistan’s federal government is separated into three branches: executive, legislative, and judicial (please refer to Fig. A1 in Appendix A). Within the federal government, the Prime Minister is assisted by the Federal Cabinet which is composed of ministers who advise the Prime Minister (National Democratic Foundation, n.d.). This system governs over four main provinces, with each province enjoying considerable autonomy1 and is headed by Chief Ministers (Country Studies, n.d.). After the 18th Amendment was unanimously passed by the National Assembly in 2010, the role of the Provincial governments was enhanced in several ways as more power was devolved to them for services such as health, education, infrastructure, social welfare, population planning, etc. (Rana, 2020).2 In looking at the topic of evaluation in Pakistan, it is important to note the division of authority and labour within the country’s government structure (expert 1, personal communication, July 30, 2021). Within this institutional government framework, there is a complex decision-making process related to project approval and assessment. There are four units that are responsible for project planning, execution, and evaluation: policymakers, central planners, project formulators, and project evaluators (Khattak, 2014). These units are intended to work together, forming a feedback loop, as project formation is dependent on evaluation methods and evaluation data is crucial for policymakers in informing future decision making. Economically, Pakistan can be described as having a mixed system with both free trade market activity as well as some government intervention. The economy has been slowly growing over the past two decades with an annual per capita growth averaging at 2%. In comparison to other South Asian countries, Pakistan’s growth rate is less than half of the average. The current pandemic has led to a severe contraction in economic activity with the GDP growth estimated to have reduced by 1.5%. However, 1 Administratively, there also exist two autonomous territories, and one federal territory within Pakistan. 2 The relationship between Federal and Provincial governments is such that if a hydroelectric power station is to be constructed in a province, the Federal government is required to consult to the concerned Provincial governments before taking a final decision for its construction (Rana, 2020).

9

EVALUATION IN PAKISTAN

275

there have been signs of recovery with increased community mobility, inflows of remittance, improvement in investment and private consumption (World Bank, 2021). The Asian Development Bank (ADB) has projected Pakistan’s economy to grow by 4% in FY 22 (Ahmed, 2021). In discussions surrounding evaluation, measuring Pakistan’s economic growth and money allocation have historically been an important factor in motivating evaluation and monitoring.

2 Institutional Structures and Processes (Political System) 2.1

General History of Evaluation in Pakistan

Development became a focus when the National Planning Board was established in 1953 and was housed under the Economic Affairs Division. Later in 1957, an independent and permanent National Planning Board was established that reported directly to the Prime Minister and encouraged the improvement and expansion of evaluation to support effective planning and development. The National Planning Board was redesignated as Planning Commission in 1959, with Monitoring and Evaluation (M&E) assigned to it, when the Ministry of Economic Affairs was dissolved, and the Economics Affair Division came under the Ministry of Finance. However, the Chairman of the Planning Commission did not possess the power typically given to a head of the planning body and there was a lack of effective coordination between the Planning Commission and the economic ministries for planning and implementation. As a result, the Planning Commission was reconstituted in 1961 and the Prime Minister became the Chairman. Additionally, the Project Division in the President’s Secretariat was abolished, and the Planning Commission became responsible for effective coordination, planning, and implementation. In 1962, the M&E divisions were passed on to the provincial Planning and Development (P&D) departments (Ministry of Planning, Development and Reform, 2019). However, M&E slowly reverted to the Planning Commission in the 1970s due to a lack of trained professionals in the province. But the Implementation and Progress Division in the Planning Commission was too small to monitor all projects so only larger projects would be selected for monitoring (Ahmed & Bamberger, 1989). In addition to the significant changes made within the Planning Commission during 1960s and early 1970s, the Planning Commission

276

A. AAQIL ET AL.

had a significant number of officials who had pursued higher education (Master’s and Ph.D.s) from prominent foreign universities and also had 10–12 advisors from the Harvard Advisory Group stationed for 3– 5 years. During this time, although evaluation was understood from the lens of project planning and design, it was not being used for conducting impact evaluations in the way it is now commonly understood (expert 1, personal communication, July 30, 2021; expert 2, personal communication, August 5, 2021). Therefore, the government enlisted the help of these experts with the primary objective of using the state’s power in helping the country move forward along the path of development. The Advisors from the Harvard Group played a dual role: they provided knowledge of M&E best practices in other countries and gave technical support to the government. For example, in the 1960s, these Advisors helped provide theoretical expertise in designing the country’s first input– output tables and also participated in helping the Planning Commission use and calculate the tables for different development sectors in the country (Ikram, 2011). During 1983–1992, a number of improvements were made to the overall M&E system housed within the Planning Commission, for example, the Minister of Finance and Economic Affairs was given the Chairman position that included additional responsibilities such as identification, facilitation, and training besides M&E. However, in 1988, it became apparent that a lack of evaluation at the closure of a project posed a huge M&E gap. This led to the redesign of the Planning Commission Forms namely PC I to PC VI, and later, the implementation of the online Project Monitoring and Evaluation Systems 1 and 2 (PMES 1 and 2). These forms and systems were designed to help create solutions identified through M&E reports rather than making them merely fault-finding documents. Between 2005 and 2011, with the signing of the Paris Declaration on Aid Effectiveness, Pakistan geared for implementing Results Based M&E (RBM&E) frameworks to effectively show how aid money was being spent to achieve development goals and outcomes. During this time, specifically in 2006, the Planning Commission was tasked with including evaluation of development programmes as part of its mandate. The Federal government used the RBM&E frameworks to clearly identify quantifiable indicators, outcomes, etc. for its National Poverty Reduction Strategy Papers (expert 3, personal communication, August 10, 2021).

9

EVALUATION IN PAKISTAN

277

In 2013, the Planning Commission was restructured as a “think tank” for the government to effectively plan for economic and social development and in order for it to achieve that M&E became a major function of the Planning Commission. It also became responsible for evaluating the impact of globalisation, developing appropriate national responses as well as evaluating the impact of new technologies on development. Additionally, it came under the umbrella of the Ministry of Planning, Development and Reform (MoPDR) and is now required to meet with the country’s premier on a bi-annual basis to report the progress of economic policies and for future guidance (Ministry of Planning, Development and Reform, 2019). Over the last two decades, Pakistan has actively worked towards creating institutional structures and processes to help strengthen systems of evaluation in the development sector. The next section analyses the nature of the evaluation frameworks, and policies at the national level, their evolution over time, and co-existence of audit structures and independent evaluation units. It also looks at how these national-level policies have translated into and differentially impacted evaluation regulations both at the federal and provincial levels. National Policies, Frameworks, and Systems on Evaluation There is no comprehensive national M&E framework in Pakistan and the term “evaluation” is not included in the Constitution (National Assembly of Pakistan, 2018). However, there is ample theoretical knowledge pertaining to evaluations in Pakistan. According to a Global Mapping Report on the status of National Evaluation Policies (NEP) in 109 countries conducted in 2015, the NEP in Pakistan was not formalised, i.e., evaluation was conducted routinely but there was no written policy. However, at that time, NEP was being developed and it was overseen by advocacy groups among evaluators, stakeholders in government international organisations, etc. Unfortunately, the report concluded that due to the non-availability of qualified resources and proper guidelines of evaluation techniques, the Evaluation Unit of Project Wing of the MoPDR3 did not make the required contribution towards the country’s economy by improving effectiveness, efficiency and transparency in project execution (Rosenstein, 2015). 3 This has been renamed as Ministry of Planning Development and Special Initiatives after the 2018 General Elections in the country.

278

A. AAQIL ET AL.

Though there is no singular national evaluation policy, there do exist frameworks and guidelines set forth by the federal government that helps various departments assess programme evaluation. For example, the MoPDR published a manual for development projects in 2019 which provides detailed information on programme evaluation including: • Purpose of evaluation, for instance, providing validity of programme objectives, efficiency of the pace of progress of project, project effectiveness, and identification of reasons for the satisfactory and unsatisfactory accomplishment of results. • Types of evaluation such as pre-evaluation, ongoing evaluation, and post-evaluation. • Evaluation indicators which include physical achievement indicators, output or impact indicators, economic indicators, and social indicators. Sector-wise national guidelines on evaluation have also helped design the criteria for identifying evaluation indicators. We see that within the Education sector, the country’s National Curriculum Framework (NCF) provides guidelines for achieving the national vision, goals, and objectives of education and also aims to evolve effective principles, processes, policies, and strategies for curriculum development, implementation, and evaluation. Regarding evaluation, NCF states that course curriculum should be evaluated periodically with respect to its objectives, content, and methodology (Ministry of Federal Education & Professional Training, 2018). Another key sectoral example of an evaluation policy is that of the Ehsaas 4 programme, launched by the Poverty Alleviation and Social Safety Division (PASS) as an umbrella initiative of 134 policies and programmes and 34 agencies of the federal government and all federating units to address poverty and inequality by building a welfare state. The agencies are tasked to ensure that they have the M&E mechanism for each programme, with related metrics and the independent means of their 4 Ehsaas is an instrument through which the Government aims to build a welfare state and address the needs of the extreme poor, orphans, widows, the homeless, the disabled, the jobless, poor farmers, labourers, the sick who risk medical impoverishment, the undernourished, students from low-income backgrounds, and poor women and elderly citizens.

9

EVALUATION IN PAKISTAN

279

verification. Priority is also given to the implementation of research and process learnings for policy while also ensuring that the third-party impact evaluations are planned in an experimental/quasi-experimental design to gauge the impact of the organisations and their projects on hard social outcomes (Ehsaas Strategy, 2019). The M&E framework of Ehsaas aims to: (i) facilitate collection of quality data on results; (ii) assess programme performance frequently and systematically through evaluative reflection and learning for course correction; (iii) assist reporting on a common set of indicators using standardised M&E methodologies and tools; and (iv) manage and respond to risk, issues, and lessons in a timely and effective manner. There are three salient components of the Ehsaas M&E framework: logical, monitoring matrix, and evaluation and research matrix. The evaluation and research matrix component will: (i) support periodic assessment of the outcomes and outputs of ongoing or completed initiatives to determine the relevance, efficiency, effectiveness, impact, and sustainability of the intended or achieved results at key moments through formative and summative evaluations for course correction and reach desired objectives; (ii) establish efficiency, effectiveness, and value for money for each selected intervention to learn lessons and invest further based on lessons learned; (iii) enhance sustainability and measure impact of desired interventions; and (iv) generate information and analyses for learning for future as well as for strengthening accountability (Poverty Alleviation and Social Safety Division, n.d.). The Government has allocated Rs. 98 million in the fiscal year 2021–22 to establish the M&E unit of the Ehsaas Programme (Planning Commission, 2021a). In the absence of explicit national laws on evaluation, various departments have designed and developed their own systems to help streamline evaluation processes. For example, the Public Procurement Regulatory Authority’s (PPRA) primary aim is to build government’s capacity to develop a modern transparent and cost-effective public procurement system. It is also responsible for M&E of the overall performance of procuring agencies as well as for evaluating laws, rules, regulations, policies, and procedures related to goods and services. This Authority is currently in the process of developing an e-procurement system that would include Regulatory Monitoring and Evaluation System (Cabinet Division, 2021). Departments have also incorporated central data dashboards to record and analyse key M&E outcomes. One such case is that of the National

280

A. AAQIL ET AL.

Accountability Bureau (NAB) which uses an M&E IT System developed in 2015 to enhance the operational, monitoring, and reporting capabilities of the database. The system allows the organisation to maintain an accurate database of complaint, complainant and accused as well as the progress and final decision or end result of each case. The system generates real-time reports which are available to the executives for decision making (National Accountability Bureau, 2020). However, the M&E System is primarily designed to maintain an accurate database to monitor case-wise progress. It has not developed the capacity to evaluate the efficiency of the Bureau for different categories of cases. Similarly, as mentioned earlier, the Planning Commission has implemented Project Monitoring and Implementation Systems (PMES), a software that functions as an online data repository of all Public Sector Development Programmes (PSDP) projects and aims to strengthen the M&E capacity of the Projects Wing and other stakeholders at federal and provincial levels. The software includes a separate component on the evaluation module (Planning Commission, 2011). However, the Planning Commission has been directed by the incumbent to further strengthen the PMES for timely and evidence-based decision making on PSDP projects and to use the existing system to its full potential (Abbas, 2020). Evaluation Regulations at Federal Level Insights from M&E expert interviews and document analysis show that the willingness to conduct evaluation across different federal ministries is not homogeneous. Ministries are still largely inclined towards project monitoring and there has been limited focus on evaluation of programmes. For example, the Ministry of Human Rights houses a Development Wing, which is responsible for monitoring and evaluating implementation of projects related to human rights (Ministry of Human Rights, n.d.). However, while the ministry has been actively involved in implementing various legislations and campaigns to improve human rights to all citizens, which also includes the establishment of a Legal, Research, and Monitoring unit, there is no mention of evaluating the progress of the programmes (Ministry of Human Rights, 2020). Nevertheless, there are quite a few examples where emphasis on impact evaluation has grown over time. One such case is that of the Ministry of National Health Services, Regulations and Coordination that places special emphasis on M&E of programmes pertaining to health and encourages third-party evaluations (Ministry of National Health Services,

9

EVALUATION IN PAKISTAN

281

Regulation and Coordination, n.d.). Similarly, the Ministry of Federal Education and Professional Training focuses on the coordination of academic evaluations across Pakistan, including the quality of student exams, and has recently approached EdTech Hub, a global non-profit research organisation, to design a robust M&E framework to help them assess the impact of leveraging technology to support school education delivery. The framework is yet to be operationalised and implemented (Ministry of Federal Education and Professional Training, n.d.; Khalayleh et al., 2021). As mentioned earlier, the Planning Commission is responsible for undertaking research studies and state policy development iniMinistry of National Health Servicestiatives for the growth of Pakistan economy. The Planning Commission is also responsible for creating Medium Term Development Framework (MTDF) as well as managing PSDP (Planning Development, Reform and Special Initiatives), the main instrument available to the federal government to provide budgetary resources for those development projects and programmes that yield maximum benefits for the society in the shortest possible time (The Express Tribune, 2012). Projects funded by the federal government have to abide by the M&E framework outlined by the PSDP. Besides ensuring the timely execution of projects, promoting transparency and accountability, and generating evidence and learning at strategic and operational levels, the framework also aims to achieve the intended project results at output, outcome, and impact levels (Planning Commission, 2021). However, M&E frameworks alone have not translated into capacity building for bolstering evaluation systems. A recent analysis of the 2021–2022 PSDP showed that out of the PKR 900 billion budget, only PKR 750 million was allocated to strengthening M&E cells at the federal level (Planning Commission, 2021). That is merely 0.0008% of the total budget. A research review of PSDP conducted by Haque et al. (2020) stated that the M&E framework has never been reviewed, updated, or evaluated and as a result, the focus has always been to use public investment to build physical infrastructure. From a technical standpoint, the Planning Commission’s evaluation processes are designed well. They follow a detailed project management lifecycle which encompasses numerous inter-related activities along with planned documents required at each stage (please refer to Fig. 1).

282

A. AAQIL ET AL.

Fig. 1 Planning Commission Project Cycle

As seen in Fig. 1,5 evaluation is the final phase of the project management lifecycle which is a critical analysis of the factual achievements and results of a project or programme against the intended objectives, underlying assumptions, and strategy and resource commitment. Government officials are required to use PC V form to provide verification of input, output, outcome, and impact. Moreover, there is also a PC VI form which is intended to be used to conduct project or programme evaluation after a specified time period (such as 5 years). However, this form was not available online and has not been used by any government official (expert 2, personal communication, August 5, 2021). 5 Project Life Cycle Stages and Project Proformas, Manual for Development Projects, Planning Commission of Pakistan, https://www.pc.gov.pk/uploads/archives/Manual-Pro jects-2021.pdf. Accessed on 3 February 2023.

9

EVALUATION IN PAKISTAN

283

To assess the overall performance of a project or programme, the government uses the following evaluation indicators: (i) physical achievement indicators such as overall physical progress and overall cost utilisation and timely or delayed completion of a project or programme; (ii) output or impact indicators such as production of crops, livestock, number of manpower trained, etc.; (iii) economic indicators such as financial rate of rate, IRR, etc.; and (iv) social indicators such as health and education facilities, income distribution with equity, etc. The M&E system of the Planning Commission includes the following functions under evaluation (Planning Commission, 2021): i) formative mid-term evaluation of selected ongoing projects through mixmethod approach (as appropriate, cost-effective) to assess the relevance (justification), efficiency (physical & financial progress) and effectiveness (contribution or intended contribution towards outputs/outcomes) of the projects; ii) summative, project end-evaluations of selected projects which are completed and for which PC-IV’s (project completion reports) are received using non-experimental, mixed-method approaches (as appropriate, but cost-effective); and iii) impact evaluation using experimental approaches for selected, flagship projects, and where evaluation findings would lead to define future investments of the government.

This elaborate evaluation process set out by the Planning Commission, however, is seldom used effectively (Haque et al., 2020). M&E experts and practitioners seem to be unanimous in their opinion that these existing tools for planning, implementation, evaluation, and monitoring are generally completed through the use of ad hoc data collection which omits identification of the outcomes and impacts of projects and lacks proper analysis of the respective data (Khattak, 2014). PC forms, to date, are used more as bureaucratic, box-checking exercises rather than as tools to improve and rethink mechanisms and systems for robust evaluations (expert 4, personal communication, August 26, 2021; expert 2, personal communication, August 5, 2021). If these M&E systems are not revisited and improved to reflect yearly progress against predetermined outputs for each ministry, they will become static regulations that don’t account for dynamic changes in the development landscape in Pakistan. M&E budgetary allocations can also not be made appropriately if M&E systems fail to operate at their optimum level (Haque et al., 2020).

284

A. AAQIL ET AL.

The Prime Minister’s Inspection Commission (PMIC),6 recently investigated Planning Commission and expressed concerns over the lack of a proper M&E system within the Planning Commission. The Planning Commission has also been ordered to strengthen the PMES to provide real-time updates on the progress of the projects. The PMIC has also demanded the immediate revival of the Evaluation Section of the Planning Commission with an adequate number of professional manpower and conducting post-evaluation of at least 10% completed projects in a financial year (Kiani, 2020). The Planning Commission itself is currently concerned with the problem that no long-term evaluation is being carried out after completion since there is no interest on part of the executing stakeholders to turn in the PC V form for evaluation. One recommendation put forth to rectify this issue is to widen the scope of the Pakistan Bureau of Statistics (PBS) to conduct evaluations of large-scale government programmes and projects. Another recommendation is for third parties to conduct independent evaluations, paid for by the Planning Commission itself (Zaman, 2020). Since the existing evaluation system of the Planning Commission is not achieving the desired outcome as per government directives, in order to improve the evaluation transparency and efficiency, the Planning Commission has recently announced their willingness to explore the latter recommendation and on-board M&E firms who can conduct third-party verifications and external evaluations (Planning Commission, 2021). Nevertheless, the establishment of an M&E framework by the Planning Commission, increasing focus of the ministries on impact evaluation as well as the demand for evaluation by the Prime Minster signals a positive change by the federal government in realising the importance of conducting evaluations. Evaluation Regulations at Provincial Level The focus on evaluation regulations varies across each province in Pakistan. The planning bodies at the provincial level are responsible for planning, coordination, implementation, and monitoring of development projects and activities of the provincial governments. However, 6 The Prime Minister’s Inspection Commission (PMIC), a constitutionally formed body mandated to inspect and monitor the public-funded projects, conduct inquiries into malpractices and mismanagement and seeks the Prime Minister’s approval on its recommendations.

9

EVALUATION IN PAKISTAN

285

budgetary allocations and systems designed for these roles vary significantly across provinces. A province-wise overview of Evaluation Regulations is presented below: The Punjab Directorate General Monitoring & Evaluation consists of Economic, Technical, and other sectors, each of which is headed by a senior chief or chief, and M&E comes under the Economic section.7 The M&E Unit in Punjab conducts regular evaluation of projects and encourages third-party evaluations (DGME, n.d.). This is the first unit at the provincial level which was the first directorate to be established and also the first division to draft an M&E Policy which is described in three sections: M&E for development budget, current budget, and policies (expert 2, personal communication, August 5, 2021; expert 3, personal communication, August 24, 2021).

The Sindh P&D Board consists of three wings: M&E Cell, Research and Training Wing, and Bureau of Statistics (Planning and Development Department, n.d.). The M&E Cell8 includes monitoring the progress and evaluation of development schemes and writing their critical appraisal. They have recently launched an Annual Development Plan (ADP)9 Progress Monitoring Dashboard, an integrated solution that will provide enhanced reporting, analysis, and visualisation of development projects. One component of the integrated solution includes a mobile application used by the field officers in the M&E Cell to provide real-time updates on financial outlay, key issues faced, and project status.10 The P&D Department at Khyber Pakhtunkhwa is responsible for the implementation and monitoring of overall development plans in the province. The province recently published its amended ADP Policy (2019–2023) which includes devising a robust M&E mechanism and a

7 The Board also coordinates with the finance department regarding provincial Annual Development Programme and development schemes (Ministry of Planning, Development & Reform 2019). 8 http://mecsindh.gov.pk. Accessed on 24 January 2023. 9 Annual Development Plan (ADP) is announced each fiscal year by each province’s

respective Minister to outline its development plan, areas of focus for welfare, and budget breakdown for upcoming programmes and schemes. 10 The monitoring reports can be accessed by the relevant departments at any time (Government of Sindh, 2020).

286

A. AAQIL ET AL.

software for informed decision making, extending the planning and monitoring set-up to all districts of Khyber Pakhtunkhwa, and giving warnings to departments that ignore monitoring reports (Planning & Development Department, 2020). Lastly, the P&D Department in Balochistan plays a pivotal role in implementing, monitoring, and evaluating the development projects through the respective executing agencies and departments. The Department is also focusing on evaluation by planning to automate PSDP to improve the delivery of public investment management, M&E of the projects, transparency, and accountability as top priority (Government of Balochistan, 2021). Overall, a holistic analysis of the ADP for the fiscal year 2021–22 indicates that the Government of Punjab is planning to spend Rs. 686 million to restructure the Directorate General Monitoring & Development (Planning & Development Board, 2021). The Government of KPK is spending Rs. 506 million on strengthening M&E and capacity building, with Rs. 286 million being spent in the establishment of Directorate of M&E in FATA Secretariat (Planning & Development Department, 2021). For the fiscal year 2021–22, the Sindh government has allocated total Rs. 325 million, out of which Rs. 110 million was for establishment of regional offices of M&E Cell at Karachi and Larkana, Rs. 20 million on construction of regional offices of M&E Cell at Hyderabad and Sukkur, Rs. 75 million on strengthening and capacity building of regional offices of M&E Cell at Hyderabad and Sukkur, and Rs. 120 million to enhance capacity of M&E Cell (Planning & Development Department, 2021). Balochistan’s budget for the fiscal year 2021–22 does not include any budget allocation towards M&E, but in 2020, the government prepared the draft for M&E policy framework which is under discussion with the stakeholders (Finance Department, 2020). It is evident that across provinces there is a difference in the total budget allocated towards M&E, and within this, each province chooses to invest on special initiatives based on their changing development needs, political willingness, total provincial budget allocated towards development, etc. 2.2

Parliamentarian and National Audit Structures

While discussing evaluation policies, it is critical to understand the relationship between law makers and their agency to demand or commission

9

EVALUATION IN PAKISTAN

287

evaluations. According to the Public Finance Management Act of 2019, the constitution of Pakistan demands the federal government to bring financial discipline in public finances through improving efficiency in using public assets and linking resources with the goals. The focused key areas in this Act are performance and plan-based budgeting. This also implies the significance of evaluation in budgeting by the government (Government of Pakistan, 2020). Although Pakistan does not have an evaluation unit at the disposal of its Parliament,11 and neither does the Parliament commission evaluations, it still plays a vital role in ensuring accountability of the public exchequer through the Public Accounts Committee (PAC). The PAC provides oversight of the exchequer by examining the annual reports prepared by the Auditor General of Pakistan (AGP), relating to the accounts of the federation. The national audits carried out by the AGP’s office are aimed at ensuring accountability and transparent use of public resources, thus minimising the possibility of waste and fraud (Public Accounts Committee, 2021). The appointment of the Auditor General is carried out under the Constitution of Pakistan and the Auditor General acts in an independent capacity and provides an objective assessment on governance at both the federal and provincial levels (Auditor General of Pakistan, 2021). Due to the Fiscal Responsibility and Debt Limitation Act 2005, the National Assembly also demands the Federal Government to lay the fiscal policy statement in front of the Assembly by the end of January every year. Key macroeconomic indicators like total net revenue, total expenditures, total fiscal deficits, and total public debt are analysed. Federal government is also to explain how these fiscal indicators will match with the principles of sound fiscal and debt management. In the fiscal policy statements, Federal Government informs the National Assembly regarding fiscal measures like taxation, expenditure, subsidies and administered borrowing, strategic priorities for the financial year in a fiscal area, and shares updates on relevant information on macroeconomic indicators.

11 However, the Parliament, specifically the Senate, does rely on a system of committee

which serves as a political nerve ending to gather information on development projects. The Committee raises questions on issues of public importance. The Senate has a committee dedicated to “Planning Development and Special Initiatives”. Furthermore, Committee hearings are held to investigate implementation of programmes (Senate of Pakistan, n.d.).

288

A. AAQIL ET AL.

Moreover, policy decisions are also analysed that are made by the Federal Government according to the medium-term budgetary statement to meet targets for economic indicators in a fiscal year. Similarly, an evaluation is also done by the Federal Government to show if current policies are matching the principle of sound fiscal and debt management as well as the targets presented in the medium-term budgetary statement (Government of Pakistan, 2020). It was in the early 1980s that the scope of audits was expanded to include performance auditing alongside financial auditing.12 Currently, even though there is no universally agreed definition of performance, the AGP office focuses on three key aspects when carrying out performance auditing. These include economy,13 efficiency,14 and effectiveness.15 Performance auditing is done in four elements including inputs, outputs, outcomes, and impact. As auditing the impact is measured over a long period of time, it is mostly done after several years of the project completion. This would make data on inputs, outputs, and outcomes become irrelevant. Due to this, performance auditors rarely audit the impact of projects in real life. On the other hand, planning for performance audit for every financial year comes under the overall annual audit plan for the field audit office (AGP, 2012). While carrying out the audits, the auditors distinguish between outputs, outcomes, and impact16 but they rarely measure impact when carrying out performance audits, as it is measured over a long period (AGP, 2012). 12 The office of the Auditor General introduced performance auditing in 1981. As of today, there are no guidelines dedicated to evaluation. The office does not single out evaluation as the differences between auditing and evaluating processes come from the academic disciplines from which they evolved. Auditing is from accounting (Auditor General of Pakistan, 2012). 13 Economy refers to the “acquisition of resources at the lowest cost keeping in view the objectives of the organization. It implies that the resources should be acquired at the right cost, at right time, at the right place, in the right quantity and of right quality” (Auditor General of Pakistan, 2012). 14 Efficiency refers to “optimum utilization of resources keeping in view the objectives of the organization. It implies maximizing output from the given resources or minimizing input for the given outputs” (Auditor General of Pakistan, 2012). 15 Effectiveness refers to the extent to which the objectives that were established have been achieved. 16 Outputs are the results produced by the organisation in response to inputs, outcomes include the external effects produced by the organisation in the short to medium term, while impacts refer to long-term effects of the outcomes on the society.

9

EVALUATION IN PAKISTAN

289

A leading M&E expert (expert 1, personal communication, July 30, 2021), highlighted that it is critical to choose the right metrics when conducting audits and evaluations. Referring to the large-scale Punjab Public Procurement study done at the Center for Economic Research in Pakistan (CERP) by Khan, he mentioned that public procurement is approximately 18% of the country’s total GDP and it is therefore critical to understand how we define “good procurement”. When surveying procurement officers, it was becoming evident that for them procurement efficiency was important but ensuring compliance with all the rules was more important. This essentially meant that procurement officers felt they would not be held accountable for procurement inefficiency as long as they had completed all the required paperwork. Adherence to rules might be an easier outcome to measure but certainly not the right one when compared with economic efficiency. Governments need to shift focus away from a narrow definition of evaluation (i.e., shift away from process evaluation to impact evaluation) and rethink evaluation beyond a merely bureaucratic, box-checking exercise where there is hardly any creativity. 2.3

Organisational Structures

In the last few years, there has been commitment shown by the leadership of the country towards strengthening Pakistan’s M&E systems. In 2018, the Government of Pakistan designed a National Sustainable Development Goals (SDGs) framework to establish institutional mechanisms in line with the United Nations (UN) 2030 Agenda. Implementing the SDGs is not without challenges including lack of awareness, coordination, and duplication of efforts while implementing projects related to SDGs (Government of Pakistan, 2019). An analysis of the National SDGs framework document showed that the framework includes policy intervention for the targets/goals (National Initiative for Sustainable Development Goals, n.d.) but there is no evidence of policy evaluation. The country aims to prioritise and localise the SDGs through the establishment of an SDG Support Unit, which functions as a form of a joint project between MoPDR at the federal level and P&D Departments at the provincial levels. The Support Unit is also responsible for setting up M&E processes that support the implementation of the goals. The government has underscored that M&E will need to be transparent in order to evaluate the efficiency, effectiveness, relevance, sustainability, and impact of

290

A. AAQIL ET AL.

programmes. Therefore, the government is actively leading the design of a comprehensive M&E framework to strengthen the use of evidence while implementing the SDGs. Prime Minister Imran Khan called for a National Strategy on the Development Statistics to underscore the importance of timely, accurate, and reliable data which can be used for evidence-based design, planning, and evaluation of national policies (Ministry of Planning, Development and Special Initiatives, 2019). Dr. Shabnum Sarfraz, Member Social Sector & Devolution at the Planning Commission, highlighted the importance of keeping vulnerable populations at the centre of designing plans since they are the most adversely impacted at the time of crises, the current pandemic is a case in point. She emphasised in a recent interview that the Planning Commission is trying to implement an M&E framework for the National Action Plan (NAP), with specific outcome indicators that use data to track policy progress and impact. She felt that understanding the impact of policies can help the government make investments in the right places (Sarfraz & Ahmad, 2020). In the status quo, the Planning Commission is only responsible for appraising PC Vs which are the Project Evaluation Forms. These are meant to be submitted by the implementing Ministries of any development project on a yearly basis for five years to showcase midline and endline impact assessment of any given programme. However, in the country’s history up until now, there has not been a single case of a submitted PC V (expert 5, personal communication, April 29th , 2022). One of the country’s leading experts on Planning & evaluation shared that this absence of evaluation is reflective of the behaviour within the government and bureaucracy which is focused more on spending allocated funds for a programme rather than being accountable for the impact that programme might have had on outcomes of interest (expert 6, personal communication, August 3, 2021). Therefore, the incumbent government is now focusing on three strategic steps to help redesign the M&E function of the development expenditure: (i) to bolster institutional capacity of the M&E Wing at the Planning Commission; (ii) to employ technology to monitor projects to gather reliable and accurate data—for this purpose, the government has signed an agreement with the Pakistan Space and Upper Atmosphere Research Commission (SUPARCO) to use satellite-based tracking systems to monitor project progress; and (iii) focus on contracting specialised

9

EVALUATION IN PAKISTAN

291

M&E firms to conduct thorough impact evaluations of development sector projects (Umar, 2021). To carry out these strategies, it is important for any organisation to have a proper structure at the national, provisional, and regional levels. The following subchapter demonstrates how these strategic steps are incorporated into different government ministries such as Health and Education. 2.4

Evaluation Practice and Use

The blueprint of evaluation policies and frameworks has been employed for conducting large-scale process and impact evaluation of development projects by the government. As discussed earlier, the rigour and frequency of evaluation vary significantly across sectors. In April 2010,17 17 key federal ministries were devolved to the provinces. These ministries include education, health, food and agriculture, social welfare, local government and rural development, tourism, sports, etc. In practice, evaluation is most frequently carried out by two of these ministries: Health and Education. This section looks at some of the more prominent evaluations carried out in these sectors. In the health sector, the M&E system was embedded in the National Health Vision 2016–2025. An M&E framework and plan was created and linked with the Planning Commission for up-to-date reporting on the SDGs, provincial and regional health departments to ensure synergised strategy making with other stakeholders. The M&E plan lays out the specifics of the different facets of the health system, e.g., data acquisition and its usage, resource requirements and organisational infrastructure needs, analysis of capacities and establishing indicators to measure performance against given timelines. The plan, therefore, allows the government to take corrective action whenever and wherever it is required. This M&E Framework serves the purpose of using different M&E tools and methodologies to measure outcomes and impact in relation to set targets. Third-party validation is also used to verify the monitoring data collected and the monitoring activity is primarily executed by the provincial health systems strengthening units. Additionally, a national body (high-level interprovincial health and population council) has been tasked 17 When the 18th Amendment of the Constitution was unanimously passed by the National Assembly of Pakistan.

292

A. AAQIL ET AL.

with overseeing the implementation of the Health Vision and endorsing the reports for presentation to the parliament on an annual basis (Ministry of National Health Services, Regulations and Coordination, 2016). In the past two years, the world has experienced a new potent challenge in the shape of the Corona virus disease (COVID-19). To contain the virus, the Ministry of National Health Services, Regulations and Coordination came up with a NAP, that is, a policy document to ensure that all guiding principles for outbreak preparedness, containment, and mitigation are followed. Part of the main objectives includes capacity development and evolving organisational reforms including reorganisation of health security establishment at national and provincial/regional levels and focusing on dedicated committees with trained personnel to continuously monitor and evaluate the plan (Ministry of National Health Services, Regulations and Coordination, 2020). Within the education sector, the National Education Assessment System (NEAS) has been set up under the administrative control of the Ministry of Federal Education and Professional Training to “build assessment capacity at the school, district, province and federal levels to measure students learning outcomes and improve the quality and effectiveness of programme interventions”. The main objectives of the NEAS include informing policymakers, monitoring standards, identifying correlates of achievement, and direct teachers’ efforts and raise students’ achievements. The NEAS aims to provide information to policymakers regarding the extent to which geography and gender contribute to differentials in student performance, it allows for monitoring of standards to determine how well the curricula taught translate into knowledge and skill building, helps identify factors contributing to low student outcomes and understand how resource re-allocation can be carried out to achieve better results, and allows teachers to benefit from using assessment data to improve learning outcomes of students (Ministry of Federal Education and Professional Training, 2019, pp. 1–2). The Higher Education Commission (HEC) has also established a separate M&E Division that is responsible for monitoring the PSDP of universities and higher education institutions (HEIs). The M&E Division undertakes field monitoring of development projects, prepares detailed monitoring reports of all ongoing projects, identifies bottlenecks, and conducts follow-up on projects, reports on the physical progress of projects, and prepares utilisation reports for the Planning Commission, Ministry of Finance, AGPR, etc. (HEC, n.d. a, b).

9

EVALUATION IN PAKISTAN

293

There are a few sectors where evaluations are rarely conducted. In the labour rights protection and laws field, there is a lack of compliance with labour laws and regulations set by the government (Ijaz, 2019). However, the Ministry of Human Rights recently proposed a draft National Action Plan (NAP) on Business and Human Rights that includes developing indicators for evaluation as a priority area and proposes qualitative and quantitative indicators to be used for M&E of the outcomes’ vis-à-vis implementation of the NAP by the responsible institutions in the time frame provided. For the monitoring, evaluation, review, and update of the NAP, it is essential for the multi-stakeholder monitoring group to oversee and critically evaluate the implementation of the NAP. The multi-stakeholder monitoring group will conduct an impact assessment, through consultations with different sectors, to identify subsequent needs/priorities that develop after the implementation of the NAP. The monitoring group will also undertake a comprehensive gap-analysis, reviewing the progress of the NAP within five years of its implementation, and providing recommendations for revision/amendment of the NAP (Ministry of Human Rights, 2021). It is important to note that prominent M&E experts in the country believe that evaluations are typically carried out to fulfil requirements of donor-funded projects (such as World Bank, DfID, UNDP, etc.). Even in such cases, rather than viewing evaluation as a scientific method of measuring impact, it is used as a tool to garner additional donor funds by nitpicking project areas that worked well rather than understanding reasons for its weaknesses and failure (expert 7, personal communication, September 2, 2021; expert 8, personal communication, August 27, 2021). For donor-funded projects in the larger development sector which is run by not-for-profit organisations, think tanks, research organisations, multi-lateral international organisations, evaluation is typically carried out by firms that includes: (i) local policy and research think tanks that include CERP, IDEAS, PIDE; (ii) international development consultancies that include Acasus, OPM, AAN Associates, McKinsey, Adam Smith International, Mott McDonald, Abacus Consulting, KPMG, Ferguson; and (iii) multi-lateral organisations that include UNDP, World Bank IEG.; or (iv) free-lance consultants. However, there does not exist any quality assurance regulation set out by the government to ensure that all evaluations adhere to a baseline level of rigour and robustness in their evaluation methodology (expert 3, personal communication, August 24, 2021; expert 7, personal communication, September 2, 2021). This is

294

A. AAQIL ET AL.

problematic because rather than using M&E as a tool for accountability, it is often used for accounting. For example, for an education project, the Ministry of Finance would be concerned about the utilisation of funds, whereas P&D Department would want to know about physical progress by measuring the number of schools built, or the number of teachers hired, meanwhile, the end beneficiary (parent or student) would care more about learning outcomes which is an outcome the system is neither measuring nor valuing. An interviewed evaluation expert (expert 1, personal communication, July 30, 2021) refers to research conducted by academics Jishnu Das and Tahir Andrabi around the concept of accounting vs. accountability. In his opinion, Pakistan’s M&E culture is overwhelmingly focused on accounting and needs to hinge more on measuring accountability to assess how well the public sector is faring in improving the overall well-being of the people it is serving (expert 1, personal communication, July 30, 2021). In a nutshell, evaluation as a tool is used by donors and is mostly used in the education and health sector in Pakistan. While many experts believe that evaluations are not conducted voluntarily, there are also those who believe that the government does want to focus on evaluation, but they do not have the resources or time to conduct programme or policy evaluations. A recommendation put forward by an interviewee (expert 8, personal communication, August 27, 2021) is to promote joint evaluations whereby external parties will be provided evaluation details by the government while they will not only build the capacity of the government but will also strengthen the general culture of evaluation.

3 Societal Dissemination/ Acceptance (Social Systems) 3.1

Use of Evaluations by Civil Society18

In Pakistan, the legal framework governing NGOs/NPOs is complex (Khushi, 2017) and there are several laws under which they can register themselves depending on the scope and objectives of their work (Pakistan Centre for Philanthropy, 2019). For example, NPOs are required to apply 18 For the purpose of this section, the term CSO, NGO, and NPO can be considered as being used interchangeably in order to be consistent with the government documents on this matter, as there is prevalence of the words NGOs/NPOs rather than CSOs.

9

EVALUATION IN PAKISTAN

295

for licence from the Securities and Exchange Commission, local NGOs working in the social welfare field and receiving government funding have to be registered with The Ministry of Social Welfare and Special Education and Provincial Social Welfare Departments, (Khushi, 2017) local NGOs/NPOs receiving international funding must be registered with the Economic Affairs Division (EAD), and INGOs have to be registered with the Ministry of Interior. NGOs/NPOs have to fulfil various reporting requirements based on the government body they are registered with. These reporting requirements have evolved over time. In order to understand how the government’s priorities have developed regarding M&E of the work done by NGOs, it will be useful to look at EAD for perspective. The EAD recently (June 30, 2021) launched an NGO portal (Economic Affairs Division, n.d. d) and according to the new NGO Policy (Economic Affairs Division, 2021), among other documents, registered NGOs must provide an Annual Plan of Action on their projects, and submit a project completion report on the portal. The Annual Plan of Action asks for quarterly details on the project’s expected outputs, annual targets/deliverables, planned activities, and planned inputs (Economic Affairs Division, n.d. a). A draft MoU between the Government and an NGO uploaded on the same portal, goes a step further and states that “Renewal of this MoU shall be subject to verification of projects, annual plans of action, annual reports and audited accounts including third party evaluations and other requisite information relevant to this MoU submitted by the NGO/ NPO” (Economic Affairs Division, n.d. e). The specific mention of “evaluation” above is promising and it appears that the government is interested in valuing contribution by NGOs/ NPOs that have shown impact. It is important to highlight here that an older draft of the MoU (Economic Affairs Division, n.d. b) had no mention of evaluation. There have been concerns over the years within factions in the NGO sector that believe that the regulatory framework imposed on NGOs is stringent (Janju, 2021). However, the government press release at the time of the launch of the NGO portal stated that the previous NGO Policy launched in 2013 lacked clarity, eligibility conditions, oversight and reporting arrangements, etc., and that the new policy was designed to address these deficiencies (Economic Affairs Division, 2021). For example, previously, the approval procedure for registering/renewing an NGO would take 12 to 18 months, whereas now the target is to complete it within 60 days. Moreover, EAD plans to build the capacity

296

A. AAQIL ET AL.

of ministry staff and NGOs so they are able to work on the e-portal and fulfil their mandatory reporting obligations (Economics Affairs Division, n.d.). Provided that the NGO e-portal was recently launched, there is no clarity on whether the government will be making the evaluation reports submitted by the NGOs public, or whether the NGOs will be required to make them public (as they are required to do so in the case of annual reports and audited accounts). It will be interesting to see in the next few years as mandatory submissions become due whether evaluation reports will be made public, and if they are, what kind of discourse they will generate within the larger public. The Pakistan Centre for Philanthropy (PCP) is a registered NPO, with the vision to enhance collaboration between the government, civil society, and business to promote social development in the country (Pakistan Centre for Philanthropy, n.d.). The PCP runs a Certification Programme that involves the evaluation of a NPO on standardised parameters of internal governance, financial management, and programme delivery for a specific period. According to their website, theirs is the first initiative of its kind in South Asia. All submissions for acquiring certification are voluntary and carried out by the respective NPO. There are approximately 2,000 NPOs that have registered with PCP from various sectors including animal rights, community development, environment, disaster risk management, environment, education, etc. (Pakistan Centre for Philanthropy, n.d.). Beyond this voluntary initiative, evaluations conducted by CSOs are primarily driven by donor requirements, for example, the Canadian Aid Agency requires impact evaluations of the programmes launched in Pakistan (expert 9, personal communication, July 26, 2021). Some NGOs may be motivated to conduct evaluations for the purpose of raising funds (expert 10, personal communication, August 26, 2021). According to an estimate, around 100,000 to 150,000 NGOs operate in Pakistan (Shah, 2016). However, there are only around 600 NGOs that have applied for registration on the EAD NGO portal and these are required to provide evaluation reports to ensure compliance and continue working. There is not much evidence of CSOs conducting evaluations by themselves purely to understand the impact of their work. To date, there is not much evidence on whether CSOs undertake evaluations to understand their own impact or to base their decisions on evidence. Overall, monitoring and evaluation tools continue to be seen as solely tools to help refine and design policies in the public domain rather

9

EVALUATION IN PAKISTAN

297

than a means of organisational improvement and refinement. Coupled with this resistance to change, CSOs are also constrained in terms of resources and skills required to employ M&E tools to conduct evaluation of their own work. This restricts their capacity to effectively use data in a more meaningful and impact drive manner (Khan et al., 2003). As mentioned earlier, after the adoption of the 2020 Agenda for Sustainable Development by the UN General Assembly in 2015, the Government of Pakistan established dedicated SDG Support Units within the Planning Institutions at the Federal and Provincial levels and enlisted support from the federal and provincial government as well as private sectors, academia, and civil society. A Pakistan Development Alliance, consisting of 111 NGOs from 64 districts of Pakistan, was established to help strengthen the capacity of the CSOs/NGOs and share information, joint advocacy, and monitoring and accountability with regard to the implementation of the development goals (Awaz CDS, 2021). Given the examples cited above, it can be inferred that there does exist an appetite at the government and the NGO/NPO level for evaluations. Whether these requirements (regulatory and voluntary) end up generating quality evaluation knowledge and reports that are then eventually used for decision making is yet to be seen. However, with NGOs/ NPOs registering with these bodies, and with the capacity of relevant staff involved in developing these evaluation reports building overtime, the next few years will be critical in learning the extent to which evaluation becomes embedded in the NGO/NPO sector overall. 3.2

Public Perception and Public Discourse

In Pakistan, under the Right to Access to Information Act 2017, all federal government bodies are required to ensure improved public access to public records to “make the government accountable to its people, to improve participation of the people in public affairs, to minimize chances of corruption and inefficiency on the part of state-owned organizations”. Records to be made publicly available include “performance reports, audit reports, evaluation reports, inquiry or investigation reports” (National Assembly 2017, p. 5). There are several national programmes which have provided public access to their evaluation reports. One such example is that of the federal safety net program, the Benazir Income Support Programme (BISP), which has made the evaluation reports conducted by a third party, the Oxford Policy Management, available on

298

A. AAQIL ET AL.

its website (Cheema, I. et al., 2020). Evaluation reports for donor-funded programmes such as UNDP, Asian Development Bank, United Nations, Norwegian Refugee Council, etc. are also available online. Apart from providing access to evaluation reports, policymakers also engage with CSOs such as the Pakistan Institute of Legislative Development and Transparency (PILDAT), the Free and Fair Elections Network (FAFEN), the Center for Peace and Development Initiative (CPDI), the Sustainable Development Policy Institute (SDPI), etc. These CSOs provide input and feedback to the policies put forth by parliamentarians, to further strengthen transparency and accountability (Habib & Rafique, 2019). But there is no comprehensive research to demonstrate whether CSOs use any evaluation results for public debates or their discussions with policymakers in the Parliament. At the provincial level, the Punjab Directorate General M&E has full versions of almost 1,400 evaluation reports available on their website (expert 2, personal communication, August 5, 2021). Evaluation reports were not available on the relevant departments’ websites in the remaining provinces. However, this may not necessarily imply that they are not available on public request. As mentioned earlier, it is not clear currently whether the evaluation reports uploaded by CSOs on the NGO portal, launched by the Economic Affairs Division (EAD), would be made public, as per their requirement. However, there are some CSOs that publish their evaluation reports on their websites. Pakistan Poverty Alleviation Fund (PPAF) has a separate section of “Impact Evaluations” reports on their website (PPAF, n.d.), while Kashf Foundation has uploaded evaluation reports to a separate section “impact assessment” (Kashf Foundation, n.d.). The overall perception of the evaluation experts interviewed for this case study is that there is a lack of understanding within the public of what evaluation entails. A select group of professionals/intellectuals may understand evaluation, but the demand for evaluation by the public is not as pronounced as it should be, given public funds are involved. For example, discussions in the media regarding assessing the performance or delivery of a project or service developed for the public, primarily revolve around whether funds were spent well and who was the implementor. More interest is shown in covering issues where a project is behind scheduled timelines, or if it has been delayed, or if it has exceeded budget, because these attract greater viewership (expert 8, personal communication, August 27, 2021), as opposed to coverage on whether an

9

EVALUATION IN PAKISTAN

299

intervention where all funds may have been judiciously spent but may not have generated the desired impact. A quick search for the word “evaluation” in three of the widely read daily newspapers, Dawn, The News, and Daily Times, showed that the word “evaluation” has only been used approximately 70 times in Dawn newspaper since 2002, approximately 100 times in The News since 2006, and approximately 80 times in Daily Times since 2014. Though the numbers represent a cumulative yearly estimation, an analysis of the search results indicates that all three newspapers have started using the term “evaluation” relatively more frequently in more recent articles. For example, there are 13 articles related to evaluation in Daily Times that were published in 2021 as compared to one article on evaluation which was published in 2014. Overall, evidence does not indicate whether CSOs use findings from their own evaluation studies in discussions with policymakers in Parliament or for public debates. There appears to be a lack of understanding among the general public on what an evaluation entails. Public interest appears to lie in who implemented the project, the wider socioeconomic ramifications of the project and adherence to timelines. While interest has increased in the recent past as evidenced by media reporting, it is still in early stages and far from being institutionalised. 3.3

Participation of Civil Society in Evaluations

Civil Society Organisations such as Kashf Foundation, Aurat Foundation, PPAF, and Rural Support Programmes Network (RSPN) are actively involved in evaluations. For example, RSPN developed an M&E framework as part of the agreement with the European Union to ensure that Sindh Union Council and Community Economic Strengthening Support (SUCCESS) programme is delivered effectively, and the impact is measured and reported in a timely manner (Shahzad et al., 2018). There are instances where the CSOs collaborated with research think tanks to conduct impact evaluation of projects. For example, CERP has worked with several government departments to help them use research and evidence to design and refine their policy-making processes through rigourous impact evaluation methods. For example, for poverty alleviation, the Punjab Skills Development Fund (PSDF) collaborated with the CERP to conduct a rigourous impact evaluation of the Big Push for the Rural Economy (BPRE) scheme using a randomised control trial (RCT) methodology (Cheema, A. et al., 2020). Similarly, CERP engaged

300

A. AAQIL ET AL.

with the Government of Punjab to deploy a methodology for measuring the value for money of procured generic goods and developed a web portal named Punjab Online Procurement System (POPS). Building on the project, CERP researchers designed and evaluated policy interventions to improve the performance of procurement officers. The evaluation results were shared with the government stakeholders including the Chief Minister of Punjab to help redesign department policies. CERP has also collaborated with the Excise, Taxation and Narcotics Control Department (E&T) Punjab to improve the performance of tax collectors by designing and evaluating interventions of performance pay incentives schemes and merit-based transfers and postings. Within the public health sector, organisations such as CERP formed an integral part of the Chief Minister Punjab’s COVID-19 Working Group, a high-powered body mandated to provide advisory on public health and economic impacts of the COVID19 outbreak in Pakistan and the government’s key policy decisions. The project included assessing the health of the COVID-19 data collected by the government and building the government’s capacity to effectively gather and manage that data. Similarly, as mentioned earlier, OPM has conducted multiple impact evaluations of the BISP programme to provide an analysis of the impact of the BISP from the time it was initiated (Cheema, A. et al., 2020). OPM has worked for a variety of donors and development partners, including DFID, EU, the World Bank, UN agencies, and other national and sub-national governments on the evaluation of projects such as strengthening sub-national governance in Pakistan, providing support to expand the provision of primary education in Sindh, reforming public financial management processes in Pakistan, service delivery innovations for education and health, etc. (Oxford Policy Management, n.d.). Overall, participatory methods are an important feature in evaluations in Pakistan. There are frequent documented instances of CSOs partnering with research think tanks for impact evaluations. Multiple government departments have also partnered with think tanks like CERP to refine their processes through impact evaluations. The foremost barrier to public participation is the lack of understanding of what evaluation entails for the layperson and the interconnectedness of evaluation on the quality of policy and programme design within the development sector. Oftentimes, government departments are also change-resistant, which further curtails the role of direct public involvement in evaluations.

9

3.4

EVALUATION IN PAKISTAN

301

Demand for Evaluations by Civil Society Organisations

To promote citizen-centric and participatory governance, the Prime Minister’s Performance Delivery Unit (PMDU) was established in 2013. The citizens would make suggestions, register complaints and grievances, report violations of laws, and/or seek guidance while the Unit ensured that the input from the citizens was addressed fairly and efficiently through the concerned government organisations. More recently in 2018, the government launched the Pakistan’s Citizen Portal which is a government-owned mobile application (available on Android and iOS), housed at the PMDU, that provides a nation-wide window to connect citizens with government organisations at all levels for complaints/grievance redressal and suggestions to the government (PMDU, n.d.). The portal has more than 3 million registered users, has received more than 2.8 million complaints, and from these resolved 2.6 million. This means that almost 93% of all complaints/suggestions were resolved. According to PMDU’s twitter account, the resolved complaints/suggestions range from arranging for a female doctor in a small town requested by a female citizen (PMDU Pakistan, 2020), stopping unlawful blasting of mountains and sale of the dismembered rocks (PMDU Pakistan, 2020), providing free transportation to senior citizens and females in a slum area (PMDU Pakistan, 2021), and dismissing officials who mishandled complaints launched by an expat on transfer or property (Anjum, 2021). Information was not available online on the extent to which citizens were demanding evaluations specifically using this terminology. But the examples above demonstrate that some level of process evaluation of government service and delivery is taking place through the portal. Moreover, in July of this year, the Prime Minister announced that the performance report of government officials will be directly linked to their efficiency in resolving problems of the people (Radio Pakistan, 2021). Overall, the popularity of the Prime Minister’s Citizen Portal seems to indicate a public interest in accountability of government departments and an interest in seeing how public complaints are addressed and resolved. The portal, however, is restricted to individuals and therefore is only indicative of individual interest in evaluations. CSOs are now increasingly involved in evaluations, particularly for donor-funded projects, but there is little evidence to suggest that they

302

A. AAQIL ET AL.

drive the demand for evaluations. Public demand for evaluations and accountability also remains infrequent and is often voiced around election cycles.19

4 Systems of Professions (Professionalisation of Evaluation) 4.1

Academic Study Courses and Training Practices

In Pakistan, there are no dedicated or tailored degree programmes geared towards M&E as a specialised field. However, across almost 62 universities in the country, various public policy courses are offered at the undergraduate and graduate levels to equip students with the fundamental knowledge and tools about M&E systems. Examples of such courses comprise of “Development Planning & Implementation and Evaluation” at Habib University, “Social Impact Evaluation” at IBA, “Economic Appraisal and Evaluation” at Bahria University, “Policy Analysis & Delivery” at Lahore University of Management Sciences (LUMS) etc. (Higher Education Commission Pakistan, n.d.). These courses are a part of a bachelor’s degree such as Economics (Institute of Business Administration Karachi Economics Department, n.d.), Social Development and Policy (Habib University, n.d.). The introduction of these courses has also been corroborated by Maroof A. Syed, CERP CEO, who applauded these institutions for taking the initiative to teach evaluation in undergraduate and graduate programmes. Evaluation offered as a subject is also incorporated in the curriculum and training of other disciplines such as Medicine, Psychology, Public Policy, Public Administration, Business Management, Economics, etc. The Pakistan Institute of Development Economics (PIDE), an autonomous research organisation, offers an M.Phil in Public Policy and includes a course on “Evaluation and Monitoring of Policy Initiatives” (Pakistan Institute of Development Economics, 2021). 19 A major hindrance to the public demand for evaluations is the lack of information for ordinary citizens. In absence of demand for transparency and accountability from the citizens, which acts as a disincentive for government departments to make such activities a regular part of their operations cycle. As mentioned above, the infrequent demand for evaluations, especially around the performance of high-profile individuals leading departments, picks up the pace close to elections but fizzles out once the next government is in place.

9

EVALUATION IN PAKISTAN

303

According to experts interviewed for this case study, a reason for the low awareness about the courses offered in this field could be because M&E is seen as part of on-the-job training (expert 10, personal communication, August 26, 2021). Rehana Afzal, General Manager M&E and Research at Punjab Social Protection Society (PSPA) took courses from Pakistan Planning & Management Institute (PPMI), an institution that falls under the Planning Commission, and provides on-the-job training along with a complete course on M&E. She also observed that the earlier training offered by PPMI focused more on output evaluation while the current training focuses more on results-based M&E (expert 3, personal communication, August 24, 2021). To build capacity, the Project Wing within the Planning Commission also organises training on Project PMES software at the Survey Training Institute (Survey of Pakistan, 2021). M&E trainings offered by independent training institutes are a part of professional development courses and/or certifications which students can acquire to complement their academic degrees. These trainings typically focus on topics such as commonly practised M&E techniques, results-based evaluation frameworks, tools required to conduct needs assessment, and end-to-end project management. Such training courses are offered by various research and educational institutes in Pakistan. The Center for Business and Society (CBS) at LUMS provides a comprehensive “Monitoring & Evaluation” executive development programme that caters to managers and leaders from the public and nonprofit sectors. The course focuses on RBM&E, logic models, preparing a results chain, developing indicators, and learning quantitative data collection tools and processes (LUMS Centre for Business and Society, n.d.). The Centre also offers customised training and conducted an M&E course in 2019 for the middle and senior level officers of the Merged Areas Khyber Pakhtunkhwa (formerly FATA region) to impart knowledge on designing M&E frameworks, developing performance indicators, conducting evaluation studies, and reporting data on M&E (LUMS Centre for Business & Society, 2019). Not-for-profit organisations last decade, have played a pivotal role in underscoring the importance of impact evaluations and evidence-driven policy-making in both the public and private sectors. For example, CERP has forged partnerships with strategic government institutions to build capacity with a focus on informed decision making and M&E. This decade-long effort has helped CERP to build credibility and relationships with senior officials in the ministries which have paved the way to scale

304

A. AAQIL ET AL.

up the work and also created a demand for M&E training at all levels of federal and provincial ministries.20 CERP training activities include a five-day intensive workshop “Evidence Based Programme Design”, through which CERP has trained around 200 mid-senior tier management representatives from the government, development sector, not-for-profit organisations, and academic and multi-lateral donor organisations. This training focuses on developing a structured approach to problem-solving and aims to help determine mechanisms that can help ensure that development policies and programmes in Pakistan achieve maximum impact which is a sore point for many ministries in Pakistan (CERP, 2021c). Another training, “Building Capacity to Use Research Evidence” (BCURE), has resulted in over 4340 government officials and civil servants from early to senior level management being trained by instilling an appreciation of evidence and skills for its use. The BCURE programme, created in collaboration with Evidence for Policy Design (EPoD) at the Harvard Kennedy School, employs a modular training curriculum that equips policy decision-makers with practical skills and frameworks for effectively applying data and evidence in their work.21 In 2020, amidst the peak of the Covid pandemic, CERP launched an online “M&E Boot Camp” which focuses on RBM&E Systems, Quantitative and Qualitative Research for Evaluation, Project Management, and Negotiation. The success of this three-course Boot Camp highlighted the latent demand22 for rigourous M&E courses from almost all development sectors in the country (CERP, 2021d).

20 CERP, due to its global network, is able to engage with a world class faculty from a diverse set of institutions (including, but not limited to): Harvard University, Duke University, London School of Economics, Warwick University, Lahore University of Management Sciences (LUMS), Institute of Business Administration (IBA), Massachusetts Institute of Technology and others (CERP, 2021a). 21 The training programme features cutting-edge blended-learning pedagogy, combining online modules targeted towards civil servants and other policy decisionmakers—with class sessions and localised case-based exercises led by 40 Harvard trained local instructors. The curriculum consists of seven modules (1) Systematic Approaches to Policy Decisions, (2) Descriptive Evidence, (3) Impact Evaluations, (4) Cost–Benefit Analyses (5) Aggregating Evidence, (6) Commissioning Evidence, (7) Using Data Systems (CERP, 2021b). 22 In just 2020, this course trained 149 participants working in the government sector, development sector, research organisations, and private sector organisations.

9

EVALUATION IN PAKISTAN

305

A list of additional institutes and organisations which provide M&Erelated training and workshops is attached in Appendix A. Journals and Communication Platforms There are various publications that focus on evaluation and are generally disseminated either through a government website for federal and provincial level project/programme evaluations or journals. PIDE has published articles specifically pertaining to M&E such as the evaluation of project monitoring system in Pakistan, accountability in development organisations, M&E by civil society organisations, and possible reasons behind the failure of development projects in Pakistan (The Pakistan Development Review, n.d.). The Punjab Economic Research Institute (PERI, n.d. a) has undertaken 415 studies in the field of socioeconomic development, conducted benchmark and mid-term evaluations of development projects and programmes, and completed feasibility studies. The Pakistan Journal of Educational Research and Evaluation (PJERE) is an open-access peer-reviewed journal published by the University of Punjab. PJERE is published biannually and includes studies in the field of educational assessment, administration and management, teacher education, educational psychology, research in education, and programme assessment and evaluation (PJERE, n.d.). Lastly, the Pakistan Journal of Medical Sciences is published monthly and includes research on hospital readmission, bacterial infections, efficacy evaluation of therapies, evaluation of outcomes of medication therapy, and more recently the evaluation of adverse effects of COVID-19 vaccinations, etc. (Pakistan Journal of Medical Sciences, n.d.). Apart from publications, there are various communication platforms that include blogs and podcasts. One such example is the Pakistan’s Growth Story which is a collaboration of the Consortium for Development Policy Research (CDPR) and International Growth Center (IGC), with contributions from the Institute of Development and Economic Alternatives (IDEAS). The blog includes articles on a diverse range of issues such as agriculture, pollution, evaluation of policies pertaining to COVID-19 and education, disaster management, gender, impact, human rights, urbanisation, poverty, and so on (Pakistan’s Growth Story, n.d.). The CERP podcasts feature leading voices with their evaluative research insights on topics ranging from education to governance, to climate change, economy, public service delivery, poverty, policy or programme evaluation, and so on (CERP, 2021e).

306

A. AAQIL ET AL.

The quality of evaluation depends on the organisation and there is a lack of strict oversight on the rigour of evaluation to ensure causal inference. However, the general understanding is that we have started to build an appetite for evaluations (expert 7, personal communication, September 2, 2021). Professional Organisations On a broader level, there are few Voluntary Organisations for Professional Evaluation (VOPE) in the country. The “Pakistan Evaluation Network” (PEN), one of the first voluntary organisations in Pakistan, was created out of the need to ensure that technical assistance and aid money were used effectively to improve the lives of the people in the country. The main issues related to M&E in Pakistan were about conceptual clarity, institutional framework, capacity, and financial resources. PEN sought to address these issues by creating awareness, disseminating information, developing an adequate institutional framework, systems, tools, and techniques for M&E, networking with local and international organisations and encouraging talent for innovation, experimentation, and exchange of new ideas through collaboration (Williams & Shankar, 2008). However, the information on the organisation is only limited to 2015 and there is no recent news regarding the work done by the organisation. The Pakistan Evaluation Association (PEA), created under the larger umbrella of the AsiaPacific Evaluation Association (APEA), is the only active voluntary organisation with almost 110 members, which promotes a culture of evaluation in Pakistan across the public and private sectors. PEA launched the EvalYouth chapter in Pakistan to engage aspiring M&E specialists in Pakistan with a large international network to help support and promote young and emerging evaluators. EvalYouth organises webinars to highlight the challenges, achievements, and learnings of the budding M&E culture in Pakistan.23 The organisation also partnered with APEA and nine additional organisations to set up the Asia Pacific Virtual Winter School for Young and Emerging Evaluators. The purpose of this Winter School is to provide an opportunity for young and emerging evaluators in the Asian Pacific to become more competent, experienced,

23 Recently, EvalYouth Pakistan organised their first webinar on “Evaluation Landscape in Pakistan” which focused on the current structure of monitoring and evaluation at federal and provincial levels, the challenges faced by the practitioners, government officials, and other stakeholders and the efforts by the government and individual organisations in enabling the environment for evaluation.

9

EVALUATION IN PAKISTAN

307

and well-networked professionals who can contribute to building the evaluation capacity of their respective countries at national, regional, and international levels (Asia-Pacific Evaluation Association, 2020). Existence and Compliance to Standards and Quality Obligations In the status quo, Pakistan has no guiding principles for evaluators or a specific Evaluation Criteria which is to be adhered to by Evaluation experts of M&E teams in the public or private sector for development programmes (expert 7, personal communication, September 2, 2021). However, different sectors have made some inroads towards drafting standards of Evaluation for their workstreams. One such example is that of the Higher Education Commission in Pakistan. Commissions like the Higher Education Commission (HEC) of Pakistan have taken an important initiative of institutional recognition based on performance. For this purpose, Performance Evaluation Standards have been outlined as a primary step for higher education institutions (HEC, 2011). Eleven standards are defined that specify dimensions of institutional quality. To achieve recognition status for certification in higher education quality provision, all eleven standards are equally relevant to be met by the HEIs (HEC, 2011). The standards primarily assess the current state of affairs of institutions in terms of quality and effectiveness. When there is a significant variation in policies, resources, and/or practices of an institution from pre-set performance evaluation standards for the HEIs, a clear justification is required on the variation if the new change is consistent with institution’s mission and objectives. It also needs to be made clear that the new policy is effective enough in meeting HEC’s standards (HEC, 2011). For correct and informed decision making, the totality of fundamental elements and relevant information related to the institutions under evaluation are important for institutions as well as the evaluators. All aspects like fundamental elements, context statements, and evidence of fulfilment of standards should be used together for decision making. This is because an institution may not have enough evidence for one fundamental element; however, it demonstrates attaining a standard through alternative information and/or analysis (HEC, 2011). This is evidence to suggest that overall, within the government’s structure in Pakistan, the longer-term goal is to inculcate an appetite for Evaluation. This would be a slow and iterative process, but examples of the HEC demonstrate that the country’s institutions are moving in the right direction. There should be an increased focus on capacity building

308

A. AAQIL ET AL.

and human capital investment to create a more robust culture of evidencedriven evaluation (expert 7, personal communication, September 2, 2021).24

5

Conclusion

Pakistan’s evaluation culture is at a nascent stage. There exist gaps in policies, regulations, and capacity building for institutionalisation of M&E in the country. Some seasoned M&E experts in the country opine that in the absence of a nationalised evaluation system or explicit policies, Pakistan’s political hierarchy creates an environment where individual self-interest is more likely to determine a policymaker’s decisions, than empirical data and evaluation (Salman, 2018). Overall, it seems that while some systems of evaluations do exist, their function and overall impact are limited. Many in the government continue to use evaluation as a tool to highlight success rather than to use it as a scientific method to assess gaps and redesign policies for the future (expert 6, personal communication, August 3, 2021; M. Syed, September 2, 2021). Furthermore, the execution of evaluation at both the national and provincial levels is constrained due to a lack of trained and skilled M&E experts. Despite the limitations in the field of evaluation, there has been a definite shift towards cultivating a culture that paves the way for more robust M&E systems in the country. The existing government’s primary mandate is transparency and accountability, and this political willingness has been translated into receptiveness by the prominent actors within the government sector who are at the forefront of economic policy-making (expert 7, personal communication, September 2, 2021). Governments, both at the national and provincial level, have become more open towards engaging researchers, consultants, and think tanks to generate new avenues for in-depth discussion on the real value of conducting evaluation (expert 2, personal communication, August 5, 2021). Another major change in the perception of evaluation is driven by the adoption of SDGs by the government, which has underscored the significance of conducting evaluations—we see this change in the redesign of PC forms, the development of RBM&E frameworks, and the implementation of dashboards to improve the quality and documentation of evaluation within and across key government ministries. The presence of voluntary organisations that are providing platforms for senior and emerging evaluators gives hope that 24 In Pakistan, the evaluation market is not regulated by a certification system; nor is there an overarching professional arbitration board entrusted with evaluations.

9

EVALUATION IN PAKISTAN

309

the value of evidence-driven evaluation is getting recognised by various stakeholders. However, in order to truly engage the public, the discourse on evaluation needs to be in language that the common man can understand and debate in (expert 7, personal communication, September 2, 2021). We have seen this in the recent example of the implementation of the Single National Curriculum25 which has generated a debate from key stakeholders such as parents, teachers, and students, who want the government to re-assess the content and objectives of the curriculum (Salim, 2021). Pakistan has to continue to build on and make progress towards institutionalising evaluation. Additionally, it seems that at all levels— public and private—the appetite for M&E is being built as institutions move away from the traditional framework of process evaluation (or monitoring) towards a more RBM&E approach.

Appendix A See Fig. A1 and Table A1

Ministries Executive Government

The Prime Minister

Federal Cabinet Advisers

The President The Parliament

Senate

Governmental Structure of Pakistan National Assembly

Provincial Assemblies Federal Shariat Court

The Judicature

Supreme Court

Provincial and High Courts

City Courts

Mohtasib

Fig. A1 Governmental structure of Pakistan (Source Author’s own illustration. National democratic foundation [n.d.])

25 Single National Curriculum is a step towards creating one system of education for all, in terms of curriculum, medium of instruction, and a common platform of assessment, so that all children have a fair and equal opportunity to receive high-quality education.

• A four-month diploma on project management, but the diploma focuses more on project planning, executing, monitoring and control and there is limited focus on evaluation (PIM, n.d.) • A 2-day course on project monitoring, evaluation, and control to enable the participants to learn the tools and techniques of effective project M&E (PIM, n.d.) • Not only provides customised trainings in the area of M&E but is also in the process of developing long-term training and coaching programmes in the field of M&E (AAN Associates, n.d. a) • M&E content includes: logical framework approach, M&E framework development, impact evaluations, baselines assessments, process monitoring, output and outcome M&E, etc. (AAN Associates, n.d. b) • A 2-day online workshop on GIT applications for M&E and Sustainable Planning, attended by 50 staff members of the P&D department of Khyber Pakhtunkhwa • The course focused on using geospatial information technologies for M&E and working with Sentinel imagery to support sustainable planning activities (UNITAR, n.d.)

The Pakistan Institute of Management (PIM)

donors, UN agencies, public entities, and non-profit organisations.

26 An international development and communication consulting firm providing advisory and implementation services to bi/multilateral

The United Nations Institute for Training and Research (UNITAR)

AAN Associates26

Type of M&E training

Overview of (additional) institutes and organisations and their M&E capacity building portfolio

Name of organisation/institute

Table A1

310 A. AAQIL ET AL.

Punjab Economic Research Institute (PERI)

(continued)

• Training conducted on Monitoring, Evaluation, Accountability, and Learning (SPDI, n.d.) • Provides various trainings which includes Training of Trainers for community management skills, a workshop on M&E frameworks, Impact Evaluation Design, and Performance Review, M&E and socioeconomic baseline training • Organised a dialogue on the newly launched Sindh Union Council and Community Economic Strengthening Support (SUCCESS) Programme (RSPN, n.d.) • In 2017, the institute organised a seminar on “Social Policy Themes and Evaluation” but there haven’t been any recent seminars on the topic (PERI, n.d. b) • The institution also provided trainings in 2015 and 2016 on capacity development of the government officials to mainstream gender in the formulation, implementation, M&E of government policies and plans, and conducted a workshop on capacity building for P&D department and conducted a session on M&E of development projects (PERI, n.d. c) • PERI’s capacity building unit includes economic forums, conferences, seminars, workshops/trainings, and internship programmes

Sustainable Development for Policy Institute (SDPI)

The Rural Support Programmes Network (RSPN)

Type of M&E training

Name of organisation/institute

9 EVALUATION IN PAKISTAN

311

Type of M&E training • Two-week training on “Project Monitoring & Evaluation”. The course caters to the participants from all sectors of the economy such as education, NGOs, health, agriculture, water, power, transport, etc. The course focuses on (PIDE, n.d.): stages in the project cycle; project implementation and management; key terms, concepts and purposes of monitoring & evaluation; sources of data for M&E; performance indicators for M&E; project M&E procedures in Pakistan and the relationship of the project topics to the PC performance network Analysis/CPM/PERT and bar charts as implementation and management techniques

Pakistan Institute of Development Economics (PIDE)

(continued)

Name of organisation/institute

Table A1

312 A. AAQIL ET AL.

9

EVALUATION IN PAKISTAN

313

References AAN Associates. (n.d. a). Training and capacity building. https://aanassoci ates.com/services/training-and-capacity-building. Accessed on 26 September 2021a. AAN Associates. (n.d. b). Monitoring and evaluation. https://aanassociates. com/services/monitoring-and-evaluation. Accessed 26 September 2021b. Abbas, G. (2020a, July 10). PM directs Planning Commission to Improve Project Evaluation, monitoring system. Profit by Pakistan Today. https://pro fit.pakistantoday.com.pk/2020a/07/10/pm-directs-planning-commission-toimprove-project-evaluation-monitoring-system/. Accessed on 26 September 2021. ADB. (2009). Pakistan—Asian Development Bank. Asian Development Bank, Asian Development Bank, May 2009. https://www.adb.org/sites/default/ files/publication/28971/csb-pak.pdf. Accessed on 6 February 2023. Ahmed, V., & Bamberger, M. (1990). Monitoring and evaluating development projects: The South Asian experience. The World Bank. Ahmed, A. (2021, September 22). ADB projects Pakistan’s economy to grow by 4PC in FY22. Dawn. https://www.dawn.com/news/1647800/adbprojects-pakistans-economy-to-grow-by-4pc-in-fy22#:~:text=The%20Asian% 20Development%20Bank%20(ADB,of%20the%20Covid%2D19%20pandemic. Accessed on 26 September 2021. Anjum, D. (2021, May 31). PM Imran dismisses officials after complaint on Citizen’s Portal. ProPakistani. https://propakistani.pk/2021/05/31/ pm-imran-takes-action-on-the-complaint-of-an-overseas-pakistani/?utm_cam paign=later-linkinbio-pro_pakistani&utm_content=later-17619366&utm_ medium=social&utm_source=linkin.bio. Accessed on 26 September 2021. Asia-Pacific Evaluation Association. (2020). EvalYouth’s regional chapter for the Asia Pacific Region. https://www.asiapacificeval.org/evalyouthasia. Accessed on 26 September 2021. Auditor General of Pakistan (AGP). (2021). About us. https://agp.gov.pk/Ove rview. Accessed 26 September 2021https://agp.gov.pk/Overview. Accessed on 26 September 2021. Auditor General of Pakistan (AGP). (2012). Performance audit manual. Supreme audit institution of Pakistan. https://agp.gov.pk/SiteImage/Misc/files/Per formance-Audit-Manual.pdf. Accessed on 30 January 2023. Awaz CDS. (2021). Decade of actions Pakistan development alliance 2010– 2020. Pakistan Development Alliance. https://www.pda.net.pk/wp-content/ uploads/2021a/05/Decade-of-Actions.pdf. Accessed on 30 January 2023. Cabinet Division. (2021). Year Book 2019–2020. Government of Pakistan. https://cabinet.gov.pk/SiteImage/Publication/Year%20Book%20(2019-20). pdf. Accessed on 30 January 2023.

314

A. AAQIL ET AL.

Centre for Economic Research in Pakistan. (2021a). Building Capacity to Use Research Evidence. https://www.cerp.org.pk/executive-education/buildingcapacity-to-use-research-evidence. Accessed on 26 September 2021a. Centre for Economic Research in Pakistan. (2021b). Evidenced Based Program Design. https://www.cerp.org.pk/executive-education/evidencebased-program-design. Accessed on 26 September 2021b. Centre for Economic Research in Pakistan. (2021c). Executive Education. https://www.cerp.org.pk/pages/executive-education. Accessed on 26 September 2021c. Centre for Economic Research in Pakistan. (2021d). Monitoring and Evaluation Boot Camp. https://www.cerp.org.pk/executive-education/monitoring-andevaluation-boot-camp. Accessed on 26 September 2021d. Centre for Economic Research in Pakistan. (2021e). Podcasts. https://www. cerp.org.pk/pages/podcasts. Accessed on 26 September 2021e. Cheema, A., Khwaja, A. I., Farooq Naseer, M., Shapiro, J. N., & Gill, J. (2020). Big push for the rural economy: Final impact evaluation report. Punjab Skills and Development Fund. https://www.psdf.org.pk/bpre-impactevaluation-report/. Accessed on 30 January 2023. Cheema, I., Farhat, M., Binci, M., Asmat, R., Javeed, S., & O’Leary, S. (2020). Benazir income support programme evaluation report. Oxford Policy Management. https://www.bisp.gov.pk/SiteImage/Misc/files/BISP_Evaluat ionReport_Ver%20without_FINAL.pdf. Accessed on 30 January 2023. Country Studies. (n.d.). Country Studies. Government Structure. Pakistan. Retrieved September 30, 2021, from http://countrystudies.us/pakistan/65. htm. Accessed on 30 September 2021. Directorate General Monitoring and Evaluation (DGME). (n.d.). Evaluation. https://dgmepunjab.gov.pk/evaluation/. Accessed on 26 September 2021. Economic Affairs Division. (2021a). NGO Policy. https://ngo.ead.gov.pk/ngopolicy. Accessed on 26 September 2021a. Economic Affairs Division. (n.d. a). Annual plan of action. Government of Pakistan. http://www.ead.gov.pk/SiteImage/Misc/files/viii_%20Annual% 20Plan%20of%20Action.pdf. Accessed on 30 January 2023a. Economic Affairs Division. (n.d. b). Draft memorandum of understanding. Government of Pakistan. http://www.ead.gov.pk/SiteImage/Misc/files/vi_ %20Draft%20MOU.pdf. Accessed on 30 January 2023b. Economic Affairs Division. (n.d. c). Frequently Asked Questions. https://ngo. ead.gov.pk/faq. Accessed on 26 September 2021b. Economic Affairs Division. (n.d. d). Launch of NGOs e-portal. https://ngo.ead. gov.pk/press-release-details/1. Accessed on 26 September 2021b. Economic Affairs Division. (n.d. e). MOU. https://ngo.ead.gov.pk/mou. Accessed on 26 September 2021c.

9

EVALUATION IN PAKISTAN

315

Ehsaas Strategy. (2019). The multi-sectoral and multi-stakeholder Ehsaas Strategy. Government of Pakistan. https://pass.gov.pk/userfiles1/files/Strategy_Ehs aas_for_online_consultation.pdf. Access on 30 January 2023. Finance Department. (2020). White Paper on the Budget 2020–21. Government of Balochistan. https://balochistan.gov.pk/wp-content/uploads/2020/06/ White-Paper-2020-21.pdf. Accessed on 30 January 2023. Government Of Balochistan. (2021). Inauguration of PSDP Automation Resource Center. https://balochistan.gov.pk/inauguration-of-psdp-automa tion-resource-center/. Accessed on 26 September 2021. Government of Pakistan. (2019). Pakistan’s implementation of the 2030 agenda for sustainable development: Voluntary national review. Pakistan. https://sus tainabledevelopment.un.org/content/documents/233812019_06_15_VNR_ 2019_Pakistan_latest_version.pdf. Accessed on 30 January 2023. Government of Pakistan. (2020). Budget Manual—Federal Government (4984(19)). Finance Division. https://www.finance.gov.pk/Bud get_Manual_1stEdition_2020.pdf. Accessed on 30 January 2023. Habib University. (n.d.). BSc (Honors) Social development & policy. https://habib.edu.pk/academics/ahss/social-development/. Accessed on 26 September 2021. Habib, S., & Rafique, Z. (2019). The role of civil society organizations in the consolidation of democratic system of governance: The case of Pakistan. Research and Analysis (ISSRA), National Defense University. Haque, N., Mukhtar, H., Ishtiaq, N., & Gray, J. (2020). Doing development better. Pakistan Institute of Development. Higher Education Commission Pakistan (HEC). (2011). Institutional performance of evaluation. Quality Assurance Agency. https://hec.gov.pk/english/ services/universities/QAA/externalQA/IPE/Documents/IPES%20Manual% 20for%20Universities%2011%20standards%2008%20Nov%2016.pdf. Accessed on 30 January 2023. Higher Education Commission Pakistan (HEC). (n.d. a). Mandate of M&E Division. https://hec.gov.pk/english/services/universities/Monitoring-Eva luation/Pages/Mandates.aspx. Accessed on 26 September 2021a. Higher Education Commission Pakistan (HEC). (n.d. b). PhD Produced by Pakistani Universities. https://www.hec.gov.pk/english/universities/Pages/ AJK/PhD-Produced-by-Pakistani-Universities.aspx. Accessed 26 Sept 2021b. Hussein, S. (2021). Critical evaluation of the budget making process in Pakistan. Knowledge Brief No. 2021:28. Retrieved March 4, 2022, from https://pide.org.pk/pdfpideresearch/kb-028-critical-evaluation-of-thebudget-making-process-in-pakistan.pdf. Accessed on 4 March 2022. Ijaz, S. (2019, January 23). “No Room to Bargain”: Unfair and abusive labor practices in Pakistan. Human Rights Watch. https://www.hrw.org/report/ 2019/01/24/no-room-bargain/unfair-and-abusive-labor-practices-pakistan.

316

A. AAQIL ET AL.

Accessed on 26 September 2021. Ikram, K. (2011). Revitalizing the planning commission; some recommendations. International Growth Core. https://www.theigc.org/wp-content/upl oads/2014/09/Ikram-2011-Working-Paper.pdf. Accessed 30 January 2023. Institute of Business Administration Karachi Department of Economics. (n.d.). Bachelor of Science (BS) Economics program. https://economics.iba.edu. pk/bs-economics.php. Accessed on 26 September 2021. Janju, H. (2021, February 11). Why is the Pakistani government cracking down on NGOs? Deutsche Welle. https://www.dw.com/en/why-is-thepakistani-government-cracking-down-on-ngos/a-56537755. Accessed on 25 September 2021. Khalayleh, A., Baloch, I., Dele-Ajayi, O., & Kaye, T. (2021). A monitoring and evaluation framework for blended learning: Pakistan Ministry of Federal education and professional training. Ed Tech Hub. https://docs.edtechhub. org/lib/XBPZPS3P/download/V8W3V6F2/Khalayleh%20et%20al.%20-% 202021%20-%20A%20Monitoring%20and%20Evaluation%20Framework% 20for%20Blended%20.pdf. Accessed on 30 January 2023. Kashf Foundation. (n.d.). Impact assessment. https://kashf.org/impact-assess ment/. Accessed on 26 September 2021. Khan, K., Waheed, A., Iqbal, S., & Khan, A. H. (2003). Monitoring and evaluation: Civil society organisations’ competitive edge in effective poverty alleviation [with Comments]. The Pakistan Development Review, 909–924. Khattak, F. H. (2014, May 8). Projects Evaluation in Pakistan. Planning, Development & Reform Division. Khushi, S. (2017, February 19). NGOs registration laws in Pakistan. LinkedIn. https://www.linkedin.com/pulse/ngos-registration-laws-pakistansamina-khushi. Accessed on 26 September 2021. Kiani, K. (2020, July 11). Planning commission told to update project evaluation, monitoring. Dawn. https://www.dawn.com/news/1568265. Accessed on 26 September 2021. LUMS Centre for Business and Society. (2019). CBS’s Monitoring & Evaluation Training Programme for Government Officials of Merged Areas of Khyber https://cbs.lums.edu.pk/news/general-news/cbs%E2%80% Pakhtunkhwa. 99s-monitoring-evaluation-training-programme-government-officials-mergedareas. Accessed on 26 September 2021. LUMS Centre for Business and Society. (n.d.). Current Training Programme. https://cbs.lums.edu.pk/current-executive-training-programme. Accessed on 26 September 2021. Oxford Policy Management. (n.d.). Evaluating the Benazir Income Support Programme. https://www.opml.co.uk/projects/evaluating-benazir-incomesupport-programme. Accessed on 26 September 2021.

9

EVALUATION IN PAKISTAN

317

Ministry of National Health Services Regulations and Coordination. (2016). National Health Vision 2016–2025. Government of Pakistan. https://ext ranet.who.int/countryplanningcycles/sites/default/files/planning_cycle_rep ository/pakistan/national_health_vision_2016-25_30-08-2016.pdf. Accessed on 30 January 2023. Ministry of National Health Services, Regulations and Coordination. (2020). National Action Plan for Corona virus disease (COVID-19) Pakistan. Government of Pakistan. https://www.nih.org.pk/wp-content/uploads/2020/03/ COVID-19-NAP-V2-13-March-2020.pdf. Accessed on 30 January 2023. Ministry of Federal Education & Professional Training. (2018). National Curriculum Framework Pakistan. Government of Pakistan. https://www.pc. gov.pk/uploads/report/NCF.pdf. Accessed on 30 January 2023. Ministry of Planning Development & Special Initiatives. (2019, July 22). Planning Commission taking lead on strengthening population statistics [Press release]. https://www.pc.gov.pk/web/press/get_press/334. Accessed on 30 January 2023. Ministry of Federal Education and Professional Training. (2019). National Education Assessment System [Brochure]. Government of Pakistan. http:// www.neas.gov.pk/SiteImage/Misc/files/NEAS%20Brochure%202019.pdf. Accessed on 30 January 2023. Ministry of Planning, Development and Reform. (2019). Manual for development projects: Identification, preparation, appraisal, approval, implementation, monitoring and evaluation. Government of Pakistan. Ministry of Human Rights. (2020). Ministry of Human Rights Report: August 2018-July 2020. Government of Pakistan. Ministry of Human Rights. (n.d.). Development Wing. http://www.mohr. gov.pk/Detail/YzkzNjQ2MzUtOThhYy00YmEwLTg0MGYtYjRmODg5Zm Y5ZmU5. Accessed on 26 September 2021. Ministry of Federal Education and Professional Training. (n.d.). Introduction. http://www.mofept.gov.pk/Detail/NDM1NDI0ZTQtZmFjMy00Z TVlLWE5M2YtYjgxOTE4YTkyYWNi. Accessed on 26 September 2021. Ministry of Planning Development & Special Initiatives. (n.d.). The Ministry. https://www.pc.gov.pk/web/ministry. Accessed on 26 September 2021. Ministry of Human Rights. (2021c). National action plan on business and human rights. (2021c–2026). Government of Pakistan. https://www.ohchr. org/Documents/Issues/Business/NationalPlans/National-Action-PlanBHR-Draft-03-03-21.pdf. Accessed on 30 January 2023. Ministry of National Health Services, Regulations and Coordination. (n.d.). Pakistan health knowledge hub. https://phkh.nhsrc.pk/. Accessed on 26 September 2021.

318

A. AAQIL ET AL.

National Assembly of Pakistan. (2017). The Right of Access to Information Act [Bill]. Government of Pakistan. http://www.na.gov.pk/uploads/documents/ 1506960942_594.pdf. Accessed on 30 January 2023. National Assembly of Pakistan. (2018). The Constitution of Islamic Republic of Pakistan: As amended up-to-date. All Pakistan Legal Decision. National Accountability Bureau. (2020). Annual Report 2020. National Accountability Bureau. National Democratic Foundation. (n.d.). Governmental structure of Pakistan. https://www.democraticfoundation.com.pk/govt-structure-of-pakistan. Accessed on 30 September 2021. Nations Online. (n.d.). Pakistan. https://www.nationsonline.org/oneworld/pak istan.htm#Countryprofile. Accessed on 30 September 2021. National Initiative for Sustainable Development Goals. (n.d.). Home. https:// www.sdgpakistan.pk/. Accessed on 26 September 2021. Operational Wings. (n.d.). Planning and Development Department. https:// pnd.sindh.gov.pk/pd-departments. Accessed on 18 August 2021. Pakistan’s Growth Story. (n.d.). Home. https://devpakblog.com/. Accessed on 26 September 2021. Pakistan Journal of Medical Sciences. (n.d.). Home. https://www.pjms.org.pk/ index.php/pjms. Accessed on 26 September 2021. Pakistan Institute of Management. (n.d.). Project monitoring, evaluation & control (Islamabad). https://www.pim.com.pk/product/project-monitoringcontrol-evaluation-isl/. Accessed on 26 September 2021. Pakistan Institute of Management. (n.d.). Professional Diploma in Project https://www.pim.com.pk/product/diploma-in-project-man Management. agement-khi/. Accessed on 26 September 2021. Pakistan Journal of Educational Research and Evaluation. (n.d.). Description. http://pjere.pu.edu.pk/. Accessed on 16 September 2021. Pakistan Centre for Philanthropy (PCP). (n.d. a). About us. https://www.pcp. org.pk/aboutus.html. Accessed on 26 September 2021f. Pakistan Centre for Philanthropy (PCP). (n.d.). PCP Certified Network. https:// pcp.org.pk/pagestyle.php?catd=664. Accessed on 26 September 2021. Pakistan Institute of Development Economics. (2021). Prospectus. https:// pide.org.pk/admissions/prospectus-2021a-2022/. Accessed on 26 September 2021. Pakistan Poverty Alleviation Fund (PPAF). (n.d.). Impact evaluations. http:// www.ppaf.org.pk/impact. Accessed on 26 September 2021. Pakistan Institute of Development Economics. (n.d.). Training Courses. https:// pide.org.pk/training-courses/. Accessed on 26 September 2021. Planning Commission. (2011). Public Sector Development Projects [Brochure]. Government of Pakistan. https://www.pc.gov.pk/uploads/psdp/psdp_broc hure_16-05-2011.pdf. Accessed on 30 January 2023.

9

EVALUATION IN PAKISTAN

319

Planning Commission. (2021a). M/O Planning, development & special initiatives: Projects wing government of Pakistan. https://www.pc.gov.pk/upl oads/tender/TORs_pre-qualification_firms_ME_PSDP.pdf. Accessed on 30 January 2023. Planning Commission. (2021b). Project Life Cycle Stages and Project Proformas, Manual for Development Projects. https://www.pc.gov.pk/upl oads/archives/Manual-Projects-2021.pdf. Accessed on 30 September 2021. Planning and Development Board. (2021). Punjab Annual Development Programme. https://pnd.punjab.gov.pk/system/files/ADP%20202122%20V2.pdf#overlay-context=apd-20-21. Accessed on 26 September 2021. Planning and Development Department. (2020). Annual Development Programme Policy. Government of Khyber Pakhtunkhwa. https://pndkp.gov. pk/adp-policy/. Accessed on 30 January 2023. Planning and Development Department. (2021). Annual Development Programme 2021–22. Government of Khyber Pakhtunkhwa. https://pndkp. gov.pk/download/adp-final-2021-22/. Accessed on 30 January 2023. Planning and Development Department. (2021). Annual Development Plan 2021–22. Government of Sindh. https://pnd.sindh.gov.pk/ADP2021-22. Accessed on 27 September 2021. Planning Commission Ministry of Planning, Development & Special Initiatives. (2021). Public Sector Development Programme 2021–22. Government of Pakistan. https://www.pc.gov.pk/uploads/archives/PSDP_2021-22.pdf. Accessed on 30 January 2023. Poverty Alleviation and Social Safety Division. (n.d.). Ehsaas M&E Framework. https://www.pass.gov.pk/Detailaea953ad-da5a-41a4-bd8f-c853c0edd133. Accessed on 26 September 2021. Prime Minister’s Performance Delivery Unit (PMDU Pakistan). (n.d.). Pakistan’s Citizen’s Portal. https://citizenportal.gov.pk/index.php#about. Accessed on 26 September 2021. Prime Minister’s Performance Delivery Unit (PMDU Pakistan). (2020d, December 8). Issue Resolved from Skardu, Gilgit Baltistan, Pakistan Citizen’s Portal (PCP) [Video]. YouTube. https://www.youtube.com/watch?v= 74tPzAmKPZU&t=18s. Accessed on 30 January 2023. Prime Minister’s Performance Delivery Unit (PMDU Pakistan). (2020, October 25). Health Related Issue Resolved, Pakistan Citizen’s Portal (PCP) [Video]. YouTube. https://www.youtube.com/watch?v=lrdyGRAGCaU. Accessed on 30 January 2023. Prime Minister’s Performance Delivery Unit (PMDU Pakistan). (2021g, February 3). Provision of transport service for senior citizens of Katchi Abbadi [Video]. YouTube. https://www.youtube.com/watch?v=pVRx2X kd56Q. Accessed on 30 January 2023.

320

A. AAQIL ET AL.

Public Accounts Committee. (2021). About PAC. http://www.pac.na.gov.pk/? q=about. Accessed on 26 September 2021. Punjab Economic Research Institute (PERI). (n.d.). Research Papers. https:// peri.punjab.gov.pk/research_papers. Accessed on 26 September 2021h. Punjab Economic Research Institute (PERI). (n.d. b). Seminars. https://peri. punjab.gov.pk/seminars. Accessed on 26 September 2021i. Punjab Economic Research Institute (PERI). (n.d. c). Workshops/Trainings. https://peri.punjab.gov.pk/workshops_trainings. Accessed on 26 September 2021. Rana, M. A. (2020). Decentralization Experience in Pakistan: The 18th Constitutional Amendment. Asian Journal of Management Cases, 17 (1), 61– 84. https://journals.sagepub.com/doi/full/https://doi.org/10.1177/097 2820119892720. Accessed on 30 January 2023. Radio Pakistan. (2021, July 9). PM directs to link performance report of Govt officials with their efficiency to resolve problems of people. Radio Pakistan. http://www.radio.gov.pk/09-07-2021/pm-directs-to-link-perfor mance-report-of-govt-officials-with-their-efficiency-to-resolve-problems-ofpeople. Accessed on 26 September 2021. Rural Support Programmes Network (RSPN), Pakistan. (n.d.). Training. http://www.rspn.org/index.php?s=training&search-type=page%2C+attach ments. Accessed on 26 September 2021. Rosenstein, B. (2015). Status of National Evaluation Policies Global Mapping Report. Parliamentarians Forum on Development Evaluation in South Asia. https://globalparliamentarianforum.files.wordpress.com/2016/02/thestatus-of-evaluation-policies.pdf. Accessed on 30 January 2023. Sustainable Development Policy Institute. (n.d.). Training. https://sdpi.org/tra ining. Accessed on 26 September 2021. Salim, Y. (2021, August 29). The Single National Curriculum Schism. TMagazine. https://tribune.com.pk/story/2317530/the-single-national-cur riculum-schism. Accessed on 26 September 2021. Salman, Z. (2018, February 23). Can we trust our policymakers to be unbiased? Pakistan’s Growth Story. https://devpakblog.com/2018/02/23/canwe-trust-our-policymakers-to-be-unbiased/. Accessed on 30 January 2023. Sarfraz, S., & Ahmad, R. (2020). 10 minutes with Dr Shabnum Sarfraz, Senior Advisor for Health Systems and Policy Research at P2IMPACT Associates and member Social Sector & Devolution, Planning Commission, Government of Pakistan, Islamabad, Pakistan. BMJ Leader, 2022(6), 248–250. https://doi. org/10.1136/leader-2021-000479. Accessed on 30 January 2023. Shah, F. (2016, August 11). The rise of NGO’s and their harmful impact on Pakistan. Herald Magazine. Retrieved September 26, 2021, from https:// herald.dawn.com/news/1152863. Accessed on 26 September 2021.

9

EVALUATION IN PAKISTAN

321

Shahzad, K., Kapri, A., Ali, S., & Ali, I. (2018). Monitoring & Evaluation Framework Nutrition-Sensitive Component—PINS ER-3. Rural Support Programmes Network. http://www.rspn.org/wp-content/uploads/ 2019/05/RSPN-PINS-ER-3-ME-Framework-Final-15-Apr-19.pdf. Accessed on 30 January 2023. Survey of Pakistan. (2021). Training on Project Monitoring & Evaluation System (PMES) Software. http://surveyofpakistan.gov.pk/NewsDetail/YTdmYzgzZ TQtYjM5Yy00ZDEwLWJmYWUtOTEwMTQ4ZmVlZDc5. Accessed on 26 September 2021. The Express Tribune. (2012, June 2). Info: What is the public sector development programme? The Tribune. https://tribune.com.pk/story/387 630/info-what-is-the-public-sector-development-programme. Accessed on 26 September 2021. The Pakistan Development Review. (n.d.). Table of Contents. http://www.the pdr.pk/pdr/index.php/pdr/index. Accessed on 26 September 2021. The World Bank. (2021j, March 29). The World Bank in Pakistan; Overview. The World Bank. https://www.worldbank.org/en/country/pakistan/overview. Accessed on 30 January 2023. Umar, A. (2021, September 3). Reforming the federal development programme. The Express Tribune. https://tribune.com.pk/story/2318298/reformingthe-federal-development-programme. Accessed on 26 September 2021. United Nations Institute for Training and Research. (n.d.). GIT Applications for M&E and Sustainable Planning (Pakistan/EO4SD). https://www.unitar. org/event/full-catalog/git-applications-me-and-sustainable-planning-pakist aneo4sd. Accessed on 26 September 2021. Williams, B., & Shankar, M. (2008). Evaluation South Asia. UNICEF South Asia. World Population Review. (2021). Pakistan Population 2021 (Live). https:// worldpopulationreview.com/countries/pakistan-population. Accessed on 30 January 2023. Zaman, A. (2020). Macro Models for Policy. Pakistan Institute of Development Economics. https://pide.org.pk/blog/macro-models-for-policy/. Accessed on 1 August 2022.

CHAPTER 10

Evaluation in the Philippines Asela Kalugampitiya, Ana Erika Lareza, and Romeo Santos

1

General Country Overview

The Republic of the Philippines is a country in Southeast Asia, an archipelago that consists of some 7,100 islands and islets. It is bounded by the West Philippine Sea to the north and west, Philippine Sea to the east, and Sulu Sea to the south west. It has an area of around 300,000 sq. km (120,000 sq. mi), which supports a population of around 110 million. In January 2018, it was declared the 8th-most populated country in Asia and the 13th-most populated country in the world. The Philippines is a founding member of the United Nations, World Trade Organization, Association of Southeast Asian Nations, the Asia–Pacific Economic Cooperation forum, and the East Asia Summit. The country is considered to be an emerging market and a newly industrialised country, which has an economy transitioning from being agriculture-based to being more on services, manufacturing, and tourism industry. Its current primary exports

A. Kalugampitiya (B) University of Sri Jayewardenepura, Nugegoda, Sri Lanka e-mail: [email protected] A. E. Lareza · R. Santos PhilDev Foundation, Pasig, Philippines © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_10

323

324

A. KALUGAMPITIYA ET AL.

include electronics, semiconductors, transport equipment, construction materials, and minerals. The Philippines is a constitutional republic, where the President functions as both the head of state and the head of government of the country in a plural, multi-party system.1 The government is equally divided into three interdependent branches whose powers are vested by the Philippine Constitution, as illustrated in the succeeding chart (Table 1). The Philippines has 4 main classes of administrative divisions that are collectively known as Local Government Units (LGUs). The divisions are the Autonomous Regions, Provinces, Municipalities, and Barangays. Table 1

The Philippine Government and its three branches

Government branch

Role/function

Main features

Executive branch

Carries out and enforces laws

Legislative branch

Interprets the meaning of laws, applies laws to individual cases, and decides if laws violate the Constitution

Judicial branch

Enacts legislation, confirms or rejects Presidential appointments, and has the authority to declare war

Includes the President, Vice President, Cabinet, Executive Departments, independent agencies, boards, commissions, and committees. The President and Vice President both serve a 6-year administrative term while Cabinet members serve as advisors and are nominated by the President. Bicameral in form. Composed of Upper House (Senate) formed by 24 elected Senators and Lower House (House of Representatives) formed by 250 elected members from legislative districts in the provinces, cities, and municipalities, and representatives elected through a party-list system of registered national, regional, and sectoral parties or organisations. Judicial power is vested in 1 Supreme Court and in Lower Courts as may be established by law.

Source Authors’ own illustration

1 Philippines, Office of the President [Corazon C. Aquino]. Executive Order No. 292, s. 1987: INSTITUTING THE “ADMINISTRATIVE CODE OF 1987”. 25 July 1987.

10

EVALUATION IN THE PHILIPPINES

325

Ambisyon Natin 2040 (NEDA, 2015a) is a nationwide vision statement developed in 2015 by the National Economic Development Authority and was officially launched in March 2016. With Asia being projected to be the centre of global economy by 2050, Ambisyon Natin 2040 is meant to keep the country in a forward-looking approach to become at par with the region’s economic growth and development. In an effort to keep the country’s citizens to be at the centre of its development planning, a series of nationwide public consultations were convened to capture the Filipino’s vision for themselves and use it as the government’s guide to design initiatives that are geared towards the attainment of the people’s aspirations. This consultation led to the development of the Ambisyon Natin 2040 and is now representing the nation’s vision for the country and its citizens for the next 25 years. This vision has been put into action after the signing of the Presidential Executive Order No. 05, Series 2016 “Approving and Adopting the Twenty-Five-Year Long Term Vision Entitled Ambisyon Natin 2040 as Guide for Development Planning” (Government of the Philippines, 2016). Ambisyon Natin 2040 was officially approved to be adopted as an anchor and guide for the development of all of the Philippine development plans until 2040 which covers up to four presidential administrations. The rationale for this executive order is to ensure sustainability and consistency of strategies, policies, programmes, and projects across political administrations. Within the same executive order there was an additional directive that all government departments, offices, and instrumentalities, such as government-owned and/or controlled corporations and LGUs shall create plans that are consistent with the Ambisyon Natin 2040.

2

Institutional Structures and Processes 2.1

Evaluation Regulations

As a follow up to the participation of the Senate of the Philippines at the EvalColombo2018 event, where it became a signatory to the Colombo Declaration on Evaluation, the Senate hosted the second day of the 2nd Asia Pacific Evaluation Association (APEA) Conference at the Senate premises, where it committed to bring an act to establish evaluation. As a result, two senate bills were filed for the establishment of a legislated national evaluation in the Philippines. First was

326

A. KALUGAMPITIYA ET AL.

the Senate Bill No. 788—An Act Establishing a National Evaluation Policy (Senate of the Philippines, 2019), filed few months after the APEA conference by Senator Risa Hontiveros. The submitted Bill refers to the National Economic and Development Authority—Department of Budget and Management (NEDA-DBM) joint memorandum for NEPF and mentions that the implementation is still challenging due to changes in government administration. The purpose of the Act is to strengthen the legal and institutional framework for the regular conduct of monitoring and evaluation of the results of public policies, programmes, and projects. The Act also suggests to establish a National Evaluation Council (Senate of the Philippines, 2019) which is an important step in many ways. The next steps for the Bill are yet to be decided by the Senate. Going beyond the above-mentioned Bill, Senator Imee R. Marcos submitted the most recent Senate Bill No. 1885—An Act Establishing a Result-Based National Evaluation Policy (RBNEP) (Senate of the Philippines, 2020a) which already had its first reading and approval of the upper house in October 2020. The Bill was then referred to the Senate committee secretary and NEDA for the refinement of the draft bill for further review. This Bill aims to improve the overall effectiveness of public policies, strategies, programmes, projects, and overall organisational performance by strengthening accountability and learning through the enactment of the Results-Based National Evaluation Policy. The Bill also aims to contribute to the achievement of inclusive development and poverty reduction goals by institutionalising the legal framework for the regular conduct of monitoring and evaluation of the results of ongoing and completed development interventions (Senate of the Philippines, 2020a): This Results-Based National Evaluation Policy (RBNEP) is aimed to apply to all branches of government, i.e. Executive, Legislative and Judiciary. Thus, the purpose of RBNEP is to harness the enormous potential of evaluations as important means for alleviating poverty and improving the lives of all Filipinos by ensuring that public policies, strategies, programs and projects are informed by sound evidence and lead to effective and equitable results.

Although both Bills are in process, they include progressive features for institutionalisation of evaluation in the Philippines and once approved, could influence the evaluation field in the country. In a commentary

10

EVALUATION IN THE PHILIPPINES

327

document provided by the Philippine Institute for Development Studies,2 having these bills enacted will enable the development and/or improvement of evaluation functions in the Philippine government. It will also guide the government for the creation of necessary infrastructures to conduct evaluations. As quoted from the Senate Economic Planning Office—Policy Brief on “Institutionalizing a National Evaluation Policy (NEP)”3 : An institutionalized NEP in the government systems would be instrumental for achieving significant progress towards a government-wide shift to development results. Promotion of an evaluation culture at various levels, securing a higher-level policy commitment and addressing capacity gaps in managing evaluations are key attributes in the utilization of evaluations. Thus, institutionalizing a NEP in the country through the passage of the proposed legislation would be a major step towards achieving this.

Currently in the Philippines, the National Evaluation Policy Framework (NEPF) in place was developed by NEDA in an effort to improve the components of the public sector management cycle. It was introduced to provide a framework for the purposive conduct of evaluation of all programmes and projects implemented by all government entities and was an effort to improve the performance of public sector management. In 2015, the NEDA and DBM issued a joint memorandum circular (No. 2015–10: National Evaluation Policy Framework) to inform all public institutions on the implementation of the NEPF. The Circular has three purposes (NEDA, 2015b): 1. Promote and support for evidence-based decisions by providing essential knowledge and evidences to stakeholders regarding respective programmes and projects to create evidence-based decisionmaking related to the current and future planning of concerned initiative.

2 https://pidswebs.pids.gov.ph/CDN/OUTREACH/pids_comments_on_senate_bills_ 788_and_1885_1612020.pdf. Accessed on 13 September 2023. 3 https://legacy.senate.gov.ph/publications/SEPO/Policy%20Brief_Institutionalizing% 20A%20National%20Evaluation%20Policy%20(NEP)12July%202021.pdf. Accessed on 1 March 2023.

328

A. KALUGAMPITIYA ET AL.

2. Ensure programme improvement by providing programme/project managers, decisions makers, and key stakeholders of feedbacks and learnings to improve concerned initiative. 3. Ensure accountability by providing to the civil society, beneficiaries, donors, and other interested parties of the evaluation results regardless whether it is positive or negative. The NEPF intended to cover the evaluation (at least once at the end of its life cycle) of all programmes and projects being implemented by all government entities (regardless of its funding source) or its instrumentalities such as Government agencies, State Universities and Colleges, Government-Owned and/or Controlled Corporations, Government Financial Institutions with budgetary support from the National Government, and other instrumentalities of the government. The above-mentioned entities were mandated to allocate adequate resources from their annual budget to ensure compliance with the provisions of the framework. The allocation should include the following: 1. capacity development during the start-up phase of the Policy Framework; 2. ongoing salaries, recruitment, and training to ensure an adequate supply of internal personnel competent in evaluation; 3. operations and maintenance, and 4. external professional service fees. For the implementation and operationalisation of the framework, an Inter-Agency Evaluation Task Force and its Secretariat composed of the NEDA and DBM Secretaries as Chairperson and Co-chair along with the President—Presidential Management Staff were established. The functions of the taskforce involved the following: 1. provide overall policy direction and coordination on the evaluation agenda of the public sector; 2. report to NEDA Board on all evaluations conducted in the public sector; 3. authorise and commission the conduct of evaluations on top of those conducted by the implementing agencies; 4. issue evaluation standards and guidelines;

10

EVALUATION IN THE PHILIPPINES

329

5. assess evaluation agenda of implementing agencies; 6. adopt sanctions and incentives system; and, 7. ensure the creation of appropriate institutional structures to mainstream the Policy Framework. Through the NEPF, the proponents believed that projects and programmes nationwide will be assessed in terms of their efficiency, outcomes, and impacts based on national priorities. This is done through a continuously maintained and updated six-year evaluation agenda (also known as the Philippine Development Plan—Results Matrix (PDP RM) which is meant to compliment and coincide with the timeframe of the Philippine Development Plan (PDP) and Public Investment Programme (PIP). As the framework covers all sectors and institutions in the country, there were no sector-specific evaluation regulations which existed and there is no need for that. The actual implementation of the NEPF was not fully realised, likely due to many valid reasons. One of these was the tough political climate that existed during the end of the Aquino III Administration (2010– 2016). The environment for the implementation of a new, untested, and radically different system of public sector management, as the NEPF, was a hard pill to swallow in the eyes of a restless public. In hindsight, the NEPF possessed weaknesses, that lead to its early termination. Aside from not having a specific set of rules, guidelines, and procedures in cascading to new M&E offices to be set up (as required of all government functionaries, including State Universities, etc.), it was not yet legally established through a legislative process, therefore, short of being completely institutionalised. There are no sectoral laws or regulations about evaluation or use of evaluation which apply throughout the country. 2.2

Parliament and National Audit Structure

The Philippine Senate is the first national parliament in East Asia to host an international evaluation event, where a Parliamentarian’s session was held at the House of the Philippine Senate during the conduct of the 2nd APEA Conference in February 2019. This session was a follow up to the previously held EvalColombo2018 in Colombo, Sri Lanka where participating countries created commitments to create a “call to action” towards the development of evaluation agendas to support Agenda 2030

330

A. KALUGAMPITIYA ET AL.

(GPFE, 2018). Through the APEA Conference, the Philippine legislation acknowledged their role to strengthen the NEPF (Senate of the Philippines, 2019) and called for evaluating all government programmes and projects. This is an example of how a Voluntary Organisation for Cooperation in Evaluation (VOPE) can be instrumental in mobilising governments and Parliaments for evaluation regulations. As mentioned in the previous section, two Senators submitted draft bills (#788 and #1885) to regulate the national evaluation policy. Both bills comprehensively articulate the current status of evaluation in the country and why evaluation should be regulated with legal provisions. This further signifies the fact that there is, while not mainstream, an increasing demand for evaluation and its results. Parliamentarians are looking for data and evidence to help aid decision-making, and the existing evaluation infrastructure, which is often integrated within planning functions, is not enough to supply the existing demand. Overall, the demand also depends on the particular interests of the respective representative. The parliament itself does not have its own evaluation unit and does not commission evaluations. Accordingly, it can be concluded that the demand for evaluations certainly exists, but it is not fully institutionalised. The need for a stronger National Evaluation Policy in the country was also addressed in recognition to evaluation being not widely and systematically integrated within all government processes and systems. Having a policy creates clearer language in setting expectations for performance, results, and transparency and will help the government prioritise and make better decisions in spending, and serves as a guide in framing other policies (Senate of the Philippines, 2020a). Commission on Audit (COA) is an independent constitutional commission of the Philippines that examines, audits, and settles all accounts pertaining to the revenue and receipts of, and expenditures or uses of assets of the government. According to the 1987 Philippine Constitution Article IX-D—The Commission on Audit, while COA has often been focused on monetary aspects, it has exclusive authority to “define the scope of its audit and examination, establish the techniques and methods required therefore, and promulgate accounting and auditing rules and regulations” (Commission on Audit, n.d.), in understanding that it’s for the prevention and disallowance of irregular, unnecessary, excessive, extravagant, or unconscionable expenditures, or uses of government funds and properties (Government of the Philippines, 1987). Hence, under the COA Resolution No. 2017–012, COA

10

EVALUATION IN THE PHILIPPINES

331

created a Performance Audit Office under its Special Services Sector. It is solely dedicated to conduct evaluation or performance audits on several low-performing government projects and programmes and provide recommendations to improve its effectiveness and efficiency. Since its establishment in 2017, the Performance of Audit Office has conducted a total of 10 evaluations. These include evaluations of several environmental, infrastructure, and health programmes, selected in response to the demands of legislators for evidence of impact or to identify causes of underperformance of the selected government programmes. For example, in 2011, the government launched the National Greening Program (NGP) to recover the lost forest cover of the Philippines. Eight years after its implementation, legislators are sceptical of the impact of the programme. Due to a lack of available evidence to prove its positive impact on the environment and beneficiaries, NGP’s budget was cut, from Php 5.15 billion to Php 2.60 billion in 2019.4 This prompted COA to conduct a performance audit to identify the impacts and the efficiency and effectiveness of the implementation of the programme. While it can be inferred that there is a regular evaluation under COA, existing evaluations showed that there are no specified criteria to tell which programmes to evaluate, including the methodologies and standards used to conduct the evaluation. Additionally, there is no available documentation stating that the results of the evaluations were used for the improvement of existing or planning of future programmes. 2.3

Organisational Structure

In 1987, under the Executive Order No. 230 (Government of the Philippines, 1987), the National Economic and Development Authority was restructured and mandated to be responsible for formulating, monitoring, evaluating, and continuing coordinated and fully integrated social and economic policies, plans, and programmes of the Philippine government. At present, NEDA is the country’s primary socioeconomic planning body and is regarded as the authority in macroeconomic forecasting and policy analysis and research of the Philippines. It provides highlevel advice to policymakers in Congress and the Executive Branch and is

4 https://www.coa.gov.ph/index.php/cy-2019/category/8085-national-greening-pro gram. Accessed on 2 March 2023.

332

A. KALUGAMPITIYA ET AL.

responsible for the national and subnational coordination for the formulation of policies, plans, and programmes. This also includes review, evaluation, and monitoring of infrastructure projects to increase investment spending for the growing demand on quality infrastructure facilities, and short-term policy reviews to provide critical analyses of development issues and policy alternatives to decision-makers. Under NEDA’s investment programming group is the monitoring and evaluation staff whose function is to monitor and evaluate the progress of ongoing strategic development programmes and projects that contribute to the achievement of the PDP goals (NEDA, 2020b). This function also includes maintenance of information systems for the aforementioned programmes and projects, as well as setting up standards, performance measurements, methodologies, procedures, and guidelines on M&E-related initiatives/activities. Under the monitoring and evaluation staff also exist various divisions: Transport Infrastructure Sector Division, Non-transport Infrastructure Sector Division, Social Sector Division, and Economic Sector Division. All divisions are in charge of monitoring the progress of their respective sector’s development programmes, specifically its alignment and strategic achievement of the PDP. Each division is also designated to provide technical assistance for M&E of ongoing programmes, conduct ex-post project evaluations on selected completed programmes, maintain information systems and set-up standards, performance measurements, methodologies, procedures, and guidelines on M&E-related initiatives/ activities including RBM, post-evaluation, among others. To fulfil its role, NEDA, throughout the years, has been conducting (along with the DBM) various reforms to build on previous planning and budgeting initiatives based on various tools and processes. By 2011, NEDA started giving greater focus on the achievement of development results on priority sectors as DBM started to shift to resultsbased budgeting to ensure that the outputs of government agencies are consistent with their mandates. It was also within this year when harmonisation of the Executive branch’s performance management systems was prioritised, strengthening the link in the country’s planning and budgeting processes to ensure coherence on national targets and priorities (PDP RM) and agency deliverables (Major Final Outputs). This harmonisation was brought upon by the Administrative Order 25—Creating an inter-agency task force on the harmonisation of national government

10

EVALUATION IN THE PHILIPPINES

333

performance monitoring, information, and reporting systems (Government of the Philippines, 2011). Through administration order No. 25, it has been acknowledged that evaluation is an essential component for an effective and efficient performance management system and that there is a need for a harmonised results-based performance management system. This change in focus brought upon a need for the capacity to monitor and gather evidences that the government is able to achieve their intended development results and adopt alternative strategies when evidence suggests that results are not being achieved. Thus in 2015, NEDA and DBM developed a national evaluation policy framework for the conduct of evaluations in the public sector to ensure support for evidence-based decisions, and ensure programme improvement and accountability which is explained in Subchapter 2.1. 2.4

Evaluation Practice and Use

With the NEDA and DBM’s focus shifted to results-based planning and budgeting processes in 2011, the demand for evaluation is steadily increasing. This is especially true for the purposes streamlining and improving existing government initiatives that are often marked with extensions, cost overruns and outputs, and outcomes that do not fit the desired national objectives. The release of Ambisyon Natin 2040, a major thrust of the current administration to further streamline government programmes towards achieving its vision, and other initiatives such as the 0–10 Socioeconomic Agenda, and country commitments in Sustainable Development Goals (SDGs) further boosted the demand for evaluation. As of 2019, NEDA has commissioned several evaluation studies of varying statuses and capacity-building activities for NEDA CO, RO, and attached agencies (David, 2019). Since 2012, there has been a spike of government-led national and regional evaluation studies. The following are some examples of completed evaluation studies commissioned by NEDA to the Philippine Institute for Development Studies (PIDS), NEDA’s attached agency, and the government’s primary socioeconomic think tank, since 2012 starting from their latest evaluation study: 1. Evaluation of the Effects of the Performance-Based Bonus Incentive Scheme

334

A. KALUGAMPITIYA ET AL.

2. Process Evaluation of the Universal Access to Quality Tertiary Education Act (RA 10931): Status and Prospects for Improved Implementation 3. Process Evaluation of the Performance-Based Bonus (PBB) Scheme 4. Evaluation of the Impact of Agricultural Insurance Program of the Philippine Crop Insurance Corporation on Agricultural Producers in Region IV-A (CALABARZON) 5. Impact Evaluation of Banana Insurance Program of the PCIC in the Davao Region 6. More than Infrastructure and Equipment: Process Evaluation of the Health Facilities Enhancement Program 7. Formative Evaluation of the DOH‘s Complete Treatment Pack (Compack) Program. Several government agencies do have their own fully established M&E units to conduct their monitoring and evaluation functions. One example is the Department of Health, which was able to develop its own monitoring and evaluation guidebook5 for local government officials and programme managers who handle sanitation projects and programmes. The guidebook provides evaluation templates and tools to help its users identify the impact of its project, specifically on intended impacts that benefit the project stakeholders. The guide also details the purpose of the evaluation, and how it can be used for the next planning phase of the project or programme. However, the creation of M&E units still remains a challenge for other agencies, specifically for Local Government Units. For such instances, their M&E functions are often integrated into the mandates of their planning departments/divisions. The following are examples of implementing agencies with their own M&E set-ups: 1. Department of Social Welfare and Development 2. Department of Transportation and Communications 3. Metropolitan Waterworks and Sewerage System 4. Technical Education and Skills Development Authority

5 https://doh.gov.ph/sites/default/files/publications/Monitoring_and_Evaluation.pdf. Accessed on 25 January 2022.

10

EVALUATION IN THE PHILIPPINES

335

5. Department of Science and Technology 6. Department of Trade and Industry 7. Supreme Court of the Philippines 8. Commission on Higher Education 9. National Council on Disability Affairs 10. National Housing Authority 11. Department of Environment and Natural Resources 12. Department of Agrarian Reform 13. Department of Agriculture There are a lot of limitations on these agencies for them to conduct their own evaluations, such as financial barriers and varying levels of evaluation capacities across agencies and levels. Limitation in capacities includes the unavailability of manpower with the necessary skills to conduct evaluations and resources to develop these skills. This also includes accessibility and capacity to use evaluation tools and methodologies. For agencies who have the financial resources to conduct evaluations, ex-ante impact evaluations are more common than any other type of evaluations. Due to a lack of capacity and experience in evaluation, these evaluations are often conducted by external evaluators. Evaluation is also more frequently conducted on foreign-assisted programmes and projects at the discretion of funding development agencies and donors, such as the Asian Development Bank and the Japan International Cooperation Agency. Evaluations are also being conducted in the climate change initiatives in the Philippines. In the Philippine National Climate Change Action Plan (NCCAP) 2011–2028 (Climate Change Commission, 2011), a provision for the establishment of an M&E system to track the progress and results of NCCAP is included (IIED, 2019) with an aim to integrate climate risks into the planning processes of the current government initiative. The NCCAP RBMES draws data from gathered information at national and subnational levels, and aggregates the results from the seven thematic areas of NCCAP. This M&E system is geared towards measuring adaptation, and focuses on evaluating the outcomes of adaptation plans using the Results-Based Monitoring and Evaluation as well as for the re-orientation of NCCAP if deemed needed. Formally it addresses both evaluation and monitoring, however, in practice it is mostly on monitoring.

336

A. KALUGAMPITIYA ET AL.

Under NEDA, 17 evaluations have been commissioned since 2017. Out of 17 evaluations, 13 are summative evaluations, while the rest are process and formative evaluations. These evaluations are from varying sectors ranging from infrastructure development to governance and institutions development. A detailed breakdown of the evaluation types and sectors is available in Tables 2 and 3. While the Philippine NEPF states that all government initiatives should at least have one evaluation conducted in their project life cycle, its implementation is only limited to the executive branch (government departments, agencies, bureaus or offices) of the Philippine government. Thus, the establishment and operationalisation of the NEPF is subject to uncertainty (Senate of the Philippines, 2020c). This is especially true for when there are changes in priorities in the advent of a new government administration and for programmes led by local government units where strategic directions are heavily dependent on the “whims” of current elect administrative leaders. In addition, assessments of project and/ or programme performance are often output oriented, and capacity for conduct of results-oriented evaluation is very limited. Should there be any, they are barely documented, often inaccessible and no standardised and systematic processes are available. Overall, the practice of monitoring and evaluation is not treated equally. In most cases, only monitoring is Table 2 Breakdown of evaluations commissioned by NEDA (by evaluation type)

Table 3

Type of evaluation Summative Process Formative

13 3 1

Breakdown of evaluations commissioned by NEDA (by sector)

Sector Infrastructure Development Social Reform and Community Development Agriculture, Agrarian Reform and Natural Resources Industry Trade and Tourism Governance and Institutions Development Source Authors’ own work

6 4 4 2 1

10

EVALUATION IN THE PHILIPPINES

337

conducted and few documentations are provided on the use of evaluation results. To strengthen the conduct of evaluations of priority government initiatives under the PDP and PIP, NEDA and UNDP have partnered up to implement the “Strategic M&E Project”, also known as the Strategic Monitoring & Evaluation (M&E). The aim of the Strategic M&E Project is to accelerate the implementation of the Philippine Development Plan (PDP) 2017–2022 and to build the evaluation capacity of NEDA and select government agencies. The project has the following components: 1. managing the conduct of independent evaluations of key government themes, sectors, and/or programmes; 2. supporting the implementation of the National Evaluation Policy Framework; 3. providing learning opportunities on evaluations to NEDA and other government agencies; 4. strengthening of a community of practice on evaluations; and 5. developing an online portal of government evaluation. For the “managing the conduct of independent evaluations of key government themes, sectors, and/or programs” (p. 2) component, UNDP have started to commission independent evaluation studies on behalf of NEDA. The evaluation studies will assess the relevance, efficiency, effectiveness, sustainability, and impact (if feasible) of completed or currently being implemented government priority programmes. The end evaluation results from these studies are predicted to inform how policies and programmes should be designed and implemented to achieve the desired results of the PDP and SDGs. The studies are also expected to contribute to the development of government’s capacity to conduct evaluations by informing the implementing agencies of the M&E design, data collection systems, and best practices. The following are the programmes that have been/are currently/will be evaluated by UNDP on behalf of NEDA (UNDP, 2020a): 1. Completed evaluation: a. Anti-Red Tape Act b. Philippine Action Plan on Nutrition c. “Payapa at Masaganang Pamayanan” (PAMANA)

338

A. KALUGAMPITIYA ET AL.

2. Currently being evaluated: a. National Early Childhood Care Development Program b. National Climate Change Action Plan c. Micro Small and Medium-Term Enterprise Development Plan d. Roll on Roll off Terminal System Implementation 3. For evaluation: a. “Salintubig”—Water Supply System Project b. Social Safety Nets. Currently, UNDP has also moved to the implementation of the 2nd component of the project: “supporting the implementation of the NEPF” (UNDP, 2020b). To operationalise the framework, it was felt that a national evaluation agenda needs to be created to provide an overall policy direction and coordination of the evaluation of the public sector. As outlined in the latest NEDA-DBM Guidelines on the National Government, evaluations to be commissioned under the agenda shall focus on “themes, sectors, or programs that concern multiple government agencies; meta-evaluations of evaluations conducted by implementing agencies and other entities; and on specific priority programs that are critical to the success of the PDP and PIP” (National Economic Development Authority and Department of Budget and Management, 2020, p. 20).6 Last update with relation to the National Evaluation Agenda was in October 2020, where UNDP secured the services of an individual consultant to formulate the National Evaluation Agenda Toolkit, and demonstrate how it can be used by drafting and validating the National Evaluation Agenda for 2017–2022. The agenda will also play a critical role in accelerating the development of national evaluation capacities by helping identify key priority areas to further advance the evaluation field in the country, which will include raising awareness, building capacity, and developing of frameworks and standards.

6 https://nep.neda.gov.ph/storage/guidelines/1601872178_NEPF-Guidelines%20B ooklet-Pass%2014%20Folder_NEPF-Guidelines%20Booklet-Pass%2014.pdf. Accessed on 01 March 2023.

10

2.5

EVALUATION IN THE PHILIPPINES

339

Use of Evaluations

With DBM shifting from output to outcome-based budgeting processes, evaluation results are used for the country’s national budget formulation. To develop the PIP 2017–2022, each government agency prepared a list of priority PAPs while taking note of the development agenda of the PDP and RM and guidelines issued by NEDA in identifying priority programmes. PAPs’ that are complementary with other productive sectors and are identified with interregional and national impacts are also prioritised. These lists of priority PAPs are then submitted to NEDA for review and inclusion in the PIP (DBM, 2016). By having NEDA review the submitted PAPs, the evaluation results of each PAPs are used to identify which should be prioritised. The PAPs’ efficiency, effectiveness, relevance, and responsiveness to the PDP targets and RM outcome indicators are all taken into consideration before being included in the PIP. The PIP is then used for the national budget preparation. Through the PIP, priority PAPs help ensure that PAPs (which were initially identified by NEDA to be able to achieve sector outcomes) are accorded priority in the review of the annual budget proposal of line agencies conducted by the oversight agencies and are expected to be provided with annual allocation. While the NEPF states that all government programmes should be evaluated, and despite evaluation being accepted as an indispensable tool for effective and evidence-based decision and policy making, evaluations (their conduct and use) are never treated as a priority in planning and budgeting. As it is not mandated by law to be implemented by government institutions, the NEPF is rarely put into practice. In cases where they are practised, it is severely undocumented or inaccessible for public view. It is also widely acknowledged that government agencies do not have the capacity to conduct M&E and use evaluation results. Especially when faced with common evaluation issues such as contention against shifting baselines, activities to results attribution issues, differentiations on the levels of results and time lags between interventions and outcomes. There are conflicting views regarding the use of evaluation and of its results in the country. For the environmental/climate change sector, evaluation results are said to be used to inform government decision-making, change policy priorities, create new programmes or implementation

340

A. KALUGAMPITIYA ET AL.

models, and reallocate resources to programmes that are more cost effective or deliver higher results/impacts. For the government’s legislative branch on the other hand, legislators are still wishing for more robust, timely, evidence-based evaluation reports of government initiatives that are readily available and they could use for a more informed decisionmaking (Senate of the Philippines, 2020b). Similar issues are being faced even with the approval of the NEDA-DBM NEPF Circular where it is not institutionalised and is only applicable to the executive branch agencies. Thus, the establishment and operationalisation of the NEPF is subject to uncertainty (Senate of the Philippines, 2020c). This is especially true for when there are changes in priorities in the advent of new government administrations and for programmes led by local government units where strategic directions are heavily dependent on the “whims” of current elect administrative leaders. In 2021, in response to the requests of government agencies to have guidance to ensure the quality of government evaluations, NEDA and DBM have developed the Guidelines on Evaluation in the National Government. The guide is still on its pilot phase and is meant to be a “living document” to be updated on a regular basis in response to the context and needs of each government agency.7 More details on the guidelines are to be discussed in the Compliance with Standards and Quality Obligations section of this chapter.

3 Societal Dissemination/ Acceptance (Social System) 3.1

Institutionalised Use of Evaluations by Civil Society

There is very limited documentation on the use of evaluation and its results in the Philippines’ public sector or civil society. Awareness on evaluation is relatively low, and is often identified together with assessments or performance audits. The most well-known evaluations, however, in civil society are impact evaluations. This is because impact evaluations are often conducted to

7 https://nep.neda.gov.ph/storage/guidelines/1601872178_NEPF-Guidelines%20B ooklet-Pass%2014%20Folder_NEPF-Guidelines%20Booklet-Pass%2014.pdf. Accessed on 1 March 2023.

10

EVALUATION IN THE PHILIPPINES

341

satisfy requirements imposed by government agencies to achieve certifications, specifically the Environmental Compliance Certificate (ECC). Any development project needs to get the ECC before implementation. The ECC is also required to take other approvals from public institutions related to a project. ECC’s primary requirement is for the project proponent to prepare an Environmental Impact Statement (EIS), which in short hand, is an impact assessment or evaluation report on the environmental impacts and mitigating measures of a project. Another example is in 2015 when the Department of Environment and Natural Resources (DENR) has mandated all mining contractors in the country to secure International Standard for Organizations Certification for Environmental Management Systems (ISO, 2021), and non-compliance shall lead to the suspension of their ECCs. All organisations with ISO 14001:2015 certifications are required to have a complete procedure to monitor, measure, analyse, and evaluate their environmental performance. In both instances, the ISO certifications and the results of these evaluation studies are only seen as regulatory or compliance documents. The systems established under ISO certifications and evaluation results are rarely used by the proponent as a tool for planning or decision-making processes. If there are any, they are often undocumented or unavailable for public view. 3.2

Public Perception and Public Discourse

In the Philippines, evaluation is often perceived as a fact-checking and fault-finding tool, instead of an instrument used for the improvement of the programme or intervention being evaluated. This is most especially true for community-based interventions where there is strong political play involved in choosing who could become beneficiaries for any government initiatives and programmes. Stakeholders involved in evaluation studies, more often than not, are more biased to say only favourable findings of the programme being evaluated to avoid being persecuted or having their status as programme beneficiary revoked. And due to a lack of use of evaluation and its weak linkage in planning processes, compounded with the issue of evaluation reports and that discussions of its findings are rarely made publicly available, participation in evaluation activities is also publicly perceived as a futile effort with regards to its influence in improving programmes/projects.

342

A. KALUGAMPITIYA ET AL.

Only exceptions where evaluation results are discussed are when “controversial” projects and programmes are involved and evaluation results (or lack thereof in certain cases) are weaponised for political play and agenda settings. This weaponisation is a common tactic often used by radical environmental advocates in the country and to create heated discussions on projects with high environmental impacts in news and social media. This use of evaluation results thus further strengthens the idea of evaluation being equivalent to a persecution tool. 3.3

Participation of Civil Society in Evaluations

Due to the large population (hundred and eleven million), and as there are many disparities and enormous social issues in Philippine society, there is a large number of social development programmes implemented mainly by non-governmental or civil society organisations (CSOs).8 Their programmes are supported by external donors who emphasise monitoring and evaluation as part of the projects. Beneficiaries in the projects are usually passive participants of the monitoring data collection or evaluations. They usually participate as respondents in evaluations when they are selected as respondents. The evaluation reports produced by nongovernmental or civil society organisations are submitted to donors as a requirement and rarely shared publicly. Therefore, the beneficiaries do not see the evaluation results although they participate and provide information for evaluations. Their participation is limited to information provision rather than in key steps of an evaluation including planning, designing, analysis, and dissemination. In this light, CSOs and citizens as a whole are merely “information providers”, which is why overall, it is possible to speak of an extremely low level of participation. On June 2021, Executive Order 1389 was released, revolutionising certain functions of the executive branch of the Philippine government to its Local Government Units (LGUs). Under this order is the strengthening of monitoring and evaluation systems of the LGUs to ensure local development and land use plans remain aligned with the goals and objectives of the Philippine Development Plan. Through the devolution, M&E

8 See Porio (2017) for an overview of citizen participation in the Philippines. 9 https://www.officialgazette.gov.ph/downloads/2021/06jun/20210601-EO-138-

RRD.pdf. Accessed on 2 March 2023.

10

EVALUATION IN THE PHILIPPINES

343

capacities of the LGUs and involved government agencies will also be strengthened to ensure that the LGUs have assumed the devolved functions and services effectively. This also allows more active participation of civil society in local government programmes and initiatives, including in its evaluation processes. However, while the devolution makes government functions and initiatives transparent, consultative, and participatory, it does not guarantee effective governance. Especially when participatory processes are used as political tools to gain local dominance and electoral support. In the case of evaluations, participatory processes will not ensure that evaluation results are used to support governance, accountability, and evidence-based decision-making. 3.4

Demand for Evaluation by Civil Society Organisations

There are multiple cases where CSO leaders and programme beneficiaries often raise evaluation-related demands during consultations. Such demand includes asking programme leaders or implementers to approach their group and inquire what their community needs and reform the programme to become more relevant to its context. And that these discussions should occur before implementing a programme. This shows that despite the lack of capacity and awareness on the concept of evaluation, stakeholders still often look for avenues to provide their feedbacks with regard to the improvement of efficiency, effectiveness, and relevance of the initiatives that they are involved in. However, due to the lack of existing infrastructures to process such feedbacks from the programme beneficiaries (especially for government-led initiatives), the relay of such information is heavily dependent on the capacity and biases of the evaluator conducting the study.

4 4.1

Professionalisation (System of Professionalisation)

Academic Study Courses and Training Practices

Programmes of higher university education for evaluators do not exist in the Philippines. Closest resemblance of evaluation being discussed in academic institutions as a scientific subject only exists in the environmental sciences field. An example is an undergraduate course for environmental sciences from the Department of Environmental Science

344

A. KALUGAMPITIYA ET AL.

of Ateneo de Manila University, established in 1992 with a population of 333 students and alumni and 20 faculty members. The environmental science course offers a discourse on “Environmental Monitoring”, “Environmental Impact Assessments”, and “Environmental Risk Assessments and Management” as major subjects. However, these subject discussions are often oriented towards scientific research or urban development. Programmes from varying sectors also cover the subject of monitoring and evaluation, however, they are often integrated into the context of project planning and management. Under government initiatives, capacity-building activities are available for implementing agencies with M&E systems. While several government agencies, such as the Department of Health10 & IRRI11 have opened public training for M&E, they are not conducted on a regular basis, nor does it have a standard competency framework. Such activities are also heavily reliant on external evaluation companies or independent evaluators to provide such services. Stakeholders pointed out a need for courses on M&E. It was said that such capacity development programmes are required at various levels. There should be a specific long-time course on M&E in the University as a separate discipline. Stand-alone courses for M&E are important, especially within the Philippines, where university degrees are highly valued for professional development and recognition of evaluators as professionals. Moreover, short-term workshops should be conducted for various stakeholders to enhance capacities in evaluation. Some awareness generation programmes are also needed at the country level so that civil society is aware of the utility of evaluation-based interventions and as a result could demand evaluations. 4.2

Journals and Communication Platforms

As usual in many Asian countries, there are no academic journals on evaluation in the Philippines also. There have been scientific articles on evaluation in other academic journals but these are often inaccessible from public view due to pay walls. Publications from government agencies, on

10 https://cph.upm.edu.ph/node/508. Accessed on 2 March 2023. 11 http://education.irri.org/learnings/short-courses/science-courses/results-based-

monitoring-and-evaluation-course. Accessed on 2 March 2023.

10

EVALUATION IN THE PHILIPPINES

345

the other hand, are often treated as confidential information. None of the three VOPEs have a website or a listserve for the dissemination of information to members and the evaluation community in the country. Philippine Evaluators for Development (PHILDEV) has a Facebook page (PHILDEV, 2021) which is regularly active with updates on ongoing programmes at the global, regional, and national level. M&E Network Philippines also has a Facebook page12 where they live telecasted the annual conference. Pilipinas Monitoring and Evaluation Society (PMES) has a website (PMES, 2021) that serves as an important information and exchange platform for its professional members. 4.3

Professional Organisations

In the IOCE database, there are three existing VOPEs in the Philippines. First is the M&E Network Philippines—it’s an informal network of development practitioners composed of government, development partner agencies, academia, M&E practitioners and consultants, and other organisations doing M&E. It is established and maintained by the NEDAProject Monitoring Staff, with the support of UNICEF and is estimated to have over 150 members, a third of which are under the government sector. The network was launched with an intention to advance professionalism in the M&E community and develop a culture of results orientation in evaluation. (NEDA, 2011) The Network is also part of the collaboration between NEDA and UNDP with the conduct of the Strategic M&E Project. Since 2011, the network has been annually conducting the M&E Network Philippines Forum participated by national government agencies, development partners, M&E practitioners, and other M&E stakeholders. The forum provides an opportunity to all its participants to enhance their skills and competencies in evaluation results by providing learning sessions on M&E tools and skills. It also serves as a platform to house discussions on building and reinforcing evaluation communities to develop the evaluation culture in the Philippines. In consideration with the COVID-19 pandemic, the network held the 8th Forum live on Facebook and has extended a webinar series with provisional theme of “Navigating the New Normal: M&E in the 2020s”. 12 https://www.facebook.com/groups/MandENetworkPH/. Accessed on 2 March 2023.

346

A. KALUGAMPITIYA ET AL.

The second VOPE is the Pilipinas Monitoring and Evaluation Society (PMES). It’s a group of 65 M&E practitioners committed for the development of the M&E profession in the Philippines. Their main goal is to promote M&E capacities and competencies among practitioners and stakeholders in the country. PMES on the other hand, does not engage in any activities addressing governmental evaluation policies. Its programmes are heavily focused on capacity building among its members and development of a database of M&E professionals for more efficient collection and distribution of M&E knowledge and information. Example activities include workshops on Impact Evaluation and Qualitative and Quantitative Analysis on Monitoring and Evaluation (M&E). Its latest activity was to conduct a five-day learning workshop called “M&E Boot Camp” in 2019, to capacitate its stakeholders on RBM&E which covers the following topics: 1. Fundamentals of Results-Based Management (RBM) and Monitoring and Evaluation (M&E) 2. Developing and Implementing Results-Based M&E Systems 3. Designing, Implementing, and Managing Programme and Impact Evaluation Studies 4. Quality-Assuring and Communicating M&E and Evaluation Outputs. The third VOPE is Philippine Evaluators for Development. A professional organisation and think tank that adopts and advocates for the use of RBM&E, and aims to develop a culture of transparency, accountability, and effective performance through evaluation. PHILDEV has 44 members and is the only VOPE that is a member of the APEA. It is heavily involved in regional and international evaluation engagements. PHILDEV hosted the 2nd APEA International Evaluation Conference in 2019 Manila, Philippines, and is now currently involved in the Decade of EVALUATION for action, also known as the Eval4Action campaign (Decade of Evaluation for Action, 2020). The campaign aims to promote the use of evaluation and its critical role in mobilising key actors for the achievement of the SDGs. Through the campaign, PHILDEV also helped develop the Asia Pacific Regional Evaluation Strategy for the realisation of its goal: ‘Greater use of evaluation contributing positively to

10

EVALUATION IN THE PHILIPPINES

347

the achievement of national development goals and SDGs in Asia Pacific Region’. PHILDEV also offers a variety of evaluation-related public training courses, among others: 1. Results-based Monitoring & Evaluation 2. Integrated Strategic Planning and RBME for Public Sector Management and LGUs 3. Impact Evaluation 4. Designing and Conducting Evaluation 5. Exit Strategy & Sustainability Planning. Out of the three VOPEs, only the Philippine M&E Network has a direct connection to NEDA. PMES and PHILDEV, on the other hand, while having their own networks and initiatives in promoting evaluation, they do not have the same network and connection to NEDA as the Philippine M&E Network does. And while there may be common networks among the three, there are no direct cooperation or coordinated activities between the three VOPEs. Outside of the above-mentioned VOPEs, there are no other known organisations that include evaluation practitioners within their memberships. While informal networks within the development sector do exists, opportunities and resources available within such networks are not catered to the needs of evaluators. Although there are three VOPEs in the country and NEDA plays the designated public institution for evaluation, there is no particular programme for certification of evaluators in the country. In the absence of professionally recognised evaluators, evaluations are often conducted by freelance practitioners (including consultancy teams) who have extensive practice and relevant experience in evaluation. In general, these are practitioners who have 10–15 years of experience conducting evaluation on local and international levels. Aside from experience, there are no other pre-determined criteria commonly accepted to recognise evaluators in the Philippines. 4.4

Compliance with Standards and Quality Obligations

During the Seventh M&E Network Forum conducted by NEDA in support from UNDP, it was announced that the preparation of the

348

A. KALUGAMPITIYA ET AL.

Philippines NEPF guidelines (NEDA, 2018) which was finally published in July 2020 as “Guidelines on Evaluation in the National Government” (NEDA, 2020a). The Guidelines include technical aspects of conducting evaluations as well as a full section on quality assurance as well. The Guidelines provided a quality assurance check list which helps to check whether the evaluation is in line with set quality standards. The Organisation for Economic Cooperation and Development—Development Assistance Committee (OECD-DAC) Evaluation Criteria13 is currently the most used standard for conduct of evaluation and is integrated into the NEPF guideline. The criteria will also be used as the basis for the currently being developed bill for the establishment of the RBNEP Act, as demanded by the upper house of the congress during the first reading of the RBNEP Act. Orientation to the standards is voluntary and not mandatory, which may contribute to a rather slow establishment of the new standards. There is also no existing local certification system, arbitration board, or professorship available for the M&E field in the country to ensure the quality and competence of local evaluators.

5

Conclusions

NEDA is the designated public institution to promote and implement evaluation in the country. The Philippines has the National Evaluation Policy Framework established under the circular No: 2015-10 which was jointly issued by the NEDA and DBM for public institutions to implement the NEPF. It has been complimented by the Philippine Development Plan Results Matrices which are followed by all public institutions. The Senate of the Philippines developed the draft national evaluation policy act and submitted it to the house for approval. Once approved, the Philippines will have a legal framework for evaluation. Demand and use of evaluation in the Senate or Congress is not evident in the Philippines. However, the progressive steps in the Senate of the Philippines regarding the national evaluation policy act are commendable. The Commission on Audit conducts performance evaluations which is a new development and noteworthy to mention compared to the other three countries. 13 https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassista nce.htm. Accessed on 2 March 2023.

10

EVALUATION IN THE PHILIPPINES

349

While evaluation practice is mainly done by NEDA as it is the institution that initiates and leads evaluations, an annual evaluation plan is missing or no systematic approach of evaluating public programmes is applied. However, there are no evaluation guidelines, ethics, and standards established in the Philippines for evaluations. Currently evaluations are conducted based on the guidance provided by NEDA. Evaluations are conducted by internal and external evaluators at the public institutions whereas UN agencies, NGOs, and other organisations mostly hire external evaluators. Evaluation use is a challenge as usual in the Philippines, too. Evaluation results are used for budget formulation by ministries and departments when preparing annual budgets. Citizens are hardly aware of evaluation, and do not demand or use evaluations. Therefore, evaluation does not become a priority of politicians or officials. The large number of civil society organisations as part of projects use M&E where monitoring gets more attention than conducting evaluations. There are no academic courses on evaluation in the Philippines and FGD participants see it is a need in the country. Although there are three VOPEs in the Philippines: M&E Network Philippines, Pilipinas Monitoring and Evaluation Society, and Philippine Evaluators for Development, evaluation is yet to be established as a profession. Network is part of the NEDA and PHILDEV is a member of the Asia Pacific Evaluation Association. The three VOPEs do not coordinate with each other or do not work together on a common agenda.

References Climate Change Commission. (2011). National Climate Change Action Plan 2011–2028. http://extwprlegs1.fao.org/docs/pdf/phi152934.pdf. Accessed on 26 May 2021. Commission on Audit. (n.d.). Performance Audit Reports. https://www.coa. gov.ph/index.php/reports/performance-audit-reports. Accessed on 12 May 2021. David, J. T. (2019). Why M&E is important in government sector. Presentation. Monitoring and Evaluation Staff, NEDA. https://designingresilience.ph/wpcontent/uploads/Why-ME-is-Important-in-the-Government-Sector-NEDA. pdf. Accessed on 18 December 2020. DBM (Department of Budget Management). (2016). National Budget Memorandum No. 127 – Budget Call for FY 2018. Department of Budget Management, Philippines. https://www.scribd.com/document/348013857/ NATIONAL-BUDGET-MEMORANDUM-NO-127-Final-pdf. Accessed on

350

A. KALUGAMPITIYA ET AL.

15 April 2021. Decade of Evaluation for Action. (2020). About Decade of Evaluation for Action. https://www.eval4action.org/about. Accessed on 27 July 2021. Government of the Philippines. (1987). Executive Order No. 230 – Reorganizing the National Economic and Development Authority. Government of the Philippines. http://www.neda.gov.ph/executive-order-no-230-reorga nizing-the-national-economic-and-development-authority/. Accessed on 23 October 2020. Government of the Philippines. (2011). Administrative Order No. 25 - Creating an Inter-Agency Task Force on The Harmonization of National Government Performance Monitoring, Information and Reporting Systems. Government of the Philippines. https://www.lawphil.net/executive/ao/ao2011/ao_25_ 2011.html. Accessed on 11 November 2020. Government of the Philippines. (2016). Presidential Executive Order No. 05, S. 2016 Approving and Adopting the Ambisyon Natin 2040 as Guide for Development Planning. Government of the Philippines. https://www.officialg azette.gov.ph/downloads/2016/10oct/20161011-EO-5-RRD.pdf. Accessed on 23 October 2020. GPFE (Global Parliamentarians Forum for Evaluation). (2018). EvalColombo2018 – Responsible Parliaments: Embracing Evaluation Agenda for 2030 Report. Global Parliamentarians Forum for Evaluation. https://EvalPa rtners.org/sites/default/files/EvalColombo2018%20Report.pdf. Accessed on 3 March 2021. IIED. (2019). How the Philippines’ national M&E system integrates climate and development. International Institute for Environment and Development. https://www.iied.org/how-philippines-national-me-system-integratesclimate-development. Accessed on 16 October 2020. International Organization for Standardization (ISO). (2021). 14001–2015 Environmental Management Systems, Clause 9.1 Monitoring, Measurement, Analysis and Evaluation. National Economic Development Authority, & Department of Budget and Management. (2020). Guidelines on Evaluation in the National Government. Philippines. NEDA. (2011). NEDA forms multisectoral network on Monitoring & Evaluation (M&E). https://www.officialgazette.gov.ph/2011/11/29/nedaforms-multisectoral-network-on-monitoring-evaluation-me/. Accessed on 10 October 2020. NEDA. (2015a). AmBisyon Natin 2040, National Economic and Development Authority of the Philippines. http://2040.neda.gov.ph/. Accessed on 11 November 2020. NEDA. (2015b). NEDA and DBM Joint Memorandum Circular No.2015–01: National Evaluation Policy Framework of the Philippines. National Economic

10

EVALUATION IN THE PHILIPPINES

351

and Development Agency. https://www.neda.gov.ph/wp. Accessed on 12 June 2020. NEDA. (2018). M&E should be institutionalized in public sector – Pernia. https://www.neda.gov.ph/me-should-be-institutionalized-in-publicsector-pernia/. Accessed on 10 December 2019. NEDA. (2020a). Guidelines on Evaluation in the National Government. National Economic Development Agency and Department of Budget and Management, the Government of Philippines. https://nep.neda.gov.ph/gui delines. Accessed on 2 March 2023. NEDA. (2020b). NEDA Secretariat. https://neda.gov.ph/neda-secretariat. Accessed on 11 December 2020b. PHILDEV. (2021). Philippine Evaluators for Development Facebook page. https://www.facebook.com/phildevevaluation/. Accessed on 8 August 2021. PMES. (2021). Pilipinas Monitoring and Evaluation Society website. https:// pmes.org.ph/. Accessed on 12 July 2021. Porio, E. (2017). Citizen participation and decentralization in the Philippines. In W. Berenshot, H. Schulte Nordholt, & L. Bakker (Eds.), Citizenship and democratization in Southeast Asia (pp. 29–50). Brill. https://library.oapen. org/bitstream/handle/20.500.12657/38050/9789004329669_webready_c ontent_text.pdf?sequence=1#page=44. Accessed on 2 March 2023. Senate of the Philippines. (2019). Senate Bill No. 788: An Act Establishing a National Evaluation Policy. Senate of the Philippines. http://legacy.senate. gov.ph/lisdata/3114228042!.pdf. Accessed on 5 August 2021. Senate of the Philippines. (2020a). Senate Bill No. 1885: An Act Establishing a Result-Based National Evaluation Policy (RBNEP). Senate of the Philippines. Senate of the Philippines. (2020b). Public Hearing of the Committee on Economic Affairs joint with Finance; National Defense and Security, Peace, Unification and Reconciliation; and Ways and Means. https://fb.watch/2CH e0Acdtu/. Accessed 10 March 2021. Senate of the Philippines. (2020c). Explanatory Note – An Act Establishing a Results-Based National Evaluation Policy (RBNEP). Senate of the Philippines. http://legacy.senate.gov.ph/lisdata/3371630616!.pdf. Accessed on 3 August 2021. UNDP. (2020a). Evaluation Consultant of the NEDA-UNDP Strategic M&E Project (For Filipino Nationals Only). United Nations Development Programme. https://jobs.undp.org/cj_view_job.cfm?cur_job_id=91693&fbc lid=IwAR19jVld5wTKfTFrqVqNzvlGOQA4g4XWGDFqy9RwyHCLtJf5kJ1Y RJEdw24. Accessed on 14 October 2020a.

352

A. KALUGAMPITIYA ET AL.

UNDP. (2020b). National Evaluation Agenda Consultant for the NEDA-UNDP Strategic M&E Project. United Nations Development Programme. https:// jobs.undp.org/cj_view_job.cfm?cur_job_id=94344&fbclid=IwAR1VqcO MYw2tEF32n1WbGHb9pljYR62CDQOiC_GO2Fz3Dc6kW9dIYjR4ENc. Accessed on 14 October 2020b.

CHAPTER 11

Evaluation in Sri Lanka Asela Kalugampitiya, Soma de Silva, and Chamara Senaratna

1

General Country Overview

Sri Lanka is an island in the Indian Ocean with a population of 21.7 million. It was colonised by the Portuguese, Dutch, and British for over 450 years and finally got its independence from the British in 1948. In 1972, the country became a republic through a new constitution which made provisions to have its own executive. Until 1972, a governor appointed by the Queen of the United Kingdom was the head of the state and the (symbolic) executive. Since independence, Sri Lanka followed British parliamentary democracy until 1978 when the Executive Presidency was introduced to the country by a new constitution. There are three main ethnic groups that live in Sri Lanka: Sinhalese, Tamils, and Muslims while Sinhalese are the majority. The country faced challenges during the thirty-year civil war in the North and East, where

A. Kalugampitiya (B) · C. Senaratna University of Sri Jayewardenepura, Nugegoda, Sri Lanka e-mail: [email protected] S. de Silva UNICEF Regional Office for South Asia, Kathmandu, Nepal © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_11

353

354

A. KALUGAMPITIYA ET AL.

there was a demand for a separate country. Following the defeat of the separatist group in 2009, there were expectations for improved economic growth and social sector advancements. In 1987, Sri Lanka signed an agreement with the Government of India introducing the 13th amendment to the constitution. The 13th amendment introduced the devolution of powers between the Central Government and the provinces. Sri Lanka is one of the eight South Asian countries and a member of the South Asian Association for Regional Cooperation (SAARC). The Sri Lanka Vision 2025, adopted in September 2017, provides the strategic directions for Sri Lanka’s future intending to strengthen the democracy and reconciliation, inclusive, and equitable growth, and ensure good governance. The document recognises the need for improved monitoring and coordination to ensure effective implementation of the policies and projects. It emphasises the need to strengthen the framework and capacity for monitoring and evaluation. The vision has been modified now as Vistas of Prosperity emphasising on economic growth in the country in the next decade. Public sector-wide digitalisation is envisaged to support coordination and monitoring to achieve more efficient and transparent governance. The document explains policies for ten major areas of development. This vision provides a strategic basis for the implementation, follow-up, and review of the Sustainable Development Goals (SDGs) in Sri Lanka.

2

The Governance System in the Country

The President is the head of state and the head of government and is elected by the people for a term of five years (Article 30 (2) the constitution—The President of the Republic shall be elected by the people, and shall hold office for a term of five years). The executive power of the state is vested with the President. The Prime Minister and the Cabinet of Ministers have charge of ministries, covering key aspects of governance. The legislative powers are exercised by the Parliament, which consists of 225 elected members, and has the power to pass laws, approve the national budget and ensure governance. Administratively, Sri Lanka is divided into nine provinces. The provincial councils, introduced in the 13th amendment, which are elected by the people are the governing bodies at provincial levels. Local authorities, including municipalities, urban councils, and “Pradeshiya Sabha”, are responsible for administrating the urban and rural areas. Local authorities are elected every five

11

EVALUATION IN SRI LANKA

355

years by local government elections. The provinces are subdivided into 25 districts, with a District Secretary appointed by the central government. The District Secretariat is responsible for implementing and monitoring development projects at the district level. The next administrative level is the Divisional Secretariat, where districts are divided into a certain number of divisions. The lowest level is Grama Niladhari Division at the village level. At the local level, there is a dual system. The Divisional Secretariat sits under the central government as a form of decentralisation. The local government comprises a democratic structure of Municipal Councils, Urban Councils, and Pradeshiya Sabhas, which are under the supervision of the Department of Local Government. In addition, each district has a District Coordinating Committee (DCC), which is chaired by the District Secretary and one of the Members of Parliament (from the governing party nominated by the Government as Chair). Therefore, the leadership consists of the highestranking public official and a politician. The DCC includes heads or representatives of all public institutions of the district. The role of the DCC is to review the progress of development interventions, decide priorities, assess issues/challenges, and allocate resources. At the Divisional Secretariat levels, Divisional Coordinating Committees are in place with similar structures and the same mandate relevant to the division. Therefore, DCCs are an important mechanism in terms of monitoring and evaluation.

3 Institutional Structures and Processes (Political System) Sri Lanka has demonstrated an ongoing commitment to evaluation through its multi-faceted efforts. The years 2003 and 2015 were turning points for the Sri Lankan evaluation community. Sri Lanka drafted the National Evaluation Policy in 2003 with the request of the then Secretary to the Ministry of Plan Implementation, who was the chief guest at the SLEvA conference 2003. This provided the evaluation association to work with the Government in the promotion of evaluation. Several global and regional events in 2015, such as the International Year of Evaluation 2015 and the EvalPartners Peer to Peer programme, and the SLEvA International Conference, revived the NEP process in Sri Lanka. Evaluation in Sri Lanka both inspired and benefitted from the EvalYear 2015. There was an increase in attention to the

356

A. KALUGAMPITIYA ET AL.

national Monitoring and Evaluation (M&E) agenda, focusing on evaluation from parliamentarians and key government stakeholders. EvalYear 2015 coincided with the launch of the 2030 Agenda. It resulted in the EvalAgenda2020 that notes the formation of the Global Parliamentarians Forum for Evaluation (GPFE) and the leadership of Sri Lanka in this regard. Going back to the past, a prominent rise of evaluation activity in Sri Lanka began in the 1970s. Efforts to improve the plan and project implementation have been a feature of development efforts since the early 1970s. One unique feature in Sri Lanka was establishing a separate Ministry of Plan Implementation (MPI) charged to serve as a national focal point for Monitoring and Evaluation of all government development projects and programmes to ensure achievement of results and development effectiveness. This development arose with the public sector reforms in Sri Lanka with support from international organisations such as UNDP and ADB since early 1970. The accountability of the government for delivering results was central to the changes compared to reporting what has been completed. As the planning function was established under the Ministry of Plan Implementation in the late 1970s, planning units were established under each ministry, department, and other public institutions. The planning service was also established where planning officials were recruited and had a professional pathway. Moreover, Results Based Planning was introduced as a new development. The monitoring and evaluation emerged with the planning function as it was widely established and was well respected at the public sector. In the 1990s, the United Nations and other development partners such as the Government of Sweden started to intensify the focus on M&E and support government initiatives that emphasised results-focused monitoring and the demand for evidence-based decision-making. Technical support was provided by the Asian Development Bank (ADB) to strengthen the Post Evaluation System in the Ministry of Plan Implementation. As a result, a number of post-evaluation of projects and programmes were conducted by the MPI. The beginnings of professional evaluation emerged in the late 1990s. An important feature of the evaluation system in the country is its virtual repository and data systems. When the M&E portfolio was with the Department of Foreign Aid and Budget Monitoring (DFABM) (until 2011), there was also an initiative to develop an Evaluation Information System (EIS) to ensure “evaluation lessons are, widely disseminated and

11

EVALUATION IN SRI LANKA

357

integrated into the planning, budgeting, and policymaking process”. A web-based Project Monitoring System (ePMS) has also been a distinctive component of the Sri Lanka M&E system. Subsequently in 2010, EIS and ePMS were combined to one system which is now known as the Integrated National Development Information System (INDIS) hosted by the Department of Project Management and Monitoring (DPMM). It has a wide range of information on progress and results of development projects and useful inbuilt features (e.g., user-defined report generation, alerts). The system includes results monitoring using Logical Framework Analysis, monitoring compliance of loan covenants, tracking cash flow, captures feedback from beneficiaries and citizens. Improved data systems and evaluation capacity development are needed to build institutional capacity. 3.1

Evaluation Regulations

Evaluation is not specifically included in the constitution of Sri Lanka, and there is no specific law on evaluation passed by the parliament. However, through the Parliament Select Committee on Evaluation (PSC), the Parliament of Sri Lanka drafted the “National Evaluation of Development Interventions and Public Policies Bill”. The Bill is expected to a National Evaluation Act. PSC drafted and finalised the bill through a series of consultations. The finalised Bill should be submitted to the Cabinet for approval before sending it to the parliament, which is yet to happen. The draft Bill has proposed to establish a National Commission on Evaluation of Development Interventions and Public Policies as a body corporate by purporting (Parliament of Sri Lanka, 2019): • to improve the effectiveness and public accountability of the development interventions and public policies; • to promote a development environment that enhances equity of development results for all people; • to promote the independence, credibility, and usefulness of the evaluation functions across public authorities; • to enable public authorities to utilise evaluations to continually improve the relevance, performance, impact, sustainability, and value for money of the development interventions;

358

A. KALUGAMPITIYA ET AL.

• to ensure that credible and objective evidence from evaluations is used in planning, budgeting, organisational improvement, policy implementation, and programme and project management; and • to enhance evidence-informed decision-making through robust, credible, and independent evaluation. The draft Bill also proposes a three-year Rolling Evaluation Plan. “Every ministry shall prepare and submit evaluation plans of that ministry including evaluations selected from the evaluation plans of the public authorities of that ministry to the Department of Project Management and Monitoring (in this Act referred to as the “Department”) with a copy to the Commission on or before the date as may be determined by the Commission” (Parliament of Sri Lanka, 2019 - Draft Bill, Sect. 19). Here, the recommendation is that the development interventions and public policies be evaluated. Accordingly, every public authority is responsible to: • prepare Three-Year Rolling Evaluation Plans to evaluate the development interventions and public policies; • take necessary steps to conduct evaluations specified in such evaluation plans; • be accountable for preparing Management Responses to recommendations emanating from evaluations; • use findings and recommendations of each evaluation to prepare an Improvement Plan and implement it; and • communicate and cause to discuss evaluations to promote their utilisation. Moreover, the Government endorsed the National Evaluation Policy on 26 June 2018, and Sri Lanka is the only country in South Asia having an endorsed NEP. The National Evaluation Policy of Sri Lanka defines evaluation as “a systematic and objective assessment of policies and ongoing or completed projects, programmes, their designs, implementation, and results” (Government of Sri Lanka, 2018, p. 1). Accordingly, there are three implementations ensured by the National Evaluation Policy, namely,

11

EVALUATION IN SRI LANKA

359

1. sustainable implementation of policies, programmes, and projects; 2. efficient utilisation of resources; 3. evidence-based decision-making by incorporating lessons learnt. In implementing the National Evaluation Policy, two main institutional arrangements and responsibilities have been indicated. Among them, DPMM is the national focal point in implementing National Evaluation Policy. DPMM is the lead organisation that ensures that the evaluation function is operationalised by public institutions in line with the National Evaluation Policy. When approving the NEP, the Cabinet of Ministers gave direction to DPMM to develop a National Evaluation Policy Framework that is yet to be finalised and sent for approval by the Cabinet. In addition to that, Line Ministries, Provincial Councils, and Local Authorities are considered as part of the National Evaluation Policy implementation as stipulated in the policy. Their role is to ensure the relevant initiatives are evaluated and findings are used accordingly. In Sri Lanka, there are no sectoral laws or regulations on evaluation. However, there are sector-specific guidelines in some sectors such as HIV/AIDS and health but mainly focusing on reporting and monitoring. For example, the National STD/AIDS Programme has the National HIV Monitoring and Evaluation Plan 2017–2022, a comprehensive document with indicators, monitoring activities, planned evaluations, and relevant formats. Once every five years, the National STD/AIDS Control Programme prepares the strategic plan and the M&E plan. Therefore, a new plan emerges periodically. Similarly, the National Dengue Control Programme also has a Monitoring and Evaluation Plan. 3.2

Parliamentarian and National Audit Structures

In Sri Lanka, Parliament had several progressive steps concerning evaluation, although a long way to go in terms of demand and use of evaluation for evidence-based policy making by the Parliament. In August 2016, an Adjournment Motion was put forward in the parliament which proposed the formation of a National Evaluation Policy and system. This was a historic moment for the Sri Lankan parliament. It was quickly followed by a second Adjournment Motion in October 2016 (Trikawalagoda, 2018). This second Motion sought the allocation of funds to establish the NEP and the national evaluation system from the national budget. Both motions were submitted by a young parliamentarian from Matara

360

A. KALUGAMPITIYA ET AL.

district indicating, the interest at the local level in having an evaluation policy in place. As a result of the motions, the 2018 budget speech held in November 2017 included an “introduction to the national evaluation policy” by the Government in 2018 (Government of Sri Lanka, 2017). In 2019 the parliament appointed a Parliament Select Committee on Evaluation. Sri Lanka is the first country to establish such a committee, and it includes members from all parties represented in the parliament. The purpose of the Committee includes: (a) formulation of national policies and legislating laws in consultation with relevant line Ministries and agencies; (b) guiding and coordinating the implementing institutions and agencies at national, provincial, and local levels; (c) promoting the values of good governance and informed decisionmaking through evaluations while preventing corruption, mismanagement, and wasting; (d) examining outputs and outcomes of the existing oversight mechanism of the Parliament, mainly the functions of the Committee on Public Accounts (COPA) and the Committee on Public Enterprises (COPE), through the lenses of evaluation; (e) extending the functions of the Parliament Research Unit (PRU) to “Parliament Research and Evaluation Unit” to facilitate the legislative and oversight functions of the Parliament. In addition to drafting the “National Evaluation of Development Interventions and Public Policies Bill”, the PSC, in partnership with development organisations, rolled out a capacity-building training for Parliament Research Unit staff. The PSC issued its interim report on 20th February 2020. The parliament was dissolved in March 2020 for elections and after the elections in August 2020, the PSC has not been activated. Technically, a Select Committee has to be re-established by the new parliament. The reason for not reactivating of the PSC was that the key champions who worked hard to establish PSC could not be re-elected in the new parliament. In addition to the above, a few other structures in the Parliament play the oversight role and evaluate the government functions. Parliamentary Committee on Public Enterprises and Committee on Public

11

EVALUATION IN SRI LANKA

361

Accounts have been actively emphasising the need for timely performance audits and evaluating state institutions and development projects over the last several years (Sivagnanasothy, 2014). The Committee on Public Accounts has to examine the accounts showing the appropriation of the sums granted by Parliament to meet the public expenditure and such other accounts laid before Parliament as the Committee may think fit, with the assistance of the Auditor General (Parliament of Sri Lanka, 2019).1 A function of this Committee is to examine the Public Corporations’ accounts and any Business Undertaking vested in the Government. These two Committees have the power to summon before them and question any person, call for and examine any paper, book, record, or other documents, and to access stores and property (Parliament of Sri Lanka, 2019).2 However, they mainly focus on monetary aspects rather than results. Despite these efforts, Parliament has not given evaluation a place that can be stated as an independent discipline in itself. Parliament also does not commission or demand evaluations nor provides for any specific budget for it. Also, the Sri Lanka Parliamentarians Forum for Evaluation (SLPFE), an informal network of parliamentarians, was formed in 2016 by likeminded parliamentarians. It played a strong role within the parliament to establish the PSC and other initiatives in the parliament. However, the Parliament does not have an Evaluation Unit or it does not commission/ conduct evaluations. Parliament Research Unit has a role in providing information and briefs to parliamentarians. However, the PRU does not have a mandate to conduct or commission evaluations. The Parliament of Sri Lanka signed an agreement on 24th February 2020 with the University Grants Commission of Sri Lanka to establish a partnership with the University system to make high-quality research available for legislative purposes, which is likely to increase the use of evaluations. Such partnerships will enable universities to share research and evaluation findings with the parliament for use in making policies and laws. This is a progressive step in using evidence. However, the agreement is yet to be operationalised.

1 https://www.parliament.lk/component/committees/commitee/showCommittee?id= 210&lang=en. Accessed on 5 October 2021. 2 https://www.parliament.lk/component/committees/commitee/showCommittee? id=%209&lang=en. Accessed on 25 October 2021.

362

A. KALUGAMPITIYA ET AL.

The SLPFE was established officially in September 2016 to advocate for the use of evaluation within parliamentary processes. SLPFE was able to advocate for the establishment of the Parliamentary Select Committee on Evaluation, capacity building at the parliament, National Evaluation Bill, and organising EvalColombo2018. In addition, Sri Lanka Parliamentarians Forum for Evaluation has conducted a set of comprehensive interventions at both national and subnational level. These interventions included awareness-raising and training sessions on SDGs and evaluation for government officials, parliamentarians to improve their capacity to understand and use evaluation (Trikawalagoda, 2018). However, as the Parliament has no M&E unit or officials to support parliamentarians in using the evaluations to serve their constituencies better, use of evaluation by parliamentarians is a challenge. The Parliament Research Unit officials are not adequately trained to review an evaluation report and extract information that could be effectively used by the Parliamentarians. Sri Lanka has the National Audit Office established under the Constitution and headed by the Auditor General. The Auditor General’s primary responsibility in terms of Article 154(6) of the Constitution is to report his audit findings to Parliament. The reports that are presented to Parliament are based on detailed audit reports submitted to managements of the respective public institutions coming under the purview of Office’s audit from time to time as and when important audit findings are made.3 The scope of the National Audit Office is defined in the Public Corporations, the Finance Act, No. 38 of 1971, which includes mainly producing financial audit reports. Although performance audits were introduced in the 1990s to focus on results through evaluation, the main focus is financial aspects and evaluations are not happening. Therefore the National Audit Office does not conduct any evaluations nor produce evaluation reports. 3.3

Organisational Structure

A key feature of the Sri Lankan M&E system has been a strategically placed government body dedicated to conducting M&E within all government-led projects. This process ensures that these projects achieve their intended results and improve their effectiveness. Due to the origins 3 http://www.auditorgeneral.gov.lk/web/index.php/en/scope-of-audit. Accessed on 4 March 2023.

11

EVALUATION IN SRI LANKA

363

of M&E focus through international development agencies (Sivagnanasothy, 2014), this function was initially carried out by the Department of Foreign Aid and Budget Monitoring under the Ministry of Plan Implementation from the late 1990s/early 2000s. Since 2011, public sector monitoring and evaluation has been the responsibility of the Department of Project Management and Monitoring as DFABM renamed as DPMM. The DPMM has a key role in the National Evaluation System with a clear mission statement: “function as the National Focal Point for monitoring and evaluation of all development policies, programmes and projects of Government, to ensure results” (Trikawalagoda, 2018, p. 10). In the late 1990s, the UNDP provided much technical support to strengthen the Results-based Monitoring and Evaluation System (RBME) in Sri Lanka. This technical assistance enabled the government officials at the national and subnational levels to understand and recognise the importance of focusing on results. The DFABM pioneered the introduction and institutionalisation of Managing for Development Results (MfDR) in Sri Lanka (Sivagnanasothy, 2009). The system introduced Logical Framework Analysis and an Agency Results Framework as part of a “whole-of-government” approach covering national, sectoral, institutional, and project levels. Sector performance targets and sector indicators form an important part of the Country’s National Development Framework (Fig. 1). The MfDR initiative is linked and connected to the functions of the Office of the Auditor General. MfDR is necessary to conduct “Performance Audit” measurements, and to audit the national budget with the view to make it a performance-based budget. Development partners also supported the flow of information from line ministries and projects to the MPI/DFABM and the strengthening of electronic Information Management System (EIMS) in the Ministry of Plan Implementation. From 2010 onwards, the MfDR was linked with the annual budget process to strengthen results-based budgeting and the connection of resources to results. Currently, MfDR is hosted by the DPMM and is operational in 35 line Ministries. This system has paved the way for establishing a base for conducting evaluations systematically. Experts believed that the evaluation system could be regularised when MfDR is fully operational in line ministries and is used for evaluation purposes. It can be seen here that to streamline the evaluation system the country’s local base is being widened, which would also increase the demand for evaluations in the long run. To address these concerns, the Sri Lankan government has been focusing

364

A. KALUGAMPITIYA ET AL.

Fig. 1 System of data flow, monitoring and evaluation (Source Trikawalagoda [2018])

on strengthening e-government systems with integrated data-gathering capability. One of the challenges at ministries and departments is that there are no M&E units within them. Only a few ministries have M&E units established. The Ministry of Agriculture is an example where there is an M&E unit headed by a Deputy Director with five staff members. DPMM has a cadre inside the department but no cadre placed in ministries and departments. However, each ministry, department, province, and district has planning units, which are coming under the Department of Planning. Officials under planning units are given the M&E tasks also in ministries, departments, provinces, and districts. However, supervision of planning units is under the Department of Planning, not DPMM. Therefore, DPMM does not have a directive or supervisory role related to planning units handling M&E functions at the ministries and decentralised level. This is a hindrance for the management of the M&E function at public institutions. Planning units are generally handling monitoring activities, and evaluations are rarely handled by them.

11

3.4

EVALUATION IN SRI LANKA

365

Evaluation Practice and Use

Demand for evaluation in Sri Lanka across national and local government, society, and the private sector is emerging. This is caused by a paradigm shift from monitoring activities to results. Systematically conducting evaluations of the public programmes was first initiated by DFABM/MPI in 1994 with support from ADB. Evaluations of public programmes were directly conducted by DFABM/MPI and the evaluation reports were posted on EIS. These evaluations were planned, conducted, and disseminated by the Government of Sri Lanka, namely by the DFABM which provides technical support for the MPI. Therefore, they were countryled evaluations and that was the start of country-led evaluations in Sri Lanka. The majority of evaluations were rigorous evaluations rather than rapid evaluations. This system was successfully functioning from 1994 to around 2007, where about 50 evaluations were conducted by DFABM/ MPI, and reports were posted on EIS. The budget for the evaluation also came from the funds allocated to DFABM. In addition to the evaluations conducted by DFABM/MPI, relevant line ministries, government projects, and research institutions such as Hector Kobbekaduwa Agrarian Research and Training Institute (HARTI), also conducted evaluations and uploaded reports on EIS. Policy evaluations were mainly undertaken by the Institute of Policy Studies (IPS) which is a private sector entity specialised in policy studies. As the evaluations were government led and useful for the donors also, donors used to access relevant evaluation reports from EIS rather than duplicate the same evaluation exercises. However, conducting country-led evaluations was changed after 2007 as the MPI was abolished and DFABM came under the Ministry of Finance. In 2010, with the merger of EIS and ePMS to INDIS, the main purpose of EIS was not served. More particularly, reports of any evaluations conducted were not systematically uploaded to the system, and use of them was not highlighted. At the time of shifting to INDIS, there were about 50 evaluation reports uploaded in EIS. Since the database of EIS is not available for comparative analysis anymore, an analysis of evaluations 3ie (International Initiative for Impact Evaluation) was used to try to gain an impression of which sectors are particularly strong in Sri Lanka’s evaluation use. Of 103 reported evaluations, most took place in the health, education, social care, and financial sectors. Accordingly, sectors such as higher education or science, technology, and innovation (STI) tend to be rather weak in evaluation use.

366

A. KALUGAMPITIYA ET AL.

Sri Lanka already has important elements of a National Evaluation System, and interest in evaluation is growing (Sivagnanasothy, 2014). A strong National Evaluation System requires the engagement of the public sector, civil society, and the private sector. However, after the DFABM period, there is no systematic system of conducting evaluations of all the projects and programmes. As per the current evaluation practice in Sri Lanka, the Department of Project Management and Monitoring is mandated to undertake ongoing, ex-post, and impact evaluations of selected mega projects (countryled evaluations) (Sivagnanasothy, 2011). Evaluations of development programmes are supposed to be conducted by line Ministries, the Department of Project Management and Monitoring, and development partners. However, in reality, DPMM is doing more monitoring than evaluations. During 2017, DPMM gathered data for monitoring 1,404 ongoing development initiatives (879 projects and 525 annual programmes) implemented by 48 line Ministries. DPMM recently initiated the establishment of monitoring cells in key line Ministries. This forms a foundation and potential mechanism for a robust government-wide evaluation system in the future (Trikawalagoda, 2018). Line ministries responsible for implementing development programmes and public policy undertake selected evaluations through their respective M&E units. As in-depth evaluations are costly, it is not uncommon that government agencies undertake project reviews that are less costly and less time consuming as a substitute arrangement for evaluation. Currently, there are about ten evaluations conducted in total by DPMM and line ministries per year, excluding project reviews. Project reviews are conducted for more than 90% of projects, whether funded from the national budget or donors but not all are documented or appropriately reported. The budget for the evaluation and project reviews comes from the line ministry or the particular project. Several Parliamentarians and officials interviewed were of the view that had there been a more systematic approach to evaluation of mega projects, a number of mistakes in the development field could have been prevented and billions of rupees could have been saved. One major challenge for conducting evaluations is the non-existence of allocated budgets for evaluations. “It is the irony that while evaluation is being regulated, there are yet no budgetary provisions for conducting evaluations. It is expected that with the approval of the policy and subsequent policy framework, the allocations for evaluation will be

11

EVALUATION IN SRI LANKA

367

channeled through the state budget in coming years” (Trikawalagoda, 2018). However, as the framework is yet to come, there is no sign of budget allocations for evaluations so far at the public sector from the national budget. Approximately 30% of the capital budget in Sri Lanka is foreignfunded. Donor-driven evaluations are conducted for such projects and programmes based on the donor requirements stipulated in loan covenants. There are no specific sectors identified where evaluations are conducted, but evaluations are conducted for programmes/projects if funding is available and if it is a requirement, such as a donor request. Due to this reason, evaluations are conducted in several sectors mainly focusing on mega projects and donor-funded projects. There is no integrated approach and evaluations tend to be sporadic, scattered and not all follow up-to-date evaluation approaches. Given the absence of implementation of the national standards or guidelines for evaluation, the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) criteria and standards are applied; multiple data collection methods are applied for triangulation and validation. Peer reviews and reference groups are encouraged within and outside the government. It has been found that it is the case during DFABM, currently these are not put fully into practice and generally remain on paper. The majority of the public sector officials both at national and subnational levels are aware of the need for outcome/ impact evaluation. In spite of the awareness about outcomes and evaluations, progress reviews at present are being conducted periodically at national and subnational (Provincial and District/Divisional Coordinating Committee (DCCs)) levels to follow-up mainly physical and financial progress. At the subnational level, evaluations (which are mainly called assessment or reviews) seem to be patchy, and quite often are undertaken to assess effectiveness of a local project without necessarily understanding an “evaluative” connotation of such an activity. Many public officials are of the view that there is considerable potential to improve evaluation capacity to include participatory and rapid evaluation methods to take decisions and corrective action faster and more effectively. Internal vs. External Evaluation Evaluations are generally ad hoc evaluations in the country, they are donor driven and external. Evaluations are conducted by internal government department staff and independent evaluators under contracts. External

368

A. KALUGAMPITIYA ET AL.

evaluations under contract are conducted through foreign donor projects and programmes. Specialised evaluations are mostly outsourced to independent academic or research institutions nationally or international experts are hired. For example, the policy evaluations are conducted by IPS as they are specialised for that. The M&E units in line Ministries, which also conduct or commission evaluations, operate independently from other management, operational, and programme implementation functions, and report directly to the Head of the Line Ministry. In the case of evaluations, an independent expert evaluation team is appointed to participate in routine evaluations. The evaluation team is made up of representatives from the Department of Project Management and Monitoring, external independent sector specialists and representatives from academia and research institutions as mentioned above, the SDGs and evaluation processes are mostly happening in parallel. The terms of reference for evaluations are not standardised and do not specifically link ethnical or cultural awareness issues explicitly, for instance in case of peace-building and reconciliation programmes. Use of Evaluations As elsewhere in the world (Stockmann et al., 2020) the use of evaluation is very limited in Sri Lanka too. When the EIS was functioning and rigorous country-led evaluations were conducted, use of evaluations was ensured. All the major evaluations conducted by DFABM/MPI were posted on the system and anyone at the public sector had access to it. Once the evaluation is completed, a presentation was made to the concerned parties and the donor. This way the evaluation results were shared in addition to dissemination through the EIS. Each evaluation report accompanied a synthesis/synopsis of findings and recommendations. This was a major plus point to ensure the use of evaluations. Key decision-makers such as ministers used the evaluation findings, and they were used for project planning also. However, after the changes to the MPI and the EIS MPI merger, the practice on the use of evaluations gradually changed. Public officials are of the view that currently use of evaluation is limited or not evident at all, despite most agreeing that evaluation had the potential to improve planning, implementation, and development results. Among the main factors identified as hindering the use of evaluation are (i) small number of evaluations conducted (ii) non-implementation of

11

EVALUATION IN SRI LANKA

369

the NEP (iii) lack of evaluation capacity resulting in low-quality evaluations (iv) lack of understanding of the difference between monitoring and evaluation, (v) lack of management response to evaluations (vi) limited dissemination of evaluation findings, (vii) socio-political influences and (viii) resistance to donor-driven or independent evaluations. Overall, there is agreement on the need for systematic mechanisms to make sure evaluation findings feedback into policy and programme cycles. In addition, there is a demand to improve data quality and quantity so that evaluations can access valid data for evaluations that would meet the needs of decision-makers. It was expected that the Cabinet approval of the NEP would enable a more systematic process where findings from evaluations could be used for evidence-based decision-making by implementing agencies, NPD, DNB, and other relevant parties. A comprehensive, integrated national evaluation system will complement the Follow-Up and Review (FUR) process. However, after the approval of the NEP, the use of evaluations has not increased; obviously that is true for conducting evaluations too. The evaluation reports generated through government ministries and development partners usually target users who are expected to take action upon recommendations. It has been found that due to poor communication of evaluations and lack of access to reports, people are not aware of the evaluations and their findings. Consequently, they cannot demand evaluations. “[M]anagement responses to evaluations are identified at high-level progress review meetings chaired by a minister. In theory, the evaluation findings and recommendations are reported to the executive branch as part of the Auditor General’s annual report, tabled to the Cabinet of Ministers and discussed at the parliament” (Trikawalagoda, 2018). but [sic] in practice the response mechanisms are not sufficiently used as a management tool, but mostly seen as a formal requirement. Few evaluation reports are made public or easily [accessible] on government websites” (Kalugampititya, 2021). Ensuring the quality of evaluations is currently not yet firmly established within governmental evaluations of Sri Lanka (e.g., through mandatory regular meta-evaluations or specific competence profile of evaluators). But there is a growing awareness of the issue.

370

A. KALUGAMPITIYA ET AL.

4 4.1

Social System

Use of Evaluations by Civil Society

This section presents the information related to the use of evaluations by the civil society at various levels. These levels include the local offices of international non-governmental organisations (NGOs), large national NGOs, localised NGOs with specific foci of interest, small communitybased organisations, and the general public. There is hardly any published information available on the aspects that are addressed within this section, and therefore, interviews were conducted with key informants (KIs) who represent the academia, state administration, non-profit organisations including the CSOs, private sector organisations, civil advocacy groups, and the members of the general public to collect information that are used for this narration. Many KIs requested discretion, which precluded providing specific examples within this section. No formal assessment of utilisation of evaluations by the organised civil society in Sri Lanka has been done to date. Documented evidence on the use of evaluations by the general public also is limited. The key informant (KI) indicated that the extent of any mandatory evaluations of the projects implemented by CSOs depends on their statuses and their sources of funding. It is known that the donor agencies such as United Nations’ agencies and overseas government agencies demand and conduct evaluations of the donor-funded projects (Trikawalagoda, 2018; UNDP IEO, 2015). Some of these evaluations are available in the public domain. The extent to which these are used in any subsequent decision-making related to policy or practice is not known. The KIs indicated that the local branches of at least some of the international NGOs conduct evaluations on their projects. This is probably not a universal practice and seems to depend on the location, the nature, and the profile of the parent organisation. However, the extent of use of such evaluations for further decision-making is not known. Semi-formal evaluations or performance reports that consist of formal structured reports on the project expenditure with details of the achieved outputs and outcomes often replace formal evaluations. The KIs indicated that some large national NGOs evaluate their projects, often due to demand by funding agencies. A few of these are known to use evaluations for decision-making and sustaining the organisation, and evaluation findings significantly influence the management of these organisations. Some organisations seem to use evaluation findings to

11

EVALUATION IN SRI LANKA

371

apply for further funding and for advocacy (Trikawalagoda, 2018). Many other organisations do not use formal evaluations in decision-making as per the KIs who indicated that decision-making on future projects and activities in these depend on perceptions, predetermined and or informally identified goals, and availability of funding for earmarked project areas. Formal performance reports and financial reports on previous projects rather than formal evaluation reports are often used in funding applications. However, these organisations use evaluation reports on state and or UN-funded projects to justify their project proposals or advocacy. At more local levels, the use of evaluations by NGOs is much more limited. However, the local organisations that focus on specific areas such as nature conservation or poverty reduction use evaluations of such projects conducted by the government or other organisations to direct and justify the aims and projects of their own. However, such use is much more limited compared to large national NGOs. These local organisations are mostly responsible to their membership unless they conduct donor-funded programmes. As any evaluations within the civil society are conducted due to third-party demands rather than voluntarily to assess or improve own performance, evaluations are not used when this is not a requirement for funding. According to key informants, the use of evaluations by the general public for any decision-making or advocacy or to pressure the political and or administrative leadership is poor. For many projects of public interest, formal evaluation findings are not publicly available for the interested segments of the general public to make use of. It is not a practice in Sri Lanka to use evaluations to provide knowledge for decision-making in referenda or other political or developmental issues. The general public is mostly unaware of the concept of evaluation and the usefulness of evaluation to drive policy and practice. The KIs perceived that several factors might have led to the poor use of evaluations in the Sri Lankan society, including the lack of end-of-project evaluations and/or lack of use of any available evaluations in decisionmaking and planning. At present, Sri Lanka lacks a formal system for evaluation or country-specific standards, values, and principles that would help guide commissioning and conducting evaluations in the state and non-state sectors (Trikawalagoda, 2018). However, the KIs perceived the established and existing socio-political and development culture within the country as the most potent factor that prevents the use of evaluation. It was a common observation that the development priorities in Sri Lanka

372

A. KALUGAMPITIYA ET AL.

change with changes in the political leadership. Development projects are perceived by KIs to be of short-term in nature without adequate long-term expectations that are tied to evaluation. Many projects are perceived to be meant to satisfy the agenda of contemporary political leadership and not often being supported by evidence. While this is more relevant to the state-sponsored projects, the same was believed to be true for the non-profit and community organisations where the leadership turnover is high. In most of these organisations, the projects are tied to contemporary funding opportunities rather than long-term goals requiring sustenance. While monitoring and evaluation plans are in-built in many funding proposals, these are not implemented unless enforced by the funding agency. Although evaluations are useful for long-term selfsustenance of projects that the donors initially funded, the KIs believed that these projects are abandoned after the initial funding, which is a reason for not evaluating such projects. This low demand for evaluations and any demand being driven more by the funding requirements rather than the need for improvement are elementary factors the KIs believed to hinder the use of evaluations. Furthermore, incentives for the use of evaluations are negligible or often absent when they are not tied to funding requirements. Lack of qualified and trained M&E personnel is a limitation that the CSOs in Sri Lanka face in general (ADB, 2013). Most of the CSOs in Sri Lanka seem to lack dedicated M&E units of their own. This may be partly due to a lack of funding commitments for such units as well as the short-term nature of many projects that are not taken up for replication. Although many large organisations or local offices of larger international non-profit organisations have M&E officers (Trikawalagoda, 2018), their work is often focused on monitoring rather than evaluation. Although there are no incentives to use the end-of-project final evaluation findings in the future, even for those organisations that do conduct evaluations, the KIs perceived this to be different for mid-term and process evaluations. The findings of mid-term and process evaluations, when conducted, are often included in the final evaluation, and therefore, these are actioned on during the project implementation. The KIs perceived the lack of awareness in citizens on evaluation as a tool to inform policy and strategic decision-making to be another key factor that negatively affects the use of evaluations. This lack of awareness is perceived to be influenced by the existing socio-political culture. There is, however, a fraction of the civil society that strives to establish evaluation

11

EVALUATION IN SRI LANKA

373

culture and use it, but such efforts are believed to be negated due to indifferent attitudes of the larger body of the general public. The limitations faced in accessing available evaluation findings by interested civil society factions are believed to bar them from using these for advocacy or other decision-making. Any evaluations that are conducted by the state agencies or the UN agencies are physically scattered between these organisations, and there is no central repository that houses these evaluations. The MOU signed between the University Grants Commission and the Parliament of Sri Lanka in 2020 to provide data generated from the academic establishment to use in decision-making was a progressive action. However, most of this information is likely to be research data rather than evaluation findings. Nevertheless, this collaboration provides a platform that can be used in the future to share evaluation findings and use them in decision-making, where the Parliament Research Unit can act as a central repository for evaluation reports. Such a central repository will have the potential to encourage civil society members that are interested in using evaluations to do so, especially if it is available to them online. A rudimentary online system had been set up several years ago (Sivagnanasothy, 2007), solely driven by the personal interests of the evaluation champions in the state sector. However, due to various reasons, including the logistical problems and lack of political and other support, have seemingly led to the demise of this initiative. 4.2

Public Perception and Discourse on Evaluation and Evaluation Results

The KIs indicated that the concept of evaluation has never been in the mainstream discussions in Sri Lanka, being limited to the dialogues within the fora of academics, professionals, and administrators. As a result, the discourse on evaluation, evaluation results that are available to use, and the use of such results has been confined to these groups to date rather than occurring in the civil society at large. The KIs believed that the perceptions of these professional groups and institutions have been favourable for the promotion of evaluation in the country. These groups seem to acknowledge evaluation and the use of evaluation as a requirement to better develop the country and to optimise the resource utilisation within this process. The dialogues on evaluation within these groups is seemingly focused on promoting evaluation among key stakeholders including the government, parliament, and development agencies

374

A. KALUGAMPITIYA ET AL.

including the civil society organisations rather than generating an interest and a demand for evaluation within the general public. This perceived top–bottom approach for promoting evaluation has resulted in attempts to mainstream evaluation by pushing for a national evaluation agenda, including a national policy on evaluation and a draft national evaluation bill, while most of the general public remains ignorant about the concept of evaluation itself. The Sri Lanka Evaluation Association has been at the forefront of promoting evaluation culture in Sri Lanka. However, as per the KIs, CSOs other than SLEvA have mixed perceptions of evaluation. Some organisations welcome evaluation to uptake it as a routine practice while others consider it an unnecessary burden, the latter being due to two reasons. Firstly, the financial cost of evaluation is significant, and some feel that these funds are better spent on the intervention itself as any findings of the evaluations are perceived as unlikely to be used in future decisionmaking. Secondly, there is no requirement or a demand for evaluation, and there are no incentives for conducting evaluations for programmes that are unlikely to be sustainable. However, the KIs indicated that some large organisations welcome evaluation as a necessary process that facilitates their institutional development and sustainability of the organisation itself. Due to the actions taken over the years by the academia, SLEvA, and the Sri Lanka Parliamentarians’ Forum for Evaluation, there is currently a renewed interest in evaluation, at least in some professional groups such as university teachers, some state sector officers in the central and provincial governments, some members of the private and non-profit sectors, and even school teachers who teach/are interested in development studies. The KIs are aware of ongoing discussions on how the evaluation can be institutionalised best and be promoted as an integral part of the development agenda. Partly due to the lack of any rigorous and directed action targeting awareness creation in the general public, penetration of these concepts into the general public are perceived by the KIs to be very low. As a result, any discussions that are on advancing the evaluation culture in Sri Lanka remains confined to the academic and professional groups, without any significant recognition, contribution, or support from the rest of the civil society in general. There is hardly any mass-media coverage on the importance of evaluations or on any evaluation findings which contribute directly to this passivity seen within the civil society. Some KIs attributed this lack of interest in the media on evaluations to perceived

11

EVALUATION IN SRI LANKA

375

political agenda that the mainstream media has and to the lack of public interest which reciprocally and negatively affect the mass-media’s interest on evaluations. Evaluation reports are largely not available online and accordingly cannot be accessed by citizens and civil society actors, making it difficult for them to be the starting point for discussions. 4.3

Participation of Civil Society in Evaluations

The KIs perceived the participation of the civil society in evaluations to be minimal. Sri Lanka has seen abundant development interventions since it became a dominion of the UK in 1948. The number of such projects grew after its full independence in 1972, some of these initiatives being statefunded while most being funded by external development agencies and donor states. Due to the donor-funded nature of many of these projects, the evaluations are often mandatorily conducted, and the civil society is involved as beneficiaries or as relevant other stakeholders (ADB, 2013). Their involvement is mostly at pre-project consultations, where the representatives of the community groups participate. They also provide data for mid-term and end-of-project evaluations. Despite participating in consultative meetings and contributing to evaluations, the general public of the country as well as those segments of the public that are directly affected by the development interventions show little or no interest in the evaluation findings and/or how these findings are used. Perceived reasons for this apathy and low involvement of the general public were few. Firstly, despite being consulted on the development intervention, their opinions are rarely considered by the implementing agencies. Secondly, any action based on evaluation findings that are relevant to the general public are rarely taken. As a result, any influence of evaluation on them is likely to be very low, if any. Thirdly, the awareness about the importance of evaluation is limited among the general public. Despite such shortcomings, some civil organisations use available evaluation findings in their subsequent advocacy and community mobilisations. 4.4

Demand for Evaluation Within Civil Society

The KIs indicated that similar to the perception and discourse, the demand for evaluation has also remained relatively high in the fora of

376

A. KALUGAMPITIYA ET AL.

academics and professionals, administrators and some institutions, rather than taking root in the wider civil society. The demand for evaluation within the civil society organisation and the general public in Sri Lanka is very low. However, the interest in evaluation in Sri Lanka is growing, at least in some groups (Trikawalagoda, 2018). The KIs of the academia and the non-profit sector believed that the demand for evaluation of success or failure of major development projects has risen among the academia and professional groups over the last decade but only slightly among the general public. Critical appraisal of the development projects and financial accountability has emerged within political agenda, although the term evaluation itself was not used in this context. At present, several groups such as professional organisations, universities, private sector actors, nonprofit organisation, and some international organisations are striving to drive a demand for evaluation in Sri Lanka to higher levels through advocacy and creating a facilitating environment (Trikawalagoda, 2018; UNDP IEO, 2015). Whenever there is a demand for evaluation within the civil society, the KIs perceived it to be due to reasons other than to use for improving practice. Most evaluations are demanded as donor requirements and any appreciation of evaluation beyond satisfying this requirement is minimal. This is the case for most of the development projects that had been funded by the UN agencies and the Western governments. Evaluations are seemingly rarely used in decision-making except in the case of some large national organisations and local offices of large international organisations. Some other organisations demand evaluation as a mean to make funding applications strong to ensure retention of donor funding. Rarely, it may be used to satisfy various stakeholders of the intervention. A key factor that is instrumental in keeping the demand for evaluation low, as indicated by the KIs, is the general lack of awareness about the evaluation concept itself. Even when the concept and the process are known, the importance of evaluation is not well comprehended by those who should ideally demand evaluation, partly driven by the lack of understanding of the usefulness of evaluations. Furthermore, despite the monitoring of financial accounts of the registered NGOs by the state (ADB, 2013), there is no legal or other requirement for the NGOs to formally evaluate any interventions or projects, and whatever legal procedures that are currently relevant to the civil society are aimed at

11

EVALUATION IN SRI LANKA

377

maintaining control of the state over their activities rather than on technical monitoring and evaluation of their interventions (IPPA, n.d.). Given that evaluation is not a requirement or does not provide any tangible and comprehensible benefits to CSOs, it is often considered an unnecessary burden. Even when it is considered useful, resource constraints, both human and financial, remain barriers for any uptake. The KIs indicated that the low demand for evaluation in CSOs is also due to the turnover of the leaders involved in the CSOs, with which the programmes and the general agenda change often, making any use of evaluations redundant.

5

Professionalisation of Evaluation

Although during the last 30 years evaluation has established itself as a field of growing interest, evaluation is not yet used to serve its critical purpose of contributing to deliver development results to people effectively, efficiently, sustainably, coherently, and impactfully. Arguably, one important reason is that evaluation is not yet professionalised adequately to institutionalise it across the government as an integral function. This situation results from the non-recognition of evaluation as a profession and the absence of academically trained and experienced evaluators. The professionals who undertake evaluations need to be well trained and experienced and possess expertise. Lack of professionals and inadequate systems have hindered specifications of quality and their enforcement as a national requirement (Mehrotra, 2013). A theory of change for establishing professionalisation could include three factors as immediate contributing results: (i) availability of academic training, (ii) mechanisms for professional networking (iii) requirement for standards and obligations. It is logical to assume that these conditions together would lead to professionalisation. The theory could be further developed and validated. This section provides an overview of the situation of professionalisation in Sri Lanka and outlines the status of these three underlying causal factors. 5.1

Academic Study Courses and Training Practices

Integral to the professionalisation of any discipline is academic training. In line with this principle, the University of Sri Jayewardenepura established a postgraduate diploma in monitoring and evaluation in 2018. The diploma programme was coordinated by the Department of Community Medicine of the Faculty of Medical Sciences. The establishment of

378

A. KALUGAMPITIYA ET AL.

this programme was a result of an effort to establish evaluation training “Teaching Evaluation in South Asia” under a collaboration of UNICEF and several tertiary education institutions in South Asia. The programme envisioned a steady supply of evaluation professionals leading to the professionalisation of evaluation. This postgraduate diploma in evaluation is the first tertiary level evaluation training programme in Sri Lanka, possibly in Asia. It is a long-awaited intervention. A potential underlying reason for the delay is the negative perceptions of evaluation. For example, evaluation is an external requirement of funders, evaluation uncovers weaknesses in programmes thus causing unnecessary problems, and evaluation is costly and time consuming. Evaluation was not seen as an effective tool for development success. It was not a tool in the toolkit of programme managers. Hence, there was no demand for an evaluation profession. The global and regional level advocacy that emerged in the recent decades, especially from United Nations Agencies, enabled national champions to pioneer this course. The diploma programme attracted students indicating a felt demand. A total of 60 students have participated in its three years of implementation. The student profile ranges from middle to senior-level professionals in various government and non-government sectors and academia. To serve the intended purpose of the course, it is necessary to expand its reach in the public sector both at the national and provincial levels. A targeted advocacy programme is necessary to motivate the public sector and private sector decision-makers to use the opportunity. Building relationships with key private sector enterprises could be one strategy. Popularising the diploma programme necessarily calls for enhancing its quality and relevance. The university has taken steps to improve the teaching. A key component of the programme is the practicum, where the students carry out a small-scale real-life evaluation to demonstrate the ability to apply their theoretical knowledge. The university is currently taking action to introduce a Master’s course. A gradual expansion to other universities would strengthen the supply of academic training. The University of Kelaniya has already established a Master’s degree, which, however, is not continued. Nonetheless, it is important to evaluate these courses and improve their quality and relevance to progressively higher academic standards. Recognising the need for evaluation knowledge among development professionals, several training institutes have introduced evaluation as

11

EVALUATION IN SRI LANKA

379

modules. This inclusion of evaluation modules in other academic streams is a common practice and is the main means of teaching evaluation, for example, in Europe (Stockmann et al., 2020). These institutes include the Postgraduate Institute of Management and the Sri Lanka Institute for Development Administration. The University of Jayewardenapura recently introduced a module on evaluation to the Master’s course on forestry and environmental science. Sri Lanka has a history of over 30 years in providing non-academic short training programmes within the field of evaluation. SLEvA, since its inception, has offered training to its members and interested others. The pre-conference workshops conducted by internationally renowned resource persons at SLEvA’s biennial conferences served as special occasions for non-academic training. Such workshops have been well attended and well appreciated. In more recent years, the training has become more focused and results-oriented. SLEvA, in collaboration with the UNICEF country office and the regional office for South Asia, implemented a much-needed comprehensive training on managing evaluations for public officials (IDEA International, 2018). The training offered first a series of online events which was followed by supplementary face-toface training. A core group of professionals participated in most training sessions forming a potential resource group for future training. The major obstacle to enhancing the quality and expanding the reach of evaluation training is the dearth of teachers. International collaboration to obtain the services of teachers for a few years until Sri Lanka could establish its own pool of resources to teach at the tertiary level seems like one effective strategy. Such a strategy has been used successfully, for example, to introduce population studies at the university level globally and in Sri Lanka. 5.2

Journals and Communication Platforms

A professional field must necessarily be supported by communication and networking platforms. A common medium of communication is journals. A tradition of evaluation journals is yet to be established in Sri Lanka. Many constraints stand in the way of producing a regular evaluation journal. A major constraint is resources which include an organisation that can undertake this responsibility. Such an organisation needs personnel such as editors, reviewers, designers, and financial resources for printing. The overall level of development of evaluation in the country is not ready

380

A. KALUGAMPITIYA ET AL.

for such an undertaking. SLEvA, the most appropriate organisation to spearhead the publication of a journal, would probably do so in time to come. Moreover, there are no other academic journals in other disciplines that deal with evaluation. Although a journal of evaluation is not published, Sri Lanka has used technology-based communication platforms for knowledge-sharing and awareness-raising. SLEvA has launched a series of webinars starting in November 2019. The characteristic feature of these webinars is that they are organised in a highly collaborative and participatory manner. The collaborating partners have included local partners such as the Center for Evaluation at Jayawardenepura University and many regional and global partners such as UN agencies and professional evaluation bodies. The collaborative approach has helped bring hundreds of senior evaluation professionals, practitioners, students, and others interested in the field. The purpose of these webinars has been twofold. First, they were designed to promote the theory and practice of evaluation. Some examples are outcome mapping, theory of change, and evaluation design. Second, the webinars aimed to draw attention to issues and challenges in evaluation and advocate for its use. In this line, Marco Segone, Director Evaluation of UNFPA, presented “Strengthening national evaluation capacities to demand, supply and use good quality evaluations for national policy-making”. Other timely topics examined theoretical and practical issues in monitoring and evaluation. A significant gap in communication that needs to be addressed is effective communication of evaluations of public institutions to central authorities, who then need to use the findings and recommendations in approving development plans and budgets. They are also not communicated to the Parliament and relevant committees for informed decisionmaking and actions. It has been observed that strong vertical and horizontal coordination is vital through the operationalisation of Sustainable Goals architecture (Trikawalagoda, 2018). Feedback loops that enable beneficiaries to use and benefit from evaluations are not developed, although evaluation findings are sometimes disseminated. Such feedback mechanisms are necessary to ensure transparency and accountability of development programmes (Trikawalagoda, 2018). Evaluation findings and management responses are expected to be presented to the executive making room for promoting the use of evaluations. However, this process is yet to be realised in practice (Trikawalagoda, 2018). Not all evaluations are made public or easily

11

EVALUATION IN SRI LANKA

381

accessible (UNDP IEO, 2015). A mechanism to communicate evaluations and their findings with stakeholders and enabling access to the public as is practised in some progressive countries (e.g., South Africa) and recommended internationally (UNEG, 2016) could be a step forward. A particularly important stakeholder is civil society. Civil society has a critical role to play and is a resource in addressing a wide range of issues in development and policy (Segone et al., n.d). For them to play this role, access to evaluation findings and opportunities to feed into evaluation systems is vital. 5.3

Professional Organisations

Professional organisations are spaces in which a profession can put down its roots and grow. They are the gatekeepers of the profession. Sri Lanka, with good foresight, established the Sri Lanka Evaluation Association as far back as 1999. It is the country’s voluntary organisation for professional evaluation. Starting with a small membership of about 30 participants, SLEvA has developed a firm position of promoting evaluation. SLEvA’s flagship programmes of biennial international conferences and accompanying professional development workshops enabled building up relationships with leading evaluators and institutions. SLEvA thereby became known and established internationally. A catalytic factor has been the cooperation of UN agencies. UNICEF played a key role in initiating discussions on the feasibility and need for an evaluation association which was unanimously accepted. Subsequently, UNICEF and many partners, especially UNDP provided critical input to promote national efforts, advocated for evaluation, and provided technical support. SLEvA has taken advantage of the recent global and regional developments in the field of evaluation. Two such influential developments are promoting VOPEs, particularly by the International Organisation for Collaboration in Evaluation (IOCE), and many initiatives such as the Year of Evaluation 2015 and its offshoot EvalAgenda 2016–2020. The role of evaluation in achieving SDGs added much impetus to these initiatives. Consequently, many new evaluation networks and activities have emerged, for example, EvalYouth, EvalGender+, EVALSDGs. SLEvA participated, coordinated, and led some of these initiatives. Young and Emerging Evaluators is one such particularly innovative intervention. SLEvA’s activities expanded its outreach to regional and global organisations and audiences.

382

A. KALUGAMPITIYA ET AL.

An overarching strategy of SLEvA, especially in recent years, has been the collaboration with a broad spectrum of influential stakeholders. The key partners included academic institutions such as the University of Sri Jayewardenepura, voluntary professional organisations such as Asia Pacific Evaluation Association, and the South Asia Community of Evaluators. Partnerships were also fostered with United Nations agencies such as UNICEF, UNFPA, UNDP, and social networks such as EvalYouth Global Network. Its collaboration with the Sri Lanka Parliamentarians Forum led to the hallmark event EvalColombo2018, which brought together evaluation champions from over 70 countries. A challenging situation of these initiatives is that they take off with much enthusiasm, but are not sustained. This wilting away of well-intentioned initiatives is, as we have observed, mainly due to a lack of demand and also due to a lack of resources to maintain the impetus, especially of champions. Collaboration with stakeholders has enabled the evaluation community in Sri Lanka to spearhead professionalisation. Of particular significance is the development of an evaluation competency framework to guide the professionalisation of evaluation. Equally progressive is the decision to publish peer-reviewed academic and training publications. A workbook on the theory of change would be launched at the National Evaluation Week to be held in 2021. SLEvA has thus seen a significant expansion of its work, both aimed at professionalisation of evaluation and more generally promoting networking. Besides the recent renaissance in evaluation, SLEvA’s catalytic leadership has been a success factor. The current president of SLEvA is also the Asia Pacific Evaluation Association (APEA) president, thus creating a good symbiosis for the growth of evaluation in the region. He is well connected to the global networks, which is a position that he takes advantage of to push the evaluation agenda and SLEvA’s agenda forward. A third facilitating factor would be the active engagement of the membership that has developed to be more informed, encouraged, and motivated. Other professional and academic organisations do not include evaluation units.4

4 Various SLEvA reports have been used for this section. SLEvA (2020–2022). Sri Lanka Evaluation Association Quarterly Reports, various issues. First issue published February 2020. 12th issue published November 2022. https://www.sleva.lk/index.php/ publications. Accessed 14 March 2023.

11

5.4

EVALUATION IN SRI LANKA

383

Compliance with Quality Standards and Obligations

Quality standards, as noted earlier, and compliance mechanisms are essential components of an enabling environment for the professionalisation of evaluation. Standards guide professionalism, and compliance ensures the quality of the evaluation process and products that facilitate evaluation use. Policy and legislative frameworks specify quality standards and engender compliance. Sri Lanka has taken significant actions to advance its policy and legislative support to the evaluation function by approving a national evaluation policy and drafting a national evaluation bill. Thus, policy implementation capacity, lack of agreed-upon standards, and mandatory systems have hindered the development or adaptation of quality standards and implementation. For example, gender, ethnic, and equity considerations are not specifically recognised in evaluations as yet (UNDP IEO, 2015). Nationally accepted, quality standards for implementing evaluations, communicating, and utilising findings are yet to be formulated. Such systems need experienced professionals to champion their development and institutionalisation. In Sri Lanka, there is no certification system for evaluators and no arbitration board established yet. Hence, professionalisation of evaluation is an urgent priority in Sri Lanka to enable evaluations of development interventions to deliver results to people.

6

Conclusion

In conclusion, Sri Lanka was the first country in South Asia to develop a National Evaluation Policy (UNDP IEO, 2015). Development of the NEP, initiated by SLEvA and subsequently supported by the Sri Lanka Parliamentarians Forum, and finalised by the Department of Project Management and Monitoring came into effect in 2018 (DPMM, 2018). Currently, a policy implementation framework is under development. The implementation of the policy is yet to come into force. Among the issues that hinder the implementation of the NEP is the absence of a national evaluation plan which identifies priority evaluations to inform government decisions in key areas of development. Equally pressing is the lack of implementation capacity. Implementation requires dedicated management structures, processes, drive, and championship. More importantly, the implementation of evaluation plans requires professional capacities that are currently grossly inadequate.

384

A. KALUGAMPITIYA ET AL.

We assume that Sri Lanka needs a legal framework as a driver for evaluation although certain knowledge-based countries have achieved a relatively advanced evaluation practice without enforcing legislature, for example Denmark (Stockmann et al., 2020). Accepting the need to provide a legal framework to strengthen the evaluation function, the Parliament of Sri Lanka established a Standing Committee to review the evaluation capacities and make suggestions for improvement. The Parliamentary Standing Committee drafted a National Evaluation Bill (Parliament of Sri Lanka, 2019), which laid down a procedure for developing national and subnational evaluation plans and mechanisms for their implementation up to specified standards. Although handed over to the Parliament in November 2019, the bill is not yet tabled in Parliament. The dissipation of the members of the Standing Committee that followed the change of government in 2020 has interrupted the momentum of the legislative process. Equally detrimental was the defeat of the political party from which most of the membership of the Sri Lanka Parliamentarians Forum (SLPF) was constituted. The SLPF’s objective was to advance the evaluation processes to develop an enabling environment for achieving development results more effectively (Rosenstein, 2015). However, the political climate was not yet attuned to be supportive of such efforts. Although DPMM is the designated public institution for M&E, it is a question whether they have a clear mandate for evaluation and whether there is a necessary capacity to perform evaluations across the public sector. The number of evaluations currently conducted particularly by the public sector is less compared to the number of initiatives in place. As usual in many countries, monitoring takes the larger portion in M&E. Not having dedicated financial resources for evaluations and capacity building on evaluation is a major challenge faced by the country. This may also be a result of a lack of demand for and use of evaluations as citizens are not aware of the importance of it. The demand for evaluative evidence particularly from the public would urge the government to allocate more resources as well as conduct evaluations regularly. It shows that civil society organisations also conduct or use evaluations where it is mandatory rather than doing it with ownership. Sri Lanka has taken a major step forward in professionalising evaluation by mainstreaming it in postgraduate education. Yet, the challenges in moving forward are many and towering. Progressive development of the academic courses suffers from the dearth of academically qualified

11

EVALUATION IN SRI LANKA

385

teachers. A programme is needed to obtain these resources from outside the country until Sri Lankan students qualify to teach. Professionalising evaluation is not only producing qualified evaluators. There needs to be recognition of the role of evaluation in the government’s development functions which is necessary to mainstream evaluation as an integral function. It is necessary to support both academic training and capacity development of the relevant public sector institutions to utilise evaluation to achieve development results. SLEvA has been there for over 20 years and implementing many initiatives to promote evaluation although there is a long way to go in terms of what should be in place to institutionalise evaluation. Sri Lanka is yet to develop national standards, guidelines, and capacity-building strategies so that concerned officials are able to produce quality evaluations. These instruments will help to conduct evaluations regularly and encourage use for decision-making which needs to be improved a lot at the moment. The Government of Sri Lanka, the SLEvA, and other actors need to pay further attention to the missing factors and address them meaningfully to establish a national evaluation system towards achieving SDGs in the country.

References ADB. (2013). Civil Society Briefs: Sri Lanka. 2013 Asian Development Bank. https://www.adb.org/sites/default/files/publication/31161/csb-srilanka.pdf. Accessed on 6 March 2023. DPMM. (2018). The National Evaluation Policy. Department of Project Management and Monitoring. Government of Sri Lanka. (2017). Budget Speech Nov 2017, section 256. Ministry of Finance, The Government of Sri Lanka. http://www.treasury. gov.lk/documents/10181/28027/Budget+Speech+2017/a092d16a-68ec46d5-8e27-9e3c00243aa8. Accessed on 28 February 2019. Government of Sri Lanka. (2018). National Evaluation of Sri Lanka. Colombo, Sri Jayewardenepura: Sri Lanka. IDEA International. (2018). Mapping Exercise on UNICEF Support to National Evaluation Capacity Development. A report submitted to UNICEF Evaluation Office. IPPA. (n.d.). Using the policy instruments of NGOs to promote locallyled development and governance in Sri Lanka. International Public Policy Association. https://www.ippapublicpolicy.org/file/paper/5b0f770a0 39d8.pdf. Accessed on 6 March 2023.

386

A. KALUGAMPITIYA ET AL.

Kalugampititya, A. (2021). Critical factors for institutionalizing evaluation at national level: Study on four countries in Asia- Sri Lanka, Nepal, Philippines and Bangladesh (PhD Thesis). Department of Sociology, Saarland University. Mehrotra, S. (2013). Monitoring, evaluation and performance management in South Asia: The challenge of building capacity. Evaluation, 19(1), 74–84. Parliament of Sri Lanka. (2019). Draft Bill on National Evaluation of Development Interventions and Public Policies, 2019. The Parliament of the Socialist Democratic Republic of Sri Lanka (unpublished). Rosenstein, B. (2015). Mapping the Status of National Evaluation Policies. (2nd ed.). https://globalparliamentarianforum.files.wordpress.com/2016/02/thestatus-of-evaluation-policies.pdf. Accessed on 7 January 2020. Segone, M., De Silva, S., Saunders, E., & Sniukaite, I. (2013). The role of civil society in equity-focussed and gender-responsive country-led Evaluation Systems. In M. Segone, & J. Rugh (Eds.), Evaluation and civil society: Stakeholders perspectives on national evaluation capacity development. Sivagnanasothy, V. (2007). Institutionalization of Monitoring and Evaluation System in Sri Lanka. Lessons Learnt, Best Practices, Issues, Challenges and the Way Forward. Department of Foreign Aid and Budget Monitoring, Ministry of Plan Implementation, Sri Lanka. Sivagnanasothy, V. (2009). Sri Lanka National M&E System. Proceedings from the National Evaluation Capacities International Conference. UNDP IEO. Sivagnanasothy, V. (2011). Use of evaluation in public policy. Proceedings from the Second International Conference on National Evaluation Capacities. UNDP IEO. Sivagnanasothy, V. (2014). Country-led national M&E system. Proceedings from the Third International Conference on National Evaluation Capacities. UNDP. Stockmann, R., Meyer, W., & Taube, L. (Eds.). (2020). The institutionalisation of evaluation in Europe. Palgrave. Trikawalagoda, P. (2018). Review of national evaluation systems and capacities for evaluating progress towards the sustainable development goals. Case study of Sri Lanka. UNDP IEO. (2015). Towards a baseline study: Insights on national evaluation capacities in 43 countries. Independent Evaluation Office of the United Nations Development Programme. UNEG. (2016). Norms and Standards for Evaluation. United Nations Evaluation Group (UNEG). http://www.unevaluation.org/document/detail/1914. Accessed on 19 July 2020.

CHAPTER 12

Evaluation in Taiwan Chi-Yeh Yung, Jan Fell, and Jiann-Jong Guo

1

General Country Overview

Taiwan (臺灣), historically known as Formosa,1 is a self-ruled island situated in East Asia circa 160 kilometres across the Taiwan Strait from the south-eastern coast of Mainland China (Government Information Office, 2011). In 2019 the population of Taiwan stood at 23.6 million. Taiwan proper and its outlying islands expand to an area of 36,197 km2 , with most of the population located in cities and counties on the densely 1 Portuguese sailors passed Taiwan in 1542 on their ship. Due to its beautiful landscape as seen from the sea they named the island “Ilha Formosa,” meaning “beautiful island” in Portuguese. Taiwan was commonly known as Formosa in the Western world until after World War II (Heylen, 2012).

C.-Y. Yung (B) · J. Fell Taiwan Assessment and Evaluation Association, New Taipei, Taiwan e-mail: [email protected] J. Fell e-mail: [email protected] J.-J. Guo Joint Credit Information Centre, New Taipei, Taiwan e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_12

387

388

C.-Y. YUNG ET AL.

populated Northern and Western coastal plains. Roughly 97% of the population are of Han Chinese ethnicity, 2% belong to a diverse number of indigenous tribes, and 1% are of foreign or other minority descent (Executive Yuan, 2016).2 The official language is Mandarin Chinese with traditional Chinese characters as the writing script. With the promulgation of the Development of National Languages Act (國家語言發展 法) in 2019, Taiwanese Minnan and Hakka—two dialect varieties of Chinese—also enjoy legal recognition together with various Austronesian indigenous languages and sign language. The gross domestic product per capita came to 28,700 US$ in 2019 (Directorate-General of Budget, 2020). In 2018, the life expectancy stood at 84 and 77 years for women and men respectively (Directorate-General of Budget, 2020). A singlepayer system of compulsory socialised healthcare, the National Health Insurance (全民健康保險), was implemented in 1995 with a positive effect on life expectancy and profoundly lessened healthcare disparities for lower income groups (Wen et al., 2008). Initially fuelled by United States aid receipts in the aftermath of World War II, the island’s economy underwent a significant transformation (Wang, 2015). First from an agrarian economy to an exporter of textiles and plastics in the 1970s (Ranis, 2002), then to a manufacturing hub of heavy industries and consumer electronics in the 1980s (Gold, 2015), and finally since the 1990s into one of the world’s most technologically advanced computer chip makers (Economist, 2018). After decades of post-war growth, Taiwan was counted among the Four Asian Tigers3 and joined the Asia–Pacific Economic Cooperation (APEC) forum in 1991 (Hsueh, 2006), as well as the World Trade Organization (WTO) in 2002 (World Trade Organization, 2017). Today, Taiwan is included in the advanced economies group by the International Monetary Fund (International Monetary Fund, 2021) and high-income economies group by the World Bank (World Bank, 2018). The island’s capital city, Taipei (臺北市), has a population of 2.6 million—or 7.1 million if including

2 Austronesian tribes are the original inhabitants of Taiwan with a history dating back to at least 3,000 B.C.; Han Chinese first settled in Taiwan in the seventeenth Century, with major waves occurring in the nineteenth Century during the Qing Dynasty and the aftermath of the Chinese Civil War in 1949. 3 The economies of Taiwan, South Korea, Hong Kong, and Singapore recorded extraordinary economic growth in the second half of the twentieth Century and came to be known as the Four Asian Tigers (Van den Berg & Lewer, 2015).

12

EVALUATION IN TAIWAN

389

the Greater Taipei metropolitan area (Directorate-General of Budget, 2020)—and is considered an Alpha-level world city (Globalization & World Cities Research Network, 2020). Having emerged from an authoritarian legacy of martial law in force between 1947 and 1987, today Taiwan is a multi-party democracy and classified as “free” by the non-governmental organisation Freedom House. In 2009, Taiwan ratified two major human rights instruments, the International Covenant on Civil and Political Rights (ICCPR) and the International Covenant on Economic, Social and Cultural Rights (ICESCR) (Chen, 2019a). In this vein, the 2021 Freedom in the World report describes the situation as follows: “Taiwan’s vibrant and competitive democratic system has allowed three peaceful transfers of power between rival parties since 2000, and protections for civil liberties are generally robust” (Freedom House, 2021). The two major political parties are the socially liberal centre-left Democratic Progressive Party (DPP; 民主進步黨) which had been in power between 2000 and 2008 and has again assumed power in 2016 under President Tsai Ing-wen (蔡 英文), as well as the socially conservative centre-right Chinese Nationalist Party also known as Kuomintang (KMT; 中國國民黨), which had been the ruling party until the year 2000 and between 2008 and 2016. There also exist other smaller political parties, the most noteworthy of which are the New Power Party (NPP; 時代力量) and the Taiwan People’s Party (TPP; 台灣民眾黨), the latter of which is chaired by Ko Wen-je (柯文哲) the current mayor of Taipei City in office since 2014. The diverse political landscape is mirrored by an increasingly pluralistic society. For instance, in the aftermath of a 2017 Constitutional Court decision,4 same-sex marriage was introduced in 2019 (Ho, 2019). This chapter focuses on three main elements of the institutionalisation of evaluation in Taiwan. Firstly, the present chapter introduces and discusses the institutional structures and processes as well as laws and regulations pertaining evaluation. Secondly, the dissemination of evaluation into society and the discourse and perception of evaluation within society are described. Thirdly, the professionalisation of evaluation as a vocation and an academic discipline are explicated. This chapter was written based on a review of secondary data, mostly comprising academic and practitioner literature, primary data such as laws, regulations, and

4 Judicial Yuan Interpretation No. 748 (司法院釋字第748號解釋).

390

C.-Y. YUNG ET AL.

government or non-governmental organisation (NGO) websites, as well as interviews with fifteen Taiwanese evaluation experts from the field. The interviewed evaluation experts were recruited from organisations involved in evaluation policy-making, the planning and execution of evaluations, decision-makers in organisations subjects to evaluations, and scholars researching evaluation. Care was taken to include experts from numerous sectors, including politics and government, higher education, hospitality, and tourism, as well as civil society and academia.

2 Institutional Structures and Processes (Political System) This section provides a review of Taiwanese evaluation regulations, evaluation practice, and use of evaluations. To support our understanding of the Taiwanese evaluation context, it is important to explore both the geopolitical and historical contexts of Taiwan within the Greater China and Asia–Pacific regions. In this vein, an introduction to the contemporary political system of Taiwan necessitates a brief excursion into twentiethCentury history. Taiwan is ruled by the government of the Republic of China (ROC; 中華民國). The ROC is a unitary state, which was established in Mainland China subsequent to the 1911 Xinhai Revolution led by the KMT that overthrew the Qing Dynasty, and thus ended several thousand years of imperial rule (Jiming & Hu, 2014). Following the First Sino-Japanese War (1894–1895), China—then still ruled by the Qing Dynasty—ceded the island of Taiwan to Japan in 1895 (Liao, 2006). As part of its imperialist endeavours,5 Japan controlled the island as a colony for the next fifty years until the conclusion of World War II in 1945, when Taiwan came under the control of the Chinese government at the time, namely the ROC (Gold, 2016). Shortly after the end of World War II, the Chinese Civil War (1945–1949) erupted between the ROC and the Communist Party of China (Lew, 2009). With imminent defeat looming, in 1949 the ROC government together with circa 1.5 million soldiers, civil servants, and civilians relocated to Taiwan (Yang, 2021). Since 1949, Taiwan and a number of outlying islands remain the only territories under

5 In 1910, Japan annexed Korea, and in 1932, Japan annexed Manchuria; this was followed by further annexations in the course of World War II, e.g., the Philippines in 1942 (Yellen, 2019).

12

EVALUATION IN TAIWAN

391

the control of the ROC government in Taipei. Conversely, the Beijingbased People’s Republic of China (PRC; 中華人民共和國), established by Mao Zedong and the Communist Party of China, controls the entirety of Mainland China and is today commonly referred to as China. Despite this drastic reduction in territory, the Taipei-based ROC government continued to represent the entirety of China at the United Nations (UN) until its expulsion in 1971 (Lipscy, 2017). Thereafter, the UN seat of China was assumed by the PRC. The PRC—which considers Taiwan as its unalienable territory—threatens to sever diplomatic ties with any country that recognises the ROC government in Taipei under its One China Principle (一個中國原則). Thus, after 1971, the number of countries maintaining diplomatic relations with the ROC steadily declined. Today only 15 relatively small countries in Latin America, the Caribbean, and Oceania maintain diplomatic relations with the government in Taipei. Most other countries, such as the United States, Japan, Germany, or France, conduct unofficial relations through representative offices instead (Hickey, 2007). In consequence, Taiwan is not a member of the UN or its specialised organisations (e.g., the World Health Organization). Membership in organisations not requiring statehood or participation in sports events, e.g., the WTO or the Olympic Games, is under alternative names such as “Chinese Taipei” (Lipscy, 2017). While acknowledging political sensitivities surrounding terminology and nomenclature, in the interest of comprehensibility, for the remainder of this chapter we shall refer to both the geographical entity and its government simply as Taiwan. 2.1

Evaluation Regulations

Since the lifting of martial law in 1987, Taiwan has experienced a democratic transformation of its institutions. Maybe counterintuitively to these developments, the political framework of Taiwan has largely remained unchanged since the end of Japanese colonial rule. Namely, the 1947 Constitution (中華民國憲法) is still in force with several amendments and has “gradually evolved to become the Taiwanese constitution since 1949” (Hwang, 2016, p. 261). This constitution was at the time enacted for all of China—encompassing both Taiwan and the Chinese Mainland—and embodies both Western and imperial Chinese traditions (Chang, 2016). A meaningful tradition retained from Chinese imperial times on the constitutional level is the Control Yuan (CY; 監察院) instituted in art. 90 of the constitution. The CY can be considered the most comprehensive

392

C.-Y. YUNG ET AL.

body in the context of assessment and auditing at the national level. It is responsible for auditing government agencies, censuring civil servants, and impeaching public officers (Lo, 2006). It is therefore comparable in scope and function to the Audit Court in France (Lacouette-Fougère & Simon, 2020). The National Audit Office (審計部) is also part of the Control Yuan, and is comparable to the identically named National Audit Office in the United Kingdom. Its functioning is regulated in the Organization Act of the National Audit Office (審計部組織法), a national law passed by parliament, the Legislative Yuan (LY; 立法院). Under art. 5 of the act, the National Audit Office audits the financial and other activities of all levels and units of governments, arguably making it the most comprehensive law without a clear sectoral focus (albeit being limited to government). Unlike ministers of the executive branch of government, members of the Control Yuan cannot be removed by the president, thus ensuring a level of independence. As the CY is primarily concerned with censure, its functions are largely limited to ensuring the accountability of government agencies and ministers. In the realm of planning activities, the executive branch of government (EY) has established the National Development Council (NDC; 國家發 展委員會) in 2014. The NDC has a number of precursor agencies, such as the Council for Economic Planning and Development (CEPD; 經濟 建設委員會) set up in 1947, and the Research, Development and Evaluation Commission (RDEC; 研究發展考核委員會) set up in 1969. These institutions all date back to the coordination of support provided by the United States in the Post-World War II years, which was undertaken by the Council for United States Aid (CUSA; 美援運用委員會) established in 1948. Hence, management, evaluation, and planning principles imported through American experts and advisors nurtured the contemporary practices of planning in Taiwan. The NDC plays an important role in providing impact evaluations of proposed policies and laws, and additionally through process evaluations provides policy recommendations on a diverse range of issues. The tasks of the NDC are specified in art. 2 of the Organic Act of the National Development Council (國家發展委 員會組織法) and encompass planning, coordination, review, and resource allocation for • national development policy; • national development plans and coordination of mid- and long-term plan gender equality impact assessment;

12

EVALUATION IN TAIWAN

393

• economic development policy; • social development policy, and guidance and assistance to local governments for the implementation of main policy and mid- and long-term plans; • industrial development policy; • manpower resources development policy; • for national land, regional, and offshore island development and sustainable development policy; • cultural and ethnic development policy; • supervision and evaluation policy; • for government information management policy; • administrative and regulatory reform policy, and coordination of regulatory impact assessments; • other national development policies. Concrete examples include green energy policy, digital technology, urban–rural projects, child care, and demographic development (National Development Council, 2021a), or the performance assessment of government bodies and state-run enterprises (National Development Council, 2021b). In addition to the establishment of the National Development Council with its comprehensive policy evaluation scope by an act of parliament, there are also further national laws passed by the LY with sectoral foci. Furthermore, there is an equally great number of administrative regulations pertaining to evaluation, assessments, and audits enacted by executive agencies. In this vein, a search in the Ministry of Justice law database6 resulted in a total of more than 800 laws, regulations, and statutes where “evaluation” in some form is mandated.7 The broadness of these sectors is very comprehensive, as evidenced by the following examples applying on a national level:

6 Laws and Regulations Database (全國法規資料庫): https://law.moj.gov.tw. Accessed on 18 September 2023. 7 As a caveat, it should be mentioned that the Chinese word for “evaluation” (評鑑) is not clearly delineated from the term “assessment”, and thus each law, regulation, and statute needs to be judged by itself.

394

C.-Y. YUNG ET AL.

• Higher education institutions are required to undergo regular external and self-evaluation (art. 5 of the University Act 大學法); • The Central Disaster Prevention Council and Municipal Disaster Prevention Councils conduct performance audits and evaluations of the disaster prevention mechanisms and performance of the central and local governments (arts. 6 and 8 of the Disaster Prevention and Protection Act 災害防救法); • The Agency Against Corruption evaluates the Government Employee Ethics Units and its programmes targeted against civil servant misconduct and corruption (art. 2 of the Organic Act of the Agency Against Corruption 廉政署組織法); • The Ministry of Culture evaluates museums with regard to their professionalism in the areas of collection, research, exhibition, education, management, and public service (art. 16 of the Museum Act 博物館法); • To alleviate possible negative effects on consumers and the public due to the liberalisation of the telecommunications market, the Ministry of Transportation and Communications first needs to conduct an impact evaluation before granting a licence to a telecom firm (art. 12 of the Telecommunications Act 電信法); • The Tourism Bureau has set up evaluation mechanisms for designated scenic spots, hotels (e.g., star rating), travel agencies, and tourist amusement enterprises (art. 66 of the Act for the Development of Tourism 發展觀光條例); • For hospitals to engage in the training of doctors and medical students, they first require accreditation by the Ministry of Health and Welfare (art. 7 of the Medical Care Act 醫療法). These national laws are complemented by enforcement rules and administrative guidelines specifying details on how evaluations and accreditations are to be conducted. Thus, while the constitution is void of any general or abstract clause requiring evaluation as a fundamental principle, and there is also not a national strategy for evaluation in place, evaluation is nevertheless a present theme in Taiwan through the institution of sectoral laws and regulations. The majority of these laws and regulations lead to process evaluations and performance audits.

12

2.2

EVALUATION IN TAIWAN

395

Parliamentarian and National Audit Structures

The Legislative Yuan has the Legislative Research Bureau (法制局) at its disposal, which is responsible for evaluating laws and policies proposed or in force, in accordance with art. 20 of the Organic Statute of the Legislative Yuan (立法院組織法). The legislators are free in the exercise of their office, and there exists no formalised evaluation mechanism for members of the LY; apart from elections every four years, they are in principle only subject to a recall motion by the electorate under the Civil Servants Election and Recall Act (公職人員選舉罷免法). There are, however, civil society organisations active in the evaluation of legislators, which function like independent evaluation institutes (IEIs) and will be addressed in detail in Sect. 3.3 of this chapter. As session minutes indicate, evaluations are an often-recurring topic of discussion among legislators in the LY, e.g., the evaluation mechanism for judges (Legislative Yuan, 2021a), vocational education institutions (Legislative Yuan, 2021b), or childcare centres (Legislative Yuan, 2018). The minutes point to legislators debating the scope, usefulness, and mechanism of extant evaluations. The meetings also indicate that the primary use of evaluations by opposition legislators is to criticise the government. The National Audit Office, which is a part of the CY, is chiefly tasked with supervising the implementation of the budget, inspecting irregularities and dereliction of duties concerning the financial activities of government agencies, and evaluating the performance of financial activities (art. 2 of the Audit Act 審計法). The institutional scope of auditing includes national and local government agencies, governmentfunded non-profit organisations, as well as public–private partnerships. The auditor general is appointed by the president (art. 104 of the Constitution 中華民國憲法) for a six-year term. The National Audit Office conducts evaluations itself and for that purpose has set up offices in all municipalities and counties (art. 14 of the Organization Act of the National Audit Office; 審計部組織法). In sum, the National Audit Office is primarily concerned with performance audits. Functions beyond financial performance audits are assumed by the National Audit Office’s parent agency, the Control Yuan. As detailed in its annual reports, through its evaluation of agencies and specific cases of government policy and (mis-) conduct, the CY emphasises not only reducing public expenditure, but also rectifying the discipline of civil servants and elected officials, as well as safeguarding human rights through monitoring and evaluation of laws and policies (Control Yuan, 2015, 2020).

396

C.-Y. YUNG ET AL.

2.3

Organisational Structure

In addition to the Control Yuan and the National Audit Office, other branches of the national government, as well as ministries and government bodies, have set up their own units to evaluate policy and performance. For example, the Agency Against Corruption (廉政署) under the Ministry of Justice evaluates the work of the Government Employee Ethics Units (政風機構). While frequent and regular, these evaluations are internal and do not involve a single centralised evaluation unit. On the local level, city and county governments have set up evaluation units usually called Research, Development and Evaluation Commissions (RDEC; 研究發展考核委員會). The mission of these units is, among others, to inspect and control municipal government projects and evaluate and improve overall performance in citizen services (Taipei City Government, 2020). A concrete example is Taipei City’s RDEC conducting regular evaluations of municipal civil servants’ etiquette in handling citizen requests. A further example from Taipei City is the evaluation of innovative policy proposals from within the ranks of the municipal service. In addition to these efforts, the NDC provides guidance for the autonomous management and implementation of self-evaluation by government agencies at national and local levels (National Development Council, 2021c). As of yet, there exists no independent overall monitoring and evaluation system. Complementing these internal evaluations, a great part of external evaluation efforts in Taiwan are undertaken on behalf of the government by IEIs. These IEIs are usually NGOs. They are commissioned by the government and depending on the sector and specific evaluation are remunerated by either the evaluation subject or the government. As such they are not government agencies, but civil society actors. Hence, they will be introduced in greater detail in Sect. 3.3 of this chapter. Figure 1 presents the different types of organisations involved in evaluations in Taiwan. The evaluation subject shown in Fig. 1 could refer to a university or a hotel or a specific policy. 2.4

Evaluation Practice and Use

Evaluations are omnipresent in Taiwan across sectors. Nevertheless, levels of frequency and intensity vary between sectors due to different needs and legal requirements. For instance, the evaluation frequency for higher

12

EVALUATION IN TAIWAN

397

Control Yuan aud i disc ts and iplin es

National Audit Office

audits au

Executive Branch

entrusts

di t s

Independent Evaluation Institute (NGO) evaluates

au s dit

Ministries and Agencies internal evaluation Local Government

Evaluation Subject evaluates Independent Evaluation Institute (NGO not entrusted by the government)

internal evaluation

Fig. 1 Evaluation and audit organisations in Taiwan (own development) (Source Author’s own work)

education institutions (HEIs) is prescribed by law, while this is not the case for secondary schools (NCEE, 2021). Important sectors and policy fields identified by interviewed experts are diverse, ranging from higher education, tourism, and health care, to procurement and public–private partnerships, and human rights. When required by law, evaluations are typically carried out in cycles ranging from as often as yearly (disaster prevention), and typically not exceeding three to four years (hotels and higher education). Evaluations not required by law are carried out ad-hoc. Evaluation has also become a trend in the corporate world, with large corporations such as Chunghwa Telecom—the island’s largest telecommunications company—commissioning evaluations of their board of directors (Chunghwa Telecom, 2021). Higher education evaluations are arguably the most comprehensive and intensive example of evaluations undertaken in Taiwan. They are also a prominent example of external evaluation, while a majority of evaluations in Taiwan are still internal. Higher education evaluations in their present form are mandated by art. 5 of the University Act

398

C.-Y. YUNG ET AL.

(大學法) and required for universities, universities of science and technology, and junior colleges and institutes of technology. In total, more than 150 HEIs undergo regular evaluations. Article 3 sub-paragraph 2 of the University Evaluation Act (大學評鑑辦法) sets intervals of four to seven years for HEI evaluations. The evaluation of an HEI concludes with (re-)accreditation. Since the year 2004, these evaluations are undertaken by non-state actors, i.e., NGOs entrusted by the Ministry of Education (MOE; 教育部) with this task. Before 2004, the MOE conducted evaluations of HEIs by itself, the first of which took place in 1974 (TWAEA, 2014). This liberalisation spawned a considerable number of evaluation actors with comprehensive as well as specialised foci. Universities are evaluated and accredited by the Higher Education Evaluation and Accreditation Council of Taiwan (財團法人高等教育評鑑中心基金 會), whereas universities of science and technology, junior colleges, and institutes of technology are accredited by the Taiwan Assessment and Evaluation Association (社團法人台灣評鑑協會). In addition to the institutional level, these two IEIs also evaluate on the programme level. IEIs with a specialised focus on specific programmes are, for example, the Institute of Engineering Education Taiwan (IEET; 中華工程教育協會), the Taiwan Medical Accreditation Council (TMAC; 臺灣醫學院評鑑委員會), and the Chinese Management Association (CMA; 社團法人中華民國管理 科學學會). Section 3.3 provides a detailed introduction of these IEIs as civil society actors. Higher education evaluations are conducted every five to six years at the institutional and programme levels. Evaluators are peer-group experts from the higher education field (e.g., chaired professors, retired university presidents) or domain-experts of a specific academic discipline or vocational trade, as well as representatives from industry to incorporate the view of future employers. Evaluation procedures include a self-evaluation portion, on-site visit, and opportunities for self-enhancement. HEIs are in principle free to choose any IEI, provided that it is recognised by the MOE. On the programme level, this includes international evaluation organisations. In this vein, between 2009 and 2021 more than twenty Taiwanese higher education institutions have attained international accreditation by the Association to Advance Collegiate Schools of Business (AACSB, 2021). The internationalisation of higher education evaluations in unison with the liberalisation of the national evaluation framework point to the main functions of evaluation in this specific sector: steering and legitimacy. Due to the low island’s low birth rate in the past

12

EVALUATION IN TAIWAN

399

three decades, higher education in Taiwan is a red ocean environment. Government allocation of funding on higher education is scrutinised by the public, and at the same time HEIs compete with another in the recruitment of students both domestically and from abroad (Hou, 2011). Internationalisation of evaluation is also a trend in policy fields where the government is interested in legitimacy—this is particularly the case in the context of Taiwan’s human rights record, which the government views as a means of diplomatic soft power and measure to improve Taiwan’s standing in the world (Chen, 2019b). Despite not being a member of the UN, Taiwan has signed the ICCPR and the ICESCR with both covenants enforced as domestic law. In addition to self-evaluation, the government regularly invites independent foreign expert groups to evaluate Taiwan’s efforts and progress in realising these covenants. The findings are published as a report on the Ministry of Justice website, with the third iteration released in 2020.8 The reports are traditionally countered by a “shadow report” released by an independent civil society organisation, which in turn also invited foreign experts to join the evaluation (Covenants Watch, 2021b). Both the cases of higher education and human rights illustrate the strong motive and focus on international and domestic legitimacy internationally in Taiwan’s institutionalisation of evaluation. In sum, the interviewed experts agreed that there is a focus or even selectiveness on evaluations in sectors that have the prospect of positively impacting the standing of Taiwan at home and abroad, which is also reflected in the resources spent on evaluations. A further focus of evaluations conducted by government units lies on efficiency and spending. The experts also agreed that while many evaluations are carried out, there is not yet an intrinsic culture of evaluation.

3 Societal Dissemination/ Acceptance (Social System) 3.1

Use of Evaluations by Civil Society

In the Taiwanese context, it is a common occurrence for civil society organisations to have evaluations carried out by themselves. In addition to NGOs functioning as IEIs entrusted by government to conduct evaluations—which will be introduced in Sect. 3.3—, there are also NGOs 8 https://www.humanrights.moj.gov.tw/17998/17999/29677/29678/. Accessed on 9 November 2021.

400

C.-Y. YUNG ET AL.

not entrusted by government to take part in evaluations, which nevertheless conduct evaluations on their own impetus. Three such organisations are briefly introduced as follows. Citizen Congress Watch (CCW; 公民 監督國會聯盟) is an NGO established in 2007 tasked with evaluating legislators to advance “public welfare, transparency, efficiency, enlightenment and integrity” (Citizen Congrss Watch, 2021). Evaluations are conducted every six months, and the names of “excellent legislators” and “watch list legislators” are released to the public. A further noteworthy NGO is Covenants Watch (CW; 人權公約施行監督聯盟), which is an association comprising numerous human rights and civil rights advocacy groups. Established in 2009, CW conducts external evaluations of the implementation progress of key human rights legislation and covenants in Taiwan (Covenants Watch, 2021a). CW releases “shadow reports” in parallel to self-evaluation reports released by the government. The release of these reports is tied to concrete demands targeting the government, i.e., the realisation of civil and human rights commitments. A final example not in the realm of civil rights, but in the business world, is the Taiwan Corporate Governance Association (TCGA; 中華公司治理 協會), which was established in 2002 and conducts evaluation projects of corporate boards and firm management. Listed stock companies are required by law to undergo evaluation of their board. In addition, any enterprise can voluntarily commission TCGA to improve their corporate governance mechanisms. The evaluation consists of several steps, encompassing a self-evaluation report, evaluation of written materials provided by the enterprise, and an on-site evaluation where the directors, executive managers, and other staff are interviewed.9 The evaluation culminates in a written report by TCGA. The participation of non-IPO companies is, however, still limited and nascent. 3.2

Public Perception and Discourse on Evaluation and Evaluation Results

According to most interviewed experts, the general perception is that in Taiwan there is a strong institutionalised use of evaluations, extending to the public sector, private sector, as well as civil society. Where required, evaluation reports are published on the websites of evaluated units and 9 https://www.cga.org.tw/f_3_07_board_evaluation_1application_assessment.aspx. Accessed on 20 November 2021.

12

EVALUATION IN TAIWAN

401

online repositories. This is a standard practice in an increasing number of highly diverse sectors, e.g., higher education,10 senior citizen services,11 community development programmes, national athletics centres,12 antimoney laundering,13 urban renewal and housing,14 veteran affairs,15 and transportation infrastructure.16 While there exist no empirical studies on the use of these published reports, the interviewed experts agreed that disseminating full evaluation reports should be seen in the wider context of government transparency. In this vein, Taiwan has enacted the Freedom of Government Information Law (政府資訊公開法) in 2005. The law aims to “facilitate people to share and fairly utilise government information, protect people’s right to know, further people’s understanding, trust and overseeing of public affairs, and encourage public participation in democracy” (art. 1). Increasingly, large enterprises and civil society organisations also publish evaluation reports on their website or in repositories. Examples from the business sector are the island’s largest telecommunications firm Chunghwa Telecom (中華電信), which publishes performance evaluation reports on its board of directors online,17 as well as Taiwan High Speed Rail Corporation (台灣高鐵) publishing an evaluation report on its corporate governance structures,18 or CTCI (鼎集團), a leading engineering, procurement, and construction services provider publishing a 10 http://iqas.twaea.org.tw/announcement. Accessed on 20 November 2021. 11 https://www.sfaa.gov.tw/SFAA/Pages/List.aspx?nodeid=463. Accessed on

20

November 2021. 12 https://www.sa.gov.tw/PageContent?n=3738. Accessed on 20 November 2021. 13 https://www.amlo.moj.gov.tw/1461/31062/1482/15235/post. Accessed on 20

November 2021. 14 https://twur.cpami.gov.tw/resources/website/theme_file/223/108年度績效評鑑報 告.pdf. Accessed on 20 November 2021. 15 https://www.vac.gov.tw/vac_service/hualien/cp-2088-35039-1.html. Accessed on 20 November 2021. 16 https://www.motc.gov.tw/ch/home.jsp?id=15&parentpath=0,2&mcustomize=mul timessages_view.jsp&dataserno=202107060001&aplistdn=ou=data,ou=bulletin,ou=chines e,ou=ap_root,o=motc,c=tw&toolsflag=Y&imgfolder=img%2Fstandard. Accessed on 20 November 2021. 17 https://www.cht.com.tw/en/home/cht/sustainability/corporate-governance/perfor mance-management-of-the-board-of-directors. Accessed on 20 November 2021. 18 https://en.thsrc.com.tw/ArticleContent/1b8be2ae-8dde-41aa-87db-aba277e6eb35. Accessed on 20 November 2021.

402

C.-Y. YUNG ET AL.

corporate social responsibility report that details the sustainability evaluation measures applied to vendors.19 An example from the civil society organisations sector is the International Cooperation and Development Fund (ICDF; 財團法人國際合作發展基金會),20 an organisation involved in outbound development aid. The evaluation reports by the ICDF generally conclude with recommendations in the form of learnings from a specific project, which aim to inform the planning of future projects. Therefore, evaluation reports have a direct effect on the spending of Taiwan’s outbound development aid. In addition to published evaluation reports, evaluation results in sectors or policy fields relevant to wider audiences are often discussed in news media. Especially relevant fields are secondary and higher education, food safety, and virtually any other field of importance to Taiwan’s international reputation. Politicians and government bodies disseminate these evaluation results to showcase successful policies or achievements. An example is the Asia/Pacific Group on Money Laundering peer evaluation. Namely, subsequent governments ascribe Taiwan’s incremental improvements in this evaluation to their policies and governance (Times, 2020). Local governments are also active in this dissemination of evaluation results to sway public opinion, e.g., GHP accreditation of more than 200 small restaurants and food vendors in a Southern Taiwanese town (United Daily News, 2020). Evaluated units, such as universities, are also active in disseminating positive evaluation results through news media. For instance, the reaccreditation of degree programmes as part of the regular evaluation cycle of numerous universities was widely reported (e.g., ETToday, 2021a; Yahoo News, 2021). Likewise, a regional teaching hospital was covered on a leading news website on the occasion of their accreditation following an evaluation (ETToday, 2021b). While there exists no empirical study involving a sentiment analysis on the reporting of evaluation in Taiwan, it appears from a cursory search as well as expert interviews that reporting is usually positive and initiated by the evaluated unit. The media coverage and public discourse at times also incorporate criticism of social phenomena related to evaluation, e.g., the effect of peer- and 19 http://www.ctci.com.tw/www/CTCI2016/pdf/EN/2018-CSR-EN.pdf. on 20 November 2021.

Accessed

20 https://www.icdf.org.tw/lp.asp?CtNode=30791&CtUnit=441&BaseDSD=7&mp=2. Accessed on 20 November 2021.

12

EVALUATION IN TAIWAN

403

student evaluations in higher education on the teacher-student relationship (United Daily News, 2021). While the constitutional protections for freedom of expression are solid, and therefore anyone can publicly discuss evaluation results, it is important to point out that such discussions often fall victim to the fast-paced news in the competitive media environment and commodification of news. A lack of deeper public interest into evaluation as a concept on part of the public also contributes to this. 3.3

Participation of Civil Society in Evaluations

Civil society does not play a major active role in the internal evaluations of government agencies or the evaluations performed by the CY. However, as presented in Sect. 2, a great part of evaluations is not undertaken by the government itself, but entrusted to NGOs. These units are generally established as registered associations (社團法人) or foundations (財 團法人). One exemplary NGO from the higher education sector is introduced as follows. Amendments in 2004 to the University Act (大學法) for the first time required evaluation and accreditation of HEIs to be undertaken by units outside of the government, i.e., not the MOE as had been the case since 1974. In response, TWAEA was established by practitioners from academia and interested members of society to provide evaluation services to HEIs. TWAEA is a registered association established in 2004 by practitioners from academia and interested members of society. The core objective of TWAEA in the realm of higher education is the evaluation and accreditation of universities of science and technology, institutes of technology, and junior colleges on the institutional and programme levels. In doing so, TWAEA fulfils a statutory requirement on behalf of the government. TWAEA has a board of directors recruited from academia, as well as a pool of external evaluators also recruited from academia and industry on the basis of experience and professional merit. There are no evaluators recruited from civil society for the sole sake of strengthening the inclusion of civil society. Beyond the selection of evaluators, their training and the smooth operation of an entire evaluation cycle including on-site visits is another core task of TWAEA. The evaluation content is determined by MOE guidelines, and the evaluation results are published online.21 On the institutional level, HEIs can be awarded with one of three grades, namely “accredited”, “conditionally accredited”, and “denial”. These grades are applied

21 http://iqas.twaea.org.tw/en/announcement. Accessed on 20 November 2021.

404

C.-Y. YUNG ET AL.

to four key items, namely “Management and Development of Institutional Affairs”, “Curriculum and Teaching”, “Student Learning and Effectiveness”, and “Performance and Self-improvement of Institutional Affairs Management”. For comprehensive universities, these duties are analogously carried out by HEEACT. Aside from higher education, a further exemplary sector where evaluations are entrusted to NGOs is healthcare. The Center for Drug Evaluation (CDE; 財團法人醫藥品查驗中心) is a foundation set up by the Ministry of Health and Welfare (MOH; 衛生福利部) and among other tasks responsible for evaluating health technologies before their widespread procurement and use in hospitals and clinics. The objective of the CDE is to assist health authorities in making informed decisions through evaluations that take into consideration an analysis of comparative effectiveness, cost effectiveness, budget impact, as well as the ethical, legal, and social implications (CDE, 2021a). The CDE’s evaluations are itself not binding, but support the decision-making of a 30-member committee consisting of government officials, health professionals, healthcare providers, and consumer representatives (CDE, 2022). Personnel are recruited from experts in the fields of pharmacy, medicine, chemistry, biology, statistics, biomedical engineering, law, pharmacoeconomics, or other life science-related fields (CDE, 2021b). The responsibilities of the CDE include drafting evaluation guidelines, designing evaluations, and also conducting evaluations. In sum, civil society participation in evaluation is mixed. On the one hand, internal evaluations of government are virtually absent of civil society participation. On the other hand, in many instances evaluation units are not part of the government but NGOs entrusted by the government to conduct evaluations. There exists no prior study on the rationale behind the organisational structure of entities involved in evaluation in Taiwan, but the interviewed experts concurred that NGOs entrusted by the government have the ability to ensure institutional independence from the evaluation subjects (often government units or organisations under control of the government), while at the same time maintain financial sustainability. In sum, there is a considerable involvement of civil society in Taiwanese evaluation processes.

12

3.4

EVALUATION IN TAIWAN

405

Demand for Evaluation Within Civil Society

The institution of evaluations is rarely demanded by civil society. The interviewed experts concurred that in the perception of regular citizens, evaluations are already omnipresent. When demanded, evaluation serves to ensure accountability, i.e., citizens wish to hold state actors accountable. This can be seen in the context of a noticeable trend, which stems from the changing relationship of government and citizens over the course of the past 30–40 years. This mostly relates to the way in which government and its policies are held accountable by the people. One prominent example where actors from civil society demand evaluations is the justice system. The Judicial Reform Foundation (JRF; 財 團法人民間司法改革基金會) is an NGO established in 1995 with the aims of “strengthening civil society to advance judicial reform in Taiwan, improving the justice, transparency, and democracy of the Taiwanese judicial system, and taking actions against judicial unfairness and court negligence” (JRF, 2016). Among other activities, the JRF has for many years been campaigning for an effective evaluation mechanism of judges’ performance. Evaluation of judges was introduced under the Judges Act (法官法) in 2012 and aims to counter incompetent or abusive judges (Library of Congress, 2012). However, to the dismay of the JRC none of the several hundred applications for evaluation filed resulted in an evaluation (Taipei Times, 2021). Hence, the JRC is highly critical of existing evaluation mechanisms and demands further reform to the evaluation of judges, e.g., that NGOs can apply for evaluation rather than only individual citizens involved in a specific court case (RTI, 2021). Two further examples are the evaluation of legislators by CCW and evaluation of human rights policies by CW (both introduced in Sect. 3.1). Together, these three examples underscore the input of the interviewed experts. The justice system, parliament, and human rights are very immediate experiences citizens have with state power. Given that Taiwan underwent a democratisation transformation since the late 1980s, it only appears logical for citizens to make use of the well-known tool of evaluations in their quest for interacting with the state.22 What hinders the further articulation of demand for evaluations is a sectoral fragmentation 22 Please note that it is important to differentiate here: While audits and performance audits are well-known and are demanded as long as citizens’ immediate interests are concerned, evaluation in the narrower sense is rather unknown and accordingly not very much in demand by civil society.

406

C.-Y. YUNG ET AL.

of demands. In specific niches there exists a strong demand for evaluations, however there is no group in civil society demanding evaluations abstractly. This hinders the development of an evaluation culture through civil society demands. Another hindering factor is an indifference towards evaluations in large parts of the population.

4 Professionalisation of Evaluation (System of Professions) 4.1

Academic Study Courses and Training Practices

Numerous universities have set up research centres concerned with evaluation. The majority of these research centres have a sectoral focus on education in general or higher education in specific and also engage in institutional research. For example, National Taiwan Normal University (國立臺灣師範大學)—which is Taiwan’s premier HEI in the field of secondary and vocational school teacher cultivation—has established the Center for Education Research and Evaluation (教育研究與評鑑中心) in 2009. The history of predecessor organisations extends back to 1990 (NTNU, 2016). A further example is the Center for Academic Development and Evaluation (校務發展暨評鑑中心) at Fu Jen Catholic University (輔仁大學). These preceding examples are distinct from the organisational units within HEIs that liaison with IEIs. A number of universities offer graduate programmes in evaluation. Examples are the Department of Education at National Taipei University of Education (國立臺北教育大學), which offers a “Master’s Degree in Educational Innovation and Assessment” (NTUE, 2021), the Institute of Educational Administration and Evaluation at University of Taipei (臺北 市立大學), which offers master’s and doctoral programmes (UT, 2021), the Department of Education and Learning Technology at National Tsing Hua University (國立清華大學), which offers a master’s programme in administration and evaluation (NTHU, 2020). As these examples show, a sectoral emphasis is again placed on the education field. There also exists no degree programme, institute, department, or chair concerned with evaluation as a general academic discipline independent of sectors.

Editor’s note: It should be noted that in Taiwan civic demand is often expressed as a general request for (evaluative) evidence, which underlines the strong civic tradition and civil society’s appreciation of accountability.

12

EVALUATION IN TAIWAN

407

The scarceness of dedicated degree programmes and academic units engaged in the endeavour of evaluation as a scientific discipline begs the question as to the community-grounding of Taiwanese evaluation scholars and the nurturing of evaluation human capital. This is largely a contribution of NGOs carrying out evaluations (Sects. 2.1 and 2.3). These NGOs recruit evaluators from diverse academic and professional backgrounds, who often lack prior experience with planning and implementing evaluations. Likewise, administrative and management staff also hail from a multitude of academic backgrounds. To make up for the absence of formal evaluation study programmes offered by HEIs, TWAEA has set up a rigorous training curriculum spanning several weeks for evaluators. This is complemented by a comprehensive training regimen for its administrative staff, which also includes regular staff exchanges with evaluation organisations in the Asia–Pacific region. To conclude, the interviewed experts agreed that the notion of “Evaluation Science” (Patton, 2018) is not yet a universally accepted branch of science among other branches of science, but rather an interdisciplinary field or a transdiscipline populated by a diverse community of researchers originating from virtually any discipline. 4.2

Journals and Communication Platforms

Taiwanese scholars and practitioners involved with evaluation publish in a number of journals. There are currently three major domestic journals dedicated to evaluation. The Journal Evaluation Bimonthly (評鑑雙 月刊) is practitioner-oriented published every two months. The Journal of Educational Administration and Evaluation (教育行政與評鑑學刊) is an academic journal published twice per year. The two aforementioned journals publish Chinese-language papers. The Journal Higher Education Evaluation and Development (高教評鑑與發展期刊) is likewise research-oriented, but publishes English-language articles; it also differs from the preceding examples in that it is published by a major international publishing house (namely Emerald). As far as domestic journals are concerned, there is a considerable corpus of literature that is published in journals with a focus on a specific academic field, such as tourism management when writing on evaluations in the tourism sector, or political science when writing on policy evaluations. Higher education is an exception to this observation, as there are journals with a specific focus on evaluations within this sector.

408

C.-Y. YUNG ET AL.

Beyond domestic journals, members of the Taiwanese evaluation community also publish their work internationally. A Web of Science search showed that there were 62 articles with at least one Taiwanaffiliated author published between 2000 and 2021 in eight leading international evaluation journals covered in the database (Table 1). In comparison, there were 2,192 contributions with at least one US-affiliated author, 451 contributions with at least one UK-affiliated author, 157 contributions with at least one Germany-affiliated author, 40 contributions with at least one France-affiliated author, 29 contributions with at least one South Korea-affiliated author, 23 contributions with at least one Japan-affiliated author, and 21 contributions with at least one Singapore-affiliated author in the same set of journals and period of time. In sum, this indicates that Taiwanese academics participate in the international scholarly discourse. Namely, while not yet on par with European and North American countries and territories, in an East Asian context, Taiwan is comparatively well-represented. A further query of Web of Science for evaluation-related articles in journals without a specific focus on evaluation yielded 4,445 papers. The query revealed a great diversity of disciplines ranging from the social and behavioural sciences to business and management, medicine, engineering, and natural sciences. This echoes the overall status of “evaluation science” in Taiwan, i.e., there is Table 1 Publications in international evaluation journals with ≥ 1 Taiwanaffiliated author(s) Journal Name Assessment and Evaluation in Higher Education Educational Evaluation and Policy Analysis Research Evaluation Evaluation and the Health Professions Educational Assessment Evaluation and Accountability American Journal of Evaluation Studies in Educational Evaluation Evaluation and Programme Planning

2020 Impact Factor

Publications

4.984

8

3.347 2.706 2.651 2.000

1 9 16 2

1.967 1.953 1.849

1 4 21

Source Author’s own work; data obtained from Clarivate Analytics (2021). Note only journals indexed in the SSCI and SCIE collections were included; entries ranked by Impact Factor in descending order

12

EVALUATION IN TAIWAN

409

no such distinct field yet and scholars involved in evaluation belong to different communities. Complementing the aforementioned empirical findings and echoing the situation of academic study courses and training practices (Sect. 4.1), the interviewed experts agree that scholarly interest into evaluation in Taiwan is active, however there is still a fragmentation into sectors and disciplines, rather than a single community of evaluation scholars. 4.3

Professional Organisations

There are no formally established professional organisations or networks uniting all or the majority of IEIs in Taiwan. Professional exchanges between individual Taiwanese evaluation organisations occur in a sectoral manner and often bilaterally or multilaterally through domestic conferences and workshops. The interviewed experts opined that this is due to Taiwan’s relatively small geographic size and population, as well as the great emphasis placed on regional and international exchanges. Thus, it is a common phenomenon that Taiwanese IEIs in, e.g., higher education such as TWAEA are members of the Asia Pacific Quality Network (APQN)23 the International Network for Quality Assurance Agencies in Higher Education (INQAAHE),24 and CHEA International Quality Group.25 IEIs in the healthcare sector such as the Taiwanese Joint Commission (財團法人醫院評鑑暨醫療品質策進會) is a member of the International Society for Quality in Health Care (ISQua).26 4.4

Compliance with Quality Standards and Obligations

Both the internal evaluations conducted by government agencies, as well as the external evaluations conducted by entrusted NGOs or the Control Yuan follow regulations. These regulations determine the breadth and 23 https://apqn.org/aqpn-members/150-taiwan-assessment-and-evaluation-association. Accessed on 20 November 2021. 24 https://www.inqaahe.org/taiwan-assessment-and-evaluation-association. Accessed on 20 November 2021. 25 https://www.chea.org/international-directory/taiwan-assessment-and-evaluation-ass ociation. Accessed on 20 November 2021. 26 https://isqua.org/membership/institutional-members.html. November 2021.

Accessed

on

20

410

C.-Y. YUNG ET AL.

depth of the evaluation, its contents, and qualification requirements for evaluators. These regulations should not be confused with national laws and regulations wherein evaluation is mandated in principle for a specific sector or policy. For example, in higher education, the University Act ( 大學法) mandates evaluation, whereas the standards and quality obligations are specified in the University Evaluation Regulations (大學評鑑辦 法). These regulations require that IEIs “have a professional and objective evaluation implementation plan, including sufficient evaluation experts and scholars, a complete evaluation committee selection and training system, sufficient administrative personnel, and a sound organisation and accounting system” (art. 4 sub-paragraph 2 point 2). In Article 3, the regulation details the specific types of HEI evaluations, e.g., institutional and programme-level, and what contents each type includes. In Article 11, the regulation requires that any evaluations be “fair and objective”, and in cases of disagreement, article 8 empowers the evaluated HEI to respond to the draft evaluation report. Similarly-detailed regulations exist in other sectors, too. Table 2 presents a number of such regulations from a diverse selection of exemplary sectors. As has been established in the previous paragraph, standards, and quality obligations are well-specified. Compliance to these standards and quality obligations is enforced by penal provisions. In other words, independent evaluation institutes not adhering to the regulations may no longer be recognised by the respective executive-branch ministry or agency. Thus, the IEI in question would lose the formal qualification to participate in evaluations. While IEIs in Taiwan are generally nonprofit NGOs, this would nevertheless mean a fatal outcome for the IEI’s survival. In addition to penal provisions, and in the absence of a national certification system for evaluators, a large proportion of IEIs has joined international evaluation networks and associations (see Sect. 4.3).

5

Conclusions

The present chapter described key aspects of the institutionalisation of evaluation in Taiwan. It is important that the levels of institutionalisation are decidedly sector-dependent, with higher education likely the most developed in terms of professionalisation, internationalisation, integration of civil society, and academic involvement and output. In the Taiwanese context, it is important to note that there is a long history of process evaluation and performance audit stemming from imperial Chinese times

12

Table 2

EVALUATION IN TAIWAN

411

Selection of regulations specifying evaluations in Taiwan

Title

Title (Official Chinese)

Sector

University Evaluation Regulations High-school Evaluation Regulations

大學評鑑辦法 高級中等學校評鑑 辦法 幼兒園評鑑辦法 長期照顧服務機構 評鑑辦法 護理機構評鑑辦法 兒童及少年福利機 構評鑑及獎勵辦法 博物館評鑑及認證 辦法 星級旅館評鑑作業 要點 營造業評鑑辦法 寵物業查核及評鑑 辦法 大眾運輸營運與服 務評鑑辦法

Education

Kindergarten Evaluation Regulations Regulations for the Evaluation of Long-term Care Service Institutions Nursing Home Evaluation Regulations Evaluation and Reward Regulations for Children and Juvenile Welfare Institutions Museum Evaluation and Accreditation Regulations Enforcement Rules for Hotel Star Rating Construction Industry Evaluation Regulations Pet Industry Evaluation Regulations Public Transport Evaluation and Service Regulations

Social Services

Culture Business

Source Author’s own work

(i.e., at least several hundred or even more than one thousand years ago). This has laid the foundation for institutions such as the Control Yuan or the implementation of evaluation in a plethora of sectors through national laws and regulations. Bearing this long history of evaluations in mind, it is also important to note that our current notion of “evaluation” has only commenced with the involvement of the United States in Post-World War II Taiwan. The synthesis of these two streams of evaluation history has produced the current institutionalisation of evaluation in Taiwan. Taiwan has since the late 1980s experienced a transformation from an authoritarian single-party system to a multi-party democracy. This is evidenced by the deep and wide involvement of civil society in evaluations. This ranges from IEIs organised as NGOs entrusted by the government to conduct evaluations to NGOs independently conducting evaluations outside of the purview of the government to further their own social or political agendas. For Taiwan as a whole, evaluations are also an important means of soft power, as they showcase the island’s qualities in numerous sectors relevant to its international reputation. Internally

412

C.-Y. YUNG ET AL.

they also legitimise policies in the face of limited international space and recognition. While the quality of evaluations and evaluators is maintained through a solid legal and regulatory framework as well as training provided by IEIs, the development of “evaluation science” still requires considerable progress in Taiwan. Despite publications in major international journals relevant to evaluation, Taiwan, as of yet, lacks academic programmes and departments or institutes devoted to evaluation science as a distinct discipline. This extends to the absence of an academic evaluation community akin to, e.g., marketing, psychology, or chemistry. Rather, scholars and practitioners are fragmented into a plurality of communities. Given the integration of Taiwanese IEIs into regional and international professional organisations, there is a potential to learn from other countries where evaluation is more established in the academic sphere. Similarly, as evaluation becomes increasingly relevant in developing and emerging economies of East and Southeast Asia, there are ample opportunities for Taiwan to share knowledge and experiences.

References CDE. (2021a). Introduction. https://www.cde.org.tw/eng/HTA/. Accessed on 15 January 2023. CDE. (2021b). Organization. https://www.cde.org.tw/eng/about/about. Accessed on 15 January 2023. CDE. (2022). How does the HTA division carry out assessments? https://www. cde.org.tw/eng/faq/faq_more?id=1292. Accessed on 15 January 2023. Chang, W.-C. (2016). Comparative discourse in constitution making: An analysis on cinstitutional framers as dialectic agent. In C.-F. Lo, N. N. T. Li, & T.-Y. Lin (Eds.), Legal thoughts between the East and the West in the multilevel legal order (pp. 93–104). Springer. Chen, Y.-J. (2019a). Isolated but not oblivious: Taiwan’s acceptance of the two major human rights covenants. In J. A. Cohen, W. P. Alford, & C.-F. Lo (Eds.), Taiwan and international human rights: A story of transformation (pp. 207–226). Springer. Chunghwa Telecom. (2021). Performance management of the board of directors. https://www.cht.com.tw/en/home/cht/sustainability/corporategovernance/performance-management-of-the-board-of-directors. Accessed on 15 January 2023. Citizen Congress Watch. (2021). About citizen congress watch. https://ccw.org. tw/about/brief. Accessed from 15 January 2023.

12

EVALUATION IN TAIWAN

413

Clarivate Analytics. (2021). Journal citation reports. http://jcr.clarivate.com. Accessed on 15 January 2023. Control Yuan. (2015). 2015 Annual Report of the Control Yuan. https://www. cy.gov.tw/EN/News.aspx?n=253&sms=8937. Accessed on 15 January 2023. Control Yuan. (2020). 2020 Annual Report of the Control Yuan. https://www. cy.gov.tw/EN/News.aspx?n=253&sms=8937. Accessed on 15 January 2023. Covenants Watch. (2021a). About us. https://en.covenantswatch.org.tw/abo ut-us/. Accessed on 17 January 2023. Covenants Watch. (2021b). Treaty review. https://en.covenantswatch.org.tw/ treaty-reviews/. Accessed on 7 March 2023. Directorate-General of Budget, A. a. S. (2020). Statistical Yearbook of the Republic of China. Directorate-General of Budget, Accounting and Statistics. ETToday. (2021a). 世新大學力壓群雄!43學制評鑑全數通過 獲6年效期. https://www.ettoday.net/news/2021a0401/1951692.htm. Accessed on 15 January 2023. ETToday. (2021b). 台東基督教醫院通過評鑑 晉升花東唯一地區教學醫院. https://www.ettoday.net/news/20191220/1606846.htm. Accessed on 15 January 2023. Executive Yuan. (2016). Land and People (土地與人民; Tˇudì yuˇ rénmín ). https://www.ey.gov.tw/state/99B2E89521FC31E1/2820610c-e97f-4d33aa1e-e7b15222e45a. Accessed on 15 January 2023. Freedom House. (2021). Freedom in the World. https://freedomhouse.org/cou ntry/taiwan/freedom-world/2021. Accessed on 15 January 2023. Globalization and World Cities Research Network. (2020). The World According to GaWC 2020. https://www.lboro.ac.uk/gawc/world2020t.html. Accessed on 15 January 2023. Gold, T. B. (2015). State and society in the Taiwan miracle. Routledge. Gold, T. B. (2016). Retrocession and authoritorian KMT rule (1945–1986). In G. Schubert (Ed.), Routledge handbook of contemporary Taiwan (pp. 36–50). Routledge. Government Information Office. (2011). The Republic of China Yearbook 2011. Government Information Office. Heylen, A. (2012). Taiwan’s historical relations with Europe: Perspectives on the past and the present. In J. Damm, & P. Lim (Eds.), European perspectives on Taiwan (pp. 27–45). Springer. Hickey, D. V. (2007). Foreign policy making in Taiwan: From principle to pragmatism. Routledge. Ho, M.-S. (2019). Taiwan’s road to marriage equality: Politics of legalizing same-sex marriage. The China Quarterly, 238, 482–503. https://doi.org/ 10.1017/S0305741018001765

414

C.-Y. YUNG ET AL.

Hou, A.Y.-C. (2011). Quality assurance at a distance: International accreditation in Taiwan higher education. Higher Education, 61(2), 179–191. https://doi. org/10.1007/s10734-010-9331-9 Hsueh, R. Y. (2006). Who rules the international economy? Taiwan’s daunting attempts at bilateralism. In V. K. Aggarwal, & S. Urata (Eds.), Bilateral trade agreements in the Asia-Pacific: Origins, evolution, and implications (pp. 160– 183). Routledge. Hwang, J.-Y. (2016). Evolution of Taiwan’s Constitutional Court from 1948 to 2014: Emergence of an active but divided court. In H. Glaser (Ed.), Constitutional jurisprudence: Function, impact and challenges (pp. 259–280). Nomos. International Monetary Fund. (2021). World Economic Outlook Database. https://www.imf.org/en/Publications/WEO/weo-database/2021/April/ select-aggr-data. Accessed on 15 January 2023. Jiming, Z., & Hu, X. (2014). Conflict and competition: A new perspective on Late Qing politics. In J. W. Esherick, & G. C. X. Wei (Eds.), China: How the empire fell (pp. 66–86). Routledge. JRF. (2016). We are committed to establish a fair, just, and trustworthy judiciary in Taiwan. https://english.jrf.org.tw. Accessed on 15 January 2023. Lacouette-Fougère, C., & Simon, B. (2020). France. In R. Stockmann, W. Meyer, & L. Taube (Eds.), The Institutionalisation of evaluation in Europe (pp. 139–165). Springer International Publishing. Legislative Yuan. (2018). Minutes of the 5th Plenary Committee of the Social Welfare and Sanitation and Environment Committee of the 9th and 5th Session of the Legislative Yuan record (立法 院第 9 屆第 5 會期社會福利及衛生環境委員會第 5 次全體委員會議 紀錄). https://lci.ly.gov.tw/LyLCEW/communique1/final/pdf//107/27/ LCIDC01_1072701_00002.pdf. Accessed on 15 January 2023. Legislative Yuan. (2021a). Minutes of the 3rd Plenary Meeting of the Judicial and Legal Affairs Committee of the 8th Legislative Yuan’s 2nd Session (立法院第 8 屆第 2 會期司法及法制委員會第 3 次全體委員會議 紀錄). https://lci.ly.gov.tw/LyLCEW/communique1/final/pdf//101/54/ LCIDC01_1015402_00004.pdf. Accessed on 15 January 2023. Legislative Yuan. (2021b). Record of the 26th Plenary Meeting of the Education and Culture Committee of the 8th Session of the Legislative Yuan (立法院第8屆第1會期教育及文化委員會第26次全體委員 會議紀錄). https://lci.ly.gov.tw/LyLCEW/communique1/work/101/48/ LCIDC01_1014801_00005.doc. Accessed on 15 January 2023. Lew, C. R. (2009). The Third Chinese Revolutionary Civil War, 1945–49: An analysis of communist strategy and leadership. Routledge. Liao, P.-H. (2006). Taiwan under Japanese Colonial Rule 1895–1945: History, culture, memory. In D. Ping-Hui, & D. Der-Wei Wang (Eds.), Taiwan under

12

EVALUATION IN TAIWAN

415

Japanese Colonial Rule 1895–1945: History, culture, memory (pp. 1–16). Columbia University Press. Liberty Times Net. (2020). 洗錢防制也有國家隊! 台灣從劣等生變成資優 生. https://ec.ltn.com.tw/article/breakingnews/3391493. Accessed on 15 January 2023. Library of Congress. (2012). Taiwan: Judges Evaluation Committee Operational as Part of New Judges Act. https://www.loc.gov/item/global-legal-mon itor/2012-01-12/taiwan-judges-evaluation-committee-operational-as-part-ofnew-judges-act/. Accessed on 15 January 2023. Lipscy, P. (2017). Renegotiating the World order: Institutional change in international relations. Cambridge University Press. Lo, C.-F. (2006). The legal culture and system of Taiwan. Kluwer Law International. National Development Council. (2021a). Forward-looking Infrastructure Development Program. https://www.ndc.gov.tw/en/Content_List.aspx?n=BCD B1EECF95E18E2&upn=7767B950199EF590. Accessed on 15 January 2023. National Development Council. (2021b). Performance Management. https:// www.ndc.gov.tw/en/Content_List.aspx?n=E25F2172695D0302. Accessed on 15 January 2023. National Development Council. (2021c). Streamlining the Supervision and Evaluation Operations of Government. https://www.ndc.gov.tw/en/Content_L ist.aspx?n=F36EC80011BE7AC6. Accessed on 15 January 2023. NCEE. (2021). Governance. https://ncee.org/country/taiwan/. Accessed on 15 January 2023. NTNU. (2016). 大事紀. http://gc.art.ntnu.edu.tw/page/history. Accessed on 15 January 2023. NTHU. (2020). 課程介紹. https://delt.site.nthu.edu.tw/p/412-1129-35.php. Accessed on 15 January 2023. NTUE. (2021). Department of Education. https://www.ntue.edu.tw/Home? Sn=RpMMzDk0J51pBdRBqI%2fdxg%3d%3d&Dept.%20of%20Education. Accessed on 15 January 2023. Patton, M. Q. (2018). Evaluation science. American Journal of Evaluation, 39(2), 183–200. Ranis, G. (2002). Lessons from Taiwan’s performance: Neither miracle nor crisis. In P. C. Y. Chow (Ed.), Taiwan in the global economy: From an Agrarian economy to an exporter of high-tech products (pp. 3–38). Praeger. RTI. (2021). 民眾請求法官檢察官評鑑皆不成立 司改會籲降低門檻、採雙軌制. https://www.rti.org.tw/news/view/id/2110175. Accessed on 15 January 2023.

416

C.-Y. YUNG ET AL.

Taipei City Government. (2020). Research, Development and Evaluation Commission. https://english.rdec.gov.taipei/cp.aspx?n=5AB37C55E C737A26. Accessed on 15 January 2023. Taipei Times. (2021). Changes to Judges Act had no effect: Foundation. https://taipeitimes.com/News/taiwan/archives/2021/09/04/200 3763778. Accessed on 15 January 2023. The Economist. (2018). TSMC is about to become the world’s most advanced chipmaker. https://www.economist.com/business/2018/04/05/ tsmc-is-about-to-become-the-worlds-most-advanced-chipmaker. Accessed 15 January 2023. TWAEA. (2014). Higher Education Evaluation in Taiwan: Present State and Future Prospect. https://www.chea.org/sites/default/files/presentat ions/Mu%20PowerPoint%20Presentation.pdf. Accessed on 15 January 2023. United Daily News. (2020). 「食安在嘉」 逾2百餐飲店獲認證. https://udn. com/news/story/7326/5051342. Accessed on 15 January 2023. United Daily News. (2021). 當大學教師評鑑 淪為私校惡整老師的野蠻遊戲. https://vip.udn.com/vip/story/122367/5736727. Accessed on 15 January 2023. UT. (2021). Curriculum Introduction. https://adeva.utaipei.edu.tw/p/4261061-124.php?Lang=zh-tw. Accessed on 15 January 2023. Van den Berg, H., & Lewer, J. L. (2015). International trade and economic growth. Routledge. Wang, P.C.-M. (2015). A bastion created, a regime reformed, an economy reengineered, 1949–1970. In M. A. Rubinstein (Ed.), Taiwan: A new history (pp. 320–338). Routledge. Wen, C. P., Tsai, S. P., & Chung, W.-S. I. (2008). A 10-year experience with universal health insurance in Taiwan: Measuring changes in health and health disparity. Annals of Internal Medicine, 148(4), 258–267. World Bank. (2018). World Bank Country and Lending Groups. https://dat ahelpdesk.worldbank.org/knowledgebase/articles/906519#High_income/. Accessed on 15 January 2023. World Trade Organization. (2017). Accessions: Chinese Taipei. https://www. wto.org/english/thewto_e/acc_e/a1_taipei_chinois_e.htm. Accessed on 15 January 2023. Yahoo News. (2021). 真理大學22系所全通過高教評鑑中心認證. https://tw. news.yahoo.com/真理大學22系所全通過高教評鑑中心認證-132224173.htm l. Accessed on 15 January 2023. Yang, D.M.-H. (2021). The Great Exodus from China: Trauma, memory, and identity in Modern Taiwan. Cambridge University Press. Yellen, J. A. (2019). The greater East Asia co-prosperity sphere: When total empire met total war. Cornell University Press.

PART III

Transnational Organisations

CHAPTER 13

The Asian Development Bank and Evaluation in Asia and the Pacific Maya Vijayaraghavan

1 Asian Development Bank Organisational Structure The Asian Development Bank (ADB) is a multilateral development bank established in 1966 with the goal of fostering economic growth and cooperation in Asia and the Pacific. It is headquartered in Mandaluyong City in the National Capital Region of the Philippines. ADB has been supporting member countries in the region for over 50 years, playing a key role in their growth and transformation. In 2018, ADB approved its long-term corporate strategy, called “Strategy 2030”, which sets out its vision and

The opinions expressed in this article represent those of the author and not necessarily those of ADB. M. Vijayaraghavan (B) Independent Evaluation Department, Asian Development Bank, Mandaluyong, Philippines e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_13

419

420

M. VIJAYARAGHAVAN

path towards a prosperous, inclusive, resilient, and sustainable Asia and the Pacific (ADB, 2018d). Since its inception with 31 member countries, ADB has grown to include 68 shareholding members, with 49 from within Asia and the Pacific and 19 from outside the region (ADB, 2021f). ADB’s highest policy-making body is the Board of Governors, which comprises representatives from member nations, with one Governor per member nation. The Governors elect 12 members to form the full-time resident Board of Directors, responsible for the general operations of the Bank. The Directors oversee ADB’s financial statements, approve its administrative budget, and review and approve all policy documents, loans, equity, and technical assistance operations. The Board is chaired by the President of ADB. The President heads the management team, which includes a Managing Director General and six Vice-Presidents. The team supervises the work of ADB’s operational, administrative, and knowledge departments. ADB has six operations departments, consisting of five regional departments and the Private Sector Operations Department. These departments provide financial and technical assistance to developing member countries, with the support of ADB country offices in the region. Two departments focus on knowledge generation, management, and leadership. The Economic Research and Regional Cooperation Department conducts economic and statistical analysis to inform ADB’s strategies and operations, and the Sustainable Development and Climate Change Department provides knowledge leadership and services in ADB’s sectors and thematic work. The Strategy, Policy and Partnerships Department is responsible for ADB’s strategic planning and direction and ensures policy and operations coordination. This department is also responsible for ADB’s corporate results framework. The Independent Evaluation Department (IED) conducts evaluations to assess the development effectiveness of ADB’s operations and provides feedback and lessons to help ADB management and developing member country stakeholders improve future policies, programmes, and projects. The IED reports to ADB’s Board of Directors through the Development Effectiveness Committee (ADB, 2021d). ADB’s organisational chart is presented in Fig. 1. A new operating model and organisational structure is expected to be finalised in 2023 and the assignment of staff is to be completed in 2024.

Fig. 1 ADB organisational chart (Note Functional Organisational Chart [as of January 2022] from ‘Who We Are’ by Asian Development Bank, 2021. Figure recreated from this chart: https://www.adb.org/sites/default/files/page/532 576/adb-organization-chart-functional.pdf. Accessed 21 February 2022. Author’s own illustration based on the source)

13 THE ASIAN DEVELOPMENT BANK

421

422

M. VIJAYARAGHAVAN

2

Institutionalisation of Evaluation

The institutionalisation of evaluation is essential for improving the operational effectiveness of the Asian Development Bank (ADB) by promoting accountability and learning. ADB maintains two levels of systematic evaluation: self-evaluation and independent evaluation. The former is carried out by ADB operational departments, while the latter is conducted by the IED, which is responsible for ensuring that ADB-wide evaluation guidelines and practices are updated and aligned with international best practices. To set the institution’s overarching evaluation principles and guide ADB’s overall evaluation system, ADB and IED jointly developed an evaluation principles document in 2021. This internal administrative document complements the existing ADB evaluation policy and range of guidance documents for self- and independent evaluation, and more clearly articulates how self- and independent evaluation work together across the institution and the principles that guide IED and Management in their respective and complementary roles. The document outlines key evaluation principles to ensure quality, such as evaluator competency, clearly defined objectives, aligned evaluation methodologies, rigorous and transparent methodology with quality control processes, secured evaluation independence, and relevance and timeliness. ADB’s self-evaluation and independent evaluation systems operate at the corporate, country, project, and technical assistance levels. Corporate, thematic, and sector evaluations aggregate data and information centred on a specific corporate priority, theme, or sector from various ADB operations and activities which may involve different modalities, sectors, or countries, while project evaluations review the design, implementation, and performance of projects and programmes. Country evaluations assess the entire ADB support to a country covering ADB’s strategy, policy dialogue, both completed and ongoing projects and technical assistance, and ADB’s performance. Technical assistance (TA) evaluations assess ADB’s support activities which fall under the category of technical assistance which include project preparation and implementation activities, advisory services, research and development, and capacity development. Table 1 shows ADB’s self-evaluations and their counterpart evaluations or validations by independent evaluation for these four different analytical levels.

13

Table 1

THE ASIAN DEVELOPMENT BANK

423

ADB Corporate, country, and project evaluations

Level

Evaluation framework

ADB self-evaluation

Independent Evaluation Department

Corporate

Corporate results framework

Annual Evaluation Review

Country

Country partnership strategy results framework

Development Effectiveness Review (by Strategy, Policy, and Partnerships Department) Country Partnership Strategy Final Review (by operations departments)

Project

Design and monitoring framework

Project Completion Report (by operations departments)

Technical assistance

Design and monitoring framework

Technical Assistance Completion Report

Validation of Country Partnership Strategy Final Review Country Assistance Programme Evaluation Validation of Project Completion Report Project Performance Evaluation Report Validation of Technical Assistance Completion Report Technical Assistance Performance Evaluation Report

Source Author’s own work

Both levels of evaluation are ingrained in and supported by ADB’s structure, policies, systems, and practices. Evaluators are ensured access to information, and completed evaluation documents are disclosed to the public in accordance with ADB’s Access to Information Policy (ADB, 2018a). All reports, including IED and self-evaluations, are accessible via the ADB website search engine. The two central pillars of evaluation within ADB are characterised in detail in the following sections. 2.1

ADB Self-Evaluation

ADB has been using corporate results frameworks to manage its performance since 2008. These self-evaluations are published annually in the Development Effectiveness Review by ADB Management. In 2019, ADB

424

M. VIJAYARAGHAVAN

developed a new corporate results framework (ADB, 2019a) to align with Strategy 2030 (ADB, 2018d), which includes 60 indicators to track the region’s progress (Level 1) and measure ADB’s contributions to Strategy 2030 and its accompanying operational priorities (Level 2–4). ADB country engagements with developing member countries are guided by country partnership strategies, which identify priorities and track alignment with country development plans. These strategies incorporate results frameworks to manage operations towards desired development outcomes. Upon project completion, borrowers are expected to prepare their own completion reports, which help prepare ADB’s project completion reports. These reports assess ADB’s development effectiveness and guide ongoing and future projects. IED has assessed ADB’s projectlevel self-evaluation system and found it to be aligned with international good practices for accountability and learning (IED, 2020a). Project self-evaluations are conducted by the operations department responsible for the programme or project’s design and implementation (IED, 2020a). These evaluations are systematic processes and products integrated into each stage of the project cycle. During the project design stage, the design and monitoring framework (DMF) and monitoring and evaluation (M&E) systems are prepared, and a project administration manual details the project’s outputs to be delivered by executing agencies. The project M&E system is in place and functioning during the implementation stage, and ADB uses a bank-wide electronic system (eOperations) to manage all project-related information. The eOperations system tracks the progress of DMF output and outcome indicators towards respective targets. 2.2

Independent Evaluation by the Independent Evaluation Department

The Independent Evaluation Department (IED) of the Asian Development Bank (ADB) was established in 1972 as an evaluation unit under the Economics Office (IED, 2014a). Over the years, it has been renamed and reorganised several times to enhance its capacity, functions, independence, and influence. In 1978, a separate Post-Evaluation Office was established and was upgraded to the Operations Evaluation Department (OED) in 2001. In 2004, an independent OED was established, reporting directly to the Board of Directors through the Development Effectiveness Committee (ADB, 2003). In 2009, OED was renamed IED

13

THE ASIAN DEVELOPMENT BANK

425

to reflect its heightened independence, which is reflected in IED’s current evaluation policy (ADB, 2008). IED evaluates ADB’s operations and activities across various types and levels and follows the principles of independence, impartiality, and integrity. To maintain IED’s effectiveness and credibility, IED follows the principles of independence, impartiality, and integrity. Following the good practice standards of evaluation agencies, IED adopted four dimensions of evaluation independence to ensure freedom from material threats to objectivity of its evaluations: behavioural autonomy, organisational independence, insulation from external influence, and avoidance of conflicts of interest (ADB, 2021d). IED evaluators and management exclude themselves from evaluating or approving a report on any project or activity that they worked on, appraised, or were responsible for in a previous capacity, or when they expect to have such involvement in the future (IED, 2012). IED discloses its 3-year rolling work programme approved by the Board and all completed evaluation reports on its website. IED is composed of the Thematic and Country and the Sector and Project divisions, and an evaluation knowledge management unit supports the evaluation functions. Table 2 shows the wide range of IED evaluation products to assess ADB’s operations and activities. Organisational processes within ADB support the active and visible follow-up on the use of evaluation findings and recommendations, which are reported in the Annual Evaluation Review each year. Evaluation’s influence in the improvement of ADB’s development effectiveness and strategic direction are demonstrated in various evaluations conducted by IED. For example, in the 2017 evaluation of ADB Support for Gender and Development (IED, 2017a), IED recommended that ADB’s new strategy should identify gender targets along the five dimensions of gender equality: human development, economic empowerment, reduced time poverty, decision-making and leadership, and resilience to external shocks. ADB Management concurred with the recommendation and the subsequent operational plan for gender equality (ADB, 2019c) and Corporate Results Framework (2019a) identified measurable indicators and targets along the recommended five dimensions of gender equality. The 2019 Development Effectiveness Review was the first issue that reported ADB’s contributions to gender equality using this framework (ADB, 2020a).

426

M. VIJAYARAGHAVAN

Table 2

Typology of IED evaluation reports

IED Evaluation reports High-level evaluations Annual Evaluation Reviews

Corporate and Thematic Evaluations

Sector-wide Evaluations

Country Assistance Programme Evaluations/Regional Programme Evaluations

Sector Assistance Programme Evaluations

Impact Evaluations

Other evaluations Validations of Country Partnership Strategy Final Review Validations of Project Completion Reports and Extended Annual Review Report Project/Programme Performance Evaluations Validation of Technical Assistance Completion Reports Technical Assistance Performance Evaluation Reports

Assess ADB’s operational performance, synthesise lessons on development effectiveness, report on findings and actions taken on evaluation recommendations Focus on thematic issues across sectors or countries, or evaluate an ADB policy or business process Assess the performance and results of ADB’s sector strategy and operations across the Asia and the Pacific over an extended period to inform future engagement of ADB in the sector Periodic assessments conducted evaluating ADB’s support to a country or region, covering ADB’s country partnership strategy, policy dialogue, completed and ongoing projects and technical assistance. It provides ADB and the country with a basis for deciding on changes in the future country partnership strategy Evaluate the performance of ADB’s support to a sector in a country, including its strategy, policy dialogue, completed and ongoing projects Use quasi-experimental designs and emphasise validity and reliability of evaluation results Validate findings and assess the quality of the country partnership strategy final review Validate project and programme completion reports Evaluate the design, implementation, and performance of projects and programmes Validate technical assistance completion reports Evaluate the design, implementation, and performance of technical assistance projects, typically covering several projects in the same report

Note From Evaluation Types by Independent Evaluation Department, 2021, in Independent Evaluation (https://www.adb.org/site/evaluation/evaluations. Accessed on 19 January 2023). Copyright 2021 by Asian Development Bank. Author’s own illustration

13

THE ASIAN DEVELOPMENT BANK

427

ADB and IED conducted separate evaluations of ADB’s Pilot ResultsBased Lending for Programs (ADB, 2013). The ADB review found that results-based lending improved accountability (ADB, 2016). IED confirmed the programme’s value addition: results-based lending helped strengthen country systems and had the potential to deliver outputs and outcomes for sector programmes. IED supported the mainstreaming of results-based lending (IED, 2017b). In September 2019, results-based lending was mainstreamed as an ADB financing modality. The Mainstreaming policy (ADB, 2019b) introduced modifications to improve the use of results-based lending based on the findings and recommendations of the ADB and IED reviews. In 2020, IED assessed the relevance of the 2009 Energy Policy and ADB’s energy programme from 2009 to 2019 (IED, 2020b). The evaluation recommended a new energy policy to incorporate the urgent consensus on climate change and highlighted the need for ADB to formally exclude financing for new coal generation projects, and help member countries phase out coal-based energy and transition to cleaner energy. In October 2021, ADB approved a new energy policy to support access to reliable, affordable, and clean energy (ADB, 2021c). The policy states, “ADB will withdraw from financing new coal-fired power and heating plants, support DMCs in achieving a planned phase-out of coal in the Asia and Pacific region, and foster a just transition that considers its impacts on people and communities”. Table 2 shows the wide range of IED evaluation products to assess ADB’s operations and activities across various types and levels.

3 Contributions to the Professionalisation of Evaluation The Independent Evaluation Department (IED) follows evaluation guidelines consistent with the Organisation for Economic Co-operation and Development/Development Assistance Committee (OECD, 2019) and the Evaluation Cooperation Group’s (ECG, 2012) Good Practice Standards. IED’s evaluation is on par with, or superior to its peers in the Evaluation Cooperation Group, as assessed by the Asian Development Bank (ADB, 2018b). As one of the five founding members of the Evaluation Cooperation Group, IED is mandated to promote the harmonisation of evaluation criteria and methods, procedures, and evaluation governance among

428

M. VIJAYARAGHAVAN

multilateral development banks (ADB, 2008). Furthermore, as mandated in its evaluation policy, IED is responsible for fostering evaluation capacity development within ADB and in developing member countries (ADB, 2008).1 To achieve these objectives, IED has hosted and implemented numerous training and knowledge-sharing programmes and activities. For instance, IED established the ADB Evaluation Academy in 2018 to strengthen evaluation culture and practices in Asia and the Pacific. The Academy offers programmes catering to various levels of evaluation skills. Additionally, IED has sent government officials from developing member countries to attend M&E training programmes, such as the Shanghai International Program for Development Evaluation Training (SHIPDET) since 2007. Training on M&E and development evaluation were also conducted at centres of excellence in South Asia and the Pacific, and in-country training on evaluation techniques for government representatives were provided. IED has provided on-the-job training (OJT) at its headquarters, where trainees conduct joint evaluations in their own countries while being mentored by IED specialists, followed by a visit to ADB headquarters to work on the evaluation report. Follow-on workshops on development evaluation complement the SHIPDET training and OJT. In addition, IED conducts workshops on the evaluation of sovereign operations and technical assistance to improve the evaluation capacities of ADB staff. At ADB, the Economic Research and Regional Cooperation Department supports the conduct of impact evaluations in close collaboration with project-executing agencies of member countries (ADB, 2018c); the department also supports the conduct of impact evaluations in close collaboration with project-executing agencies of member countries. The Asian Evaluation Week (AEW) is a forum for the advancement of evaluation practice and development effectiveness at the national, regional, and international levels. It brings together government officials, policymakers, and evaluation experts to share knowledge, experiences, and ideas, and forge partnerships. Since 2016, IED and the Asia–Pacific 1 The capacity development activities aim to contribute to accountability and the SDG16. “Capacity to enhance the accountability of government institutions.” For example, see, https://www.adb.org/projects/54416-001/main. Accessed on 19 September 2023.

13

Table 3

THE ASIAN DEVELOPMENT BANK

429

Asian evaluation week: 2016–2022

Year

Host city in China

Theme

Participants

Countries

Organisations

2016

Xi’an, Shaanxi Hangzhou, Zhejiang Chengdu, Sichuan

Inaugural

158

39

45

190

37

40

241

48

55

213

47

50

1292

113

400

850

120

130

970

97

395

2017 2018

2019

2020 2021

2022

Evaluation for policy-making Making evaluation work at the country level Kunming, Quality evaluation Yunnan for better results: local, national, regional perspectives Virtual event Evaluating for a Better Future Virtual event Transformational Evaluation: Moving from Uncertainties to Resilience Hybrid event Reframing Evaluation for Green, Inclusive, and Resilient Recovery

Source Author’s own work

Finance and Development Institute of the Ministry of Finance, People’s Republic of China have collaborated to host the AEW. From 2016 to 2019, the AEW was hosted by select cities in China. The themes over the years reflected key issues faced by the evaluation community. As shown in Table 3, the breadth and number of participants grew over the years, with virtual events hosted in 2020 and 2021, a hybrid event in 2022, and a significant expansion in the number of participants.

4

Other Products

To make evaluation more accessible to a wider audience, IED creates additional products based on its major evaluation reports. Two such products are Evaluation in Brief, which condenses the reports into two-page

430

M. VIJAYARAGHAVAN

summaries, and Evaluation Illustrated, which uses infographics to reach a wider audience. IED’s ADB corporate website page2 receives a high number of page views and users, making it one of the most popular subsites within ADB. IED also has a presence on social media platforms such as Facebook,3 Twitter,4 YouTube,5 and LinkedIn.6 In addition to these, email listservs and email marketing (using Mailchimp) are also utilised as online dissemination methods for evaluation reports.

5

Conclusions

In conclusion, the Asian Development Bank (ADB) is a key player in the development landscape of Asia and the Pacific, and its commitment to evaluation is an important part of its institutional identity. This chapter has provided an overview of the ADB’s organisational structure, evaluation policy framework, and evaluation practices, including both self-evaluation and independent evaluation. Through its independent evaluation function, the ADB has contributed significantly to the professionalisation of evaluation in the region. The Independent Evaluation Department has established itself as a credible and independent voice, providing valuable feedback to the Bank’s operations and making important contributions to the broader development community. While the ADB has made significant progress in institutionalising evaluation and improving its evaluation practices over time, there are still opportunities for further improvement. The ADB must continue to strengthen its evaluation capacity, including by ensuring that evaluation findings and recommendations are used effectively in decision-making processes.

2 https://www.adb.org/site/evaluation/main. Accessed on 19 January 2023. 3 https://www.facebook.com/adbevaluation. Accessed on 19 January 2023. 4 https://twitter.com/adbevaluation. Accessed on 19 January 2023. 5 https://www.youtube.com/c/ADBEvaluation. Accessed on 19 January 2023. 6 https://www.linkedin.com/showcase/independent-evaluation-at-the-asian-develo

pment-bank. Accessed on 19 January 2023.

13

THE ASIAN DEVELOPMENT BANK

431

Overall, the ADB’s contributions to the professionalisation of evaluation in the region are significant. The ADB’s experience provides important lessons for other development organisations seeking to strengthen their evaluation practices and enhance their impact on the lives of people in Asia and the Pacific.

References Asia Pacific Finance and Development Institute and Independent Evaluation Department. (2020). Asian Evaluation Week: A retrospective. https:// asianevaluationweek.org/asian-evaluation-week-retrospective. Accessed on 19 January 2023. Asian Development Bank and Independent Evaluation Department. (2021). Asian Development Bank. Evaluation principles. Asian Development Bank. Asian Development Bank. (2003). Enhancing the independence and effectiveness of the operations evaluation department. Asian Development Bank. Asian Development Bank. (2008). Review of the independence and effectiveness of the operations evaluation department. Asian Development Bank. Asian Development Bank. (2013). Piloting results-based lending for programs. Asian Development Bank. Asian Development Bank. (2016). Midterm review of results-based lending for programs. Asian Development Bank. Asian Development Bank. (2017). Guidelines for the economic analysis of projects. Asian Development Bank. Asian Development Bank. (2018a). Access to information policy. Asian Development Bank. Asian Development Bank. (2018b). An external review of the independent evaluation department. Asian Development Bank. Asian Development Bank. (2018c). Regional: Developing impact evaluation methodologies, approaches, and capacities in selected developing member countries (Subproject 2). https://www.adb.org/projects/46185-003/main. Accessed on 24 October 2021. Asian Development Bank. (2018d). Strategy 2030: Achieving a prosperous, inclusive, resilient, and sustainable Asia and the Pacific. Asian Development Bank. Asian Development Bank. (2019a). ADB results framework, 2019–2024. Asian Development Bank. Asian Development Bank. (2019b). Mainstreaming the results-based lending for programs. Asian Development Bank. Asian Development Bank. (2019c). Strategy 2030 operational plan for priority 2. Accelerating progress in gender equality, 2019–2024. Asian Development Bank.

432

M. VIJAYARAGHAVAN

Asian Development Bank. (2020a). Development effectiveness review 2019. Asian Development Bank. Asian Development Bank. (2020b). Guidelines for preparing and using a design and monitoring framework: Sovereign operations and technical assistance. Asian Development Bank. Asian Development Bank. (2021a). Country partnership strategy and results framework review. Asian Development Bank. Asian Development Bank. (2021b). Development effectiveness and results. https://www.adb.org/who-we-are/development-effectiveness/adb-resultsframework. Accessed on 24 October 2021. Asian Development Bank. (2021c). Energy policy. Supporting low-carbon transition in Asia and the Pacific. Asian Development Bank. Asian Development Bank. (2021d). Independent evaluation. https://www.adb. org/site/evaluation/main. Accessed on 24 October 2021. Asian Development Bank. (2021e). Terms of reference of the development effectiveness committee of the board of directors. https://www.adb.org/ sites/default/files/institutional-document/32128/tor-development-effective ness-committee.pdf. Accessed on 24 October 2021. Asian Development Bank. (2021f). Who we are. https://www.adb.org/who-weare/main. Accessed on 24 October 2021. Evaluation Cooperation Group (ECG). (2012). ECG big book on good practice standards. https://www.ecgnet.org/document/ecg-big-book-good-practicestandards. Accessed on 19 January 2023. Independent Evaluation Department. (2012). Guidelines to avoid conflict of interest in independent evaluations. Asian Development Bank. Independent Evaluation Department. (2014a). Evaluation for better results. Asian Development Bank. Independent Evaluation Department. (2014b). Guidelines for the preparation of project performance evaluation reports on nonsovereign operations. Asian Development Bank. Independent Evaluation Department. (2015). 2015 Guidelines for the preparation of country assistance program evaluations and country partnership Strategy final review validations. Asian Development Bank. Independent Evaluation Department. (2016). Guidelines for the evaluation of public sector operations. Asian Development Bank. Independent Evaluation Department. (2017a). Asian Development Bank support for gender and development 2005–2015. Asian Development Bank. Independent Evaluation Department. (2017b). Corporate evaluation: Resultsbased lending at the Asian Development Bank: An early assessment. Asian Development Bank. Independent Evaluation Department. (2020a). 2020 Annual evaluation review. ADB’s project level self-evaluation system. Asian Development Bank.

13

THE ASIAN DEVELOPMENT BANK

433

Independent Evaluation Department. (2020b). Sector-wide evaluation: ADB’s 2009 energy policy and program, 2009–2019. Asian Development Bank. Independent Evaluation Department. (2020c). Technical assistance completion report validation guidelines. Asian Development Bank. Network on Development Evaluation. (n.d.). Better Criteria for Better Evaluation. Revised Evaluation Criteria Definitions and Principles for Use. https://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec2019.pdf. Accessed on 19 January 2023. Organisation for Economic Co-operation and Development. (OECD). (2019). Better criteria for better evaluation. Revised evaluation criteria definitions and principles for use. OECD/DAC Organisation for Economic Co-operation and Development. (OECD). (n.d.) Evaluation criteria. https://www.oecd.org/dac/evaluation/daccriteriaforeval uatingdevelopmentassistance.htm. Accessed on 19 January 2023.

CHAPTER 14

UNICEF Support to Institutionalising Evaluation in East Asia and the Pacific Region Koorosh Raffii and Xin Xin Yang

1

Introduction

This article is solely based on the personal experience of the authors and their desk review of relevant documents and publications. Please note that wherever region or regional is mentioned without further explanation in this article, it refers to UNICEF’s East Asia and the Pacific (EAP) Region. The authors present their view on the status of the evaluation function in UNICEF, and the organisation’s support to the institutionalisation of evaluation in the EAP Region. The article is organised as follows: (a) An overview of UNICEF as an organisation; (b) The structure and details

K. Raffii (B) Evaluation and Head of Evaluation Section, UNICEF East Asia and Pacific Regional Office (EAPRO), New York, NY, USA e-mail: [email protected] X. X. Yang UNICEF, New York, NY, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_14

435

436

K. RAFFII AND X. X. YANG

of the evaluation function within UNICEF; (c) The relationship of the Regional Evaluation Function (REF) with EAP countries; (d) UNICEF REF relationship with civil society; (e) UNICEF’s contributions to the professionalisation of evaluation both at the global and regional levels. The article concludes by presenting the future vision for the Evaluation Function in UNICEF, which will impact the REF. Hyperlinks to information and documents that can be accessed by the public are presented as appropriate. This is done to enable the readers that are interested in additional information on a particular subject matter to easily access sources referred to or used by the authors. It is presumed that the readers are familiar with the limitations of articles that are based on personal experience, personal views and desk reviews. These limitations, and the fact that limited time could be allocated to this endeavour, should be carefully considered when reading this article. This article does not present the viewpoint of UNICEF, the East Asia and the Pacific Regional Office (EAPRO) or any other entity. Any viewpoints expressed are solely those of the authors.

2

Overview of the Organisational Structure

Established in 1946, UNICEF is mandated by the United Nations General Assembly to advocate for the protection of children’s rights, to help meet their basic needs, and to expand their opportunities to reach their full potential (UNICEF’s mission statement).1 UNICEF is guided by the Convention on the Rights of the Child2 and relevant human rights3 instruments. The organisation strives to establish children’s rights through enduring ethical principles and international standards of behaviour towards children. UNICEF insists that the survival, protection and development of children are universal development imperatives that are integral to human progress. Its core programmes include health; nutrition; education; child protection; water, sanitation and hygiene (WASH);

1 UNICEF’s mission statement: https://sites.unicef.org/about/who/index_mission. html. Accessed on 25 March 2022. 2 https://www2.ohchr.org/english/bodies/crc/docs/gc/crc_c_gc_14_eng.pdf. Accessed on 25 March 2022. 3 https://socialprotection-humanrights.org/legal-depository/legal-instruments/unhuman-rights-instruments/. Accessed on 25 March 2022.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

437

sustainable safe climate environment; social inclusion and poverty reduction. All UNICEF interventions take the human rights approach to programming, gender equality/girls’ empowerment, enhanced equity and emergency relief into consideration. In addition, UNICEF interventions integrate a number of cross-cutting dimensions, such as adaptation to climate change, disabilities, the triple nexus of development, emergency plus peacebuilding, early childhood and adolescent development. In carrying out its mandate, UNICEF prioritises the most disadvantaged children and countries in greatest need. The principle of leaving no child behind is adhered to within the various elements of all interventions. Details of UNICEF work and the nature of its interventions can be found through its new Strategic Plan, which covers the period from 2022 to 2025 (UNICEF, 2022a). Evaluations are included as one of the change strategies and are considered to be key to accelerating progress towards the Sustainable Development Goals (SDGs). Please refer to Annex one for the organisation chart of evaluation in EAPRO (see Fig. 1 in Appendix). UNICEF is led by an Executive Director appointed by the UN Secretary-General. The Executive Board is the governing body of the organisation, providing intergovernmental support and oversight through reviewing activities and approving policies, country programmes and budgets.4 UNICEF headquarters (HQ) are hosted in New York City, USA, where most of the corporate services and HQ programme divisions are based, and global policy on children is shaped. The New York Office is complemented by other headquarters locations across the globe, including an Office of Research-Innocenti (OoR) in Florence,5 a supply hub in Copenhagen,6 a global shared services centre in Budapest and additional offices in Brussels, Geneva, Seoul, and Tokyo. The 34 National Committees are an integral part of UNICEF’s global organisation and a unique feature of UNICEF.7 UNICEF is funded exclusively by voluntary contributions, and the National Committees

4 https://www.unicef.org/executiveboard/. Accessed on 25 March 2022. 5 UNICEF Office of Research-Innocenti: https://www.unicef-irc.org/. Accessed on 25

March 2022. 6 UNICEF Supply Hub Accessed on 25 March 2022.

Copenhagen:

https://www.unicef.org/supply/about-us.

7 https://www.unicef.org/unicef-national-committees. Accessed on 25 March 2022.

438

K. RAFFII AND X. X. YANG

collectively raise around one-third of UNICEF’s annual income. Contributions from corporations, civil society organisations (CSO), and more than six million individual donors worldwide form the backbone of the organisation’s budget. The heart of UNICEF’s work is located in the field. UNICEF operates in more than 190 countries and territories through its country offices (CO) and special offices that support multiple countries. The majority of UNICEF staff work in CO. They support the planning, implementation and monitoring of country programmes in collaboration with national government, UN entities and other local partners. At UNICEF, CO are clustered into seven regions, each having a Regional Office (RO) that provides technical and managerial guidance as well as oversight of country programmes and budgets. These seven regional offices are EAPRO, Europe and Central Asia Regional Office (ECARO), Eastern and Southern Africa Regional Office (ESARO), Latin America and Caribbean Regional Office (LACRO), Middle East and North Africa Regional Office (MENARO), Regional Office of South Asia (ROSA), and West and Central Africa Regional Office (WCARO).8

3 Institutionalisation of the Evaluation Function Within UNICEF’s Organisational Structure The UNICEF Evaluation Office (EO) is headed by a director who reports to the Executive Director, with independent briefings to the Executive Board of the organisation during its annual sessions. The EO leads and coordinates the evaluation function within UNICEF and is an important member of the United Nations Evaluation Group (UNEG). Bringing together accountability and learning, the evaluation function plays a critical role to ensure the organisation delivers results for children by fostering evidence-based decision-making. At UNICEF, evaluation has intrinsic links to all stages of the programme/project cycle. Decentralisation is an important characteristic of the UNICEF evaluation system: Together with programme countries, UNICEF CO commission the bulk of UNICEF’s evaluation work.

8 Each regional office has its own public website which can be accessed here: https:// www.unicef.org/where-we-work. Accessed on 25 March 2022.

14

3.1

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

439

The Evolution of UNICEF’s Evaluation Function

The changes, evolution and institutionalisation of the evaluation function at UNICEF are documented in its Executive Board reports, and based on recommendations from the DAC-EVALNET/UNEG Peer Reviews, and reports on the organisational effectiveness of multilateral organisations, such as the Multilateral Organisation Performance Assessment Network.9 In the 1980s, UNICEF’s primary focus shifted from providing relief to promoting development. In April 1984, the Executive Board further clarified the relationship between monitoring and evaluation, requesting that “monitoring and evaluation (M&E) be strengthened in the field, regions and headquarters and that the quantity and quality of M&E be improved” (UNICEF, 1991, p. 3). The first meeting of UNICEF’s field staff who were in evaluation focal points was held in Florence in June 1989, which unfolded the discussion of particular fundamental issues related to evaluation, such as evaluative assessments of the UNICEF’s co-operation frameworks, funding, expertise, action and communication. During its 2002 annual session, the Executive Board of UNICEF adopted important decisions regarding its evaluation function. The Board “reiterates, in the context of the evaluation function, the principles of universality and country-driven programming for the activities of UNICEF, and emphasizes the importance of preserving the decentralized nature of the evaluation system in UNICEF” (Economic & Social Council Official Records, 2002, p. 99). Since then, two Development Assistance Committee (DAC)—UNEG professional peer reviews of UNICEF evaluation function have been conducted, first in 2006 and second in 2017. Both assessments aimed at analysing the strategic positioning of the evaluation function at the central and decentralised levels against the three core principles of independence, credibility and utility. The 2006 Peer Review 10 recognised the strength of UNICEF’s EO and the ongoing challenge of ensuring evaluation quality and adequate coverage in a decentralised system of programme evaluation. It observed

9 https://www.mopanonline.org/assessments/unicef2020/index.htm. Accessed on 25 March 2022. 10 https://www.alnap.org/help-library/peer-review-of-evaluation-function-at-unitednations-childrens-fund-unicef. Accessed on 25 March 2022.

440

K. RAFFII AND X. X. YANG

that “the central Evaluation Office demonstrates a high level of independence and produces evaluations that are credible and useful for learning and decision-making within the organisation. The decentralised evaluation system is appropriate for the operational nature of the organisation, but its credibility and usefulness are limited by critical gaps in resources” (Peer Review of UNICEF by Canadian International Development Agency, 2006 May, p. 2). A key recommendation of the 2006 Peer Review was that “UNICEF should update previous policy statements into a comprehensive evaluation policy document that is consistent with UNEG Norms and Standards11 and adapted to the present UNICEF context” (ibid., p. 11). This led to the preparation of the first evaluation policy for UNICEF, a stand-alone document approved by the Executive Board in 2008. The UNICEF Evaluation Policy (UNICEF, 2008) addressed some organisational constraints, paving the way to further realising the evaluation function’s potential to strengthen accountability and organisational learning. The measures included strengthened linkages among the EO, RO, and CO; improved strategic planning for evaluation; adequate resource allocation and improved mechanisms for quality assurance, management response and reporting on the evaluation function. The 2008 policy retained the decentralised system where responsibility for different types of evaluations rested at different levels in the organisation. The 2017 Peer Review 12 sought to further strengthen the UNICEF evaluation function “so that it is ‘fully-fit’ for purpose and well-placed to make the best contribution to the work and strategic positioning of the organisation, whilst furthering developments in the field of evaluation” (DAC/UNEG Peer Review, 2017, p. v). Per request of the Executive Board, UNICEF engaged a DAC/UNEG joint task force to conduct this second peer review of the UNICEF evaluation function, which started in October 2016 by mobilisation of a Review Panel and initiation of preparatory work. The Peer Review was to be forwardlooking, providing guidance on how the evaluation function can be further strengthened to meet emerging challenges and opportunities both within the UN system and more broadly. This peer review was carried out around the following topics: Evaluation policy of UNICEF; Governance

11 http://www.unevaluation.org/document/detail/1914. Accessed on 25 March 2022. 12 http://www.uneval.org/document/detail/2045. Accessed on 25. March 2022.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

441

arrangements; Management of the Evaluation Office, focusing on both operational arrangements and leadership and vision; Evaluation planning; Evaluation quality; Evaluation follow-up and use and External influence, Partnerships and Positioning. In its analysis, the 2017 Peer Review identified several weaknesses of the evaluation function that stemmed from the tension between two different views, namely: (1) evaluation as a shared and decentralised management tool contributing to evidence-generation, where its oversight function is not a primary concern and (2) independence of the evaluation function as a fundamental condition for its credibility and utility requiring that evaluation duties and responsibilities be clearly separate from any planning and management roles. Trusting that UNICEF would pick up the challenge, the Peer Review developed six major recommendations to institutionalise an independent and decentralised UNICEF evaluation function. These recommendations shape the current set-up of the evaluation function at UNICEF which is articulated in the Revised Evaluation Policy (UNICEF, 2018). This policy requires that all UNICEF evaluations be compliant with UNEG Norms and Standards (UNEG, 2016). 3.2

The Current Structure of Evaluation Function at UNICEF

The current UNICEF evaluation function is guided by the revised UNICEF Evaluation Policy, the UNEG Norms and Standards, Principles and Code of Conduct for Evaluation and was endorsed by the Executive Board in 2018. UNICEF EO reports to the Executive Board13 on the performance of the evaluation function and plans for evaluations throughout the organisation. The annual report encompasses the status of evaluation in all regions, including the EAP Region, the number and types of evaluations produced, progress in monitoring of evaluation activity and improving the 13 Overview of UNICEF Executive Board documents: https://www.unicef.org/execut iveboard/documents. Accessed 28 March 2022. For information on the EB session at its 2021 second regular session: https://www.unicef.org/executiveboard/media/8576/file/ 2021-7-Rev1-Board_report-EN-ODS.pdf, page 18 Section E for evaluations (Accessed on 25 March 2022). For other relevant examples of submissions please refer to https://www. unicef.org/executiveboard/first-regular-session-2022#session-documents (Accessed on 25 March 2022)—for example the 2022 Country Evaluation Plans submitted to the Board (Item 4) and the Plan for Global Evaluations (Item 8).

442

K. RAFFII AND X. X. YANG

quality of the evaluations produced. UNICEF maintains an Evaluation Dashboard14 that presents this same information on a real time basis. The Executive Board approves evaluation policies, budgets, as well as global and country costed evaluation plans (CEP) which are designed to inform the agency’s global strategic plans and country programme documents, respectively. The Executive Director of UNICEF15 safeguards the integrity of the evaluation function, creates demand for evaluation evidence, ensures that evaluation recommendations are acted upon, and supports the provision of human and financial resources. The Executive Director is also responsible for fostering a culture of learning and accountability at all levels of the organisation. Appointed by the Executive Director, in consultation with the Executive Board, the Director of Evaluation16 is accountable for the oversight of the evaluation function and leads the implementation of the evaluation policy, including conducting independent global evaluations and developing methodologies for evaluations. The Director of Evaluation has full discretion and control over the allocated resources. The Evaluation Office’s (www.unicef.org/evaluation/) portfolios include independent corporate evaluations (e.g., with a focus on specific Goal Areas of the organisation’s strategic plan); humanitarian evaluations; capacity development; partnerships and coherence and innovation, learning and uptake. At UNICEF, all final evaluation reports are qualityassessed by an external independent company specialised in assessing evaluation reports and production of meta-evaluation against the standards set in the Global Evaluation Reports Oversight System (GEROS).17 Furthermore, UNICEF offices must produce a formal management response in accordance with timelines set out in guidance issued by the EO. The EO monitors and reports on the implementation status of every evaluation management response. At UNICEF, all evaluation reports, 14 https://www.unicef.org/executiveboard/first-regular-session-2022#session-doc uments. Accessed on 31 March 2022. 15 https://www.unicef.org/media/experts/catherine-russell. Accessed on 25 March 2022. 16 https://www.unicef.org/media/experts/robert-mccouch. Accessed on 28 March 2022. 17 https://www.unicef.org/evaluation/global-evaluation-reports-oversight-systemgeros. Accessed on 25 March 2022.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

443

their GEROS ratings and management responses are made public at the UNICEF Evaluation Website and kept at a central database containing all evaluations the organisation commissions.18 Following the 2018 Revised Evaluation Policy, several modifications to the decentralised structure of the evaluation function were undertaken based on the recommendations of the 2017 DAC/UNEG Peer Review. For example, all regions established Regional Evaluation Advisors (REA) that would report to the Regional Directors, with a technical reporting line to the Evaluation Director. The previous practice in some regions where the REA (or head of an evaluation unit) reported to the Regional Planning Chief/Advisor was discontinued. Furthermore, the evaluation function established multi-country evaluation specialists (MCES) in all regions that had either direct or technical reporting lines to the REA. The MCES model was initially piloted for three EAP countries, namely Cambodia, Myanmar, and Malaysia. Through internal reviews and the 2017 Peer Review, the pilot phase was assessed as a success in showing improved independence, credibility, and utility of evaluations. Based on this assessment, it was replicated in all regions of UNICEF. The additional information on the work of MCES is presented under the “Decentralised Evaluation Function” below.

4

Relationship to National Policies in EAP Countries and Their Evaluation Practice

Stretching from Mongolia in the North to Tonga in the South, the EAP Region19 is the largest and most culturally, economically, and politically diverse among UNICEF regions. It is home to one-third of the world’s population, including more than one-quarter of its children. Based in Bangkok, Thailand, the EAPRO plays a key role in supporting the fourteen CO covering 13 East Asian countries and 14 Pacific Island countries. These country offices are at the frontline of the region’s work striving to improve the lives of children. The EAP-REF retains an oversight, technical assistance, and quality assurance role so that evaluations managed or commissioned by UNICEF (RO and CO) uphold the norms and standards of the organisation’s 18 https://www.unicef.org/evaluation/reports. Accessed on 25 March 2022. 19 https://www.unicef.org/eap/where-we-work. Accessed on 25 March 2022.

444

K. RAFFII AND X. X. YANG

evaluation policy. A Regional Evaluation Strategy was designed in 2017 (UNICEF, 2017; currently being updated) to help senior managers prioritise the evaluation function with a view of generating high-quality evidence that informs policy, programming, and advocacy and ultimately contributes towards better results for children in the region. The Strategy also intends to improve CO evaluation planning, budgeting, implementation, dissemination and use of findings. 4.1

Decentralised Evaluation Function

Guided by the results-based programming and implementation approaches, UNICEF programmes and projects contain evaluative activities inline with the norms and standards set in the Evaluation Policy. The 2018 Evaluation Policy clearly established the benchmark and coverage of CO-level evaluations, including cross-cutting issues. It provides “at least one country thematic evaluation, country programme component evaluation or project evaluation per year for each country programme. For small country programmes, evaluation frequency may be reduced to three per programme cycle” (UNICEF, 2018, p. 11). These commitments are reflected in Integrated Monitoring and Evaluation Plans (IMEP), tracked in the Evidence Information Systems Integration (EISI) platform of the organisation which describes the contents and budgets of all evaluations and evaluative activities. At the time when the Executive Board considers the Country Programme Documents (CPD) for each country, important evaluations are presented for endorsement through a CEP.20 According to the 2018 Revised Evaluation Policy, there are various types of evaluations that must be included in the CEP: Country Programme Evaluations (CPE), joint UN evaluations, country-led evaluations, country thematic evaluations and humanitarian evaluations. The subjects of the evaluations undertaken by CO reflect priorities for UNICEF at the country level. The decentralised nature of the evaluation function ensures that the evidence generated is relevant to the local context and, therefore, more likely to inform national policies for 20 CEP of each country office can be found at the UNICEF’s Executive Board website: Country programme documents | UNICEF Executive Board: https://www.unicef.org/ executiveboard/country-programme-documents please refer to the fourth column labelled CEP.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

445

children. These CO-level evaluations are planned and conducted in partnership with national authorities, addressing issues relevant to the national agenda.21 Importantly, the terms of reference (ToR) of an evaluation that are agreed upon by national counterparts and UNICEF should specify the intended use and the intended users of evaluation results. To generate national partners’ buy-in, local ownership of the evaluative activity, that in turn ensures the utility of evaluation reports, UNICEF CO engage the relevant counterparts from the design stage of an evaluation (e.g., drafting the ToR,) throughout the implementation (e.g., inviting counterparts to sit in the evaluation reference committees), to the validation and dissemination of the evaluation reports. Country Programme Evaluation in UNICEF fulfils a key role in identifying lessons which can inform the design of the next country programme or adjustments in the current country programme, and opportunities to improve UNICEF’s performance, including the contribution of the Programme of Cooperation to national development results. In the EAP Region, CPEs are conducted “at least once every two programme cycles, sequenced to feed into subsequent country programme document and United Nations Sustainable Development Cooperation Framework (UNSDCF). At least once per programme cycle if monitoring and audit information points to a significant shift in the programming context or a significant increase in the level of risk” (UNICEF, 2018, p. 11). To improve the independence of the evaluation function, the EAP Region added four MCES posts in 2020, to the existing MCES previously mentioned. Four of these MCES each cover three CO in the region, the fifth post supports the Pacific Islands and Papua New Guinea. The REA oversees the work of these MCES through a matrix management relationship. This matrix management relationship is to ensure the independence of the MCES work, as stipulated in the 2018 Revised Evaluation Policy. These staff advanced evaluation capacity at the country level, promoting efficiencies (including cost-sharing between CO), sharing learning and good practice, and anchoring the evaluation independence in the region (Polastro, 2022).

21 In a 2016 Resolution, the General Assembly reaffirms the central role and the importance of the active and full participation of national Governments in the preparation, implementation, monitoring and evaluation of the United Nations Development Assistance Framework, …. (General Assembly Resolution 71/243, para. 48, Link: https:// undocs.org/A/RES/71/243. Accessed 14 March 2023).

446

K. RAFFII AND X. X. YANG

The key accountabilities of MCES are as follows: (1) Evaluation Leadership: Support CO under his/her responsibility and their national partners to develop and implement a plan of evaluation activities that will provide relevant and strategic information to manage Country Programmes and relevant national policies and programmes towards achieving results for children in both humanitarian and non-humanitarian environments. (2) Evaluation Conduct and Use: Ensure that UNICEF-supported evaluations are designed and implemented to quality standards, and the results are disseminated and used by stakeholders in order to improve programme performance and contribute to wider learning; and (3) Evaluation Capacity Strengthening and Partnership Building: Ensure that evaluation capacities of CO staff and national partners—including government and civil society—are strengthened, enabling them to increasingly engage in and lead evidencegenerating processes. Ensure that the CO is effectively linked to wider UNICEF evaluation capacity-building developments in a way that both contributes to and benefits from organisational learning. The MCES are important posts that ensure independence, credibility, and utility of evaluations within the REF. During the COVID-19 pandemic, they also expanded their portfolios to support various forms of research and knowledge management initiatives within the region. 4.2

SDGs and Country-Led Evaluation

The adoption of the SDGs in September 2015 provided a common ground for collaboration in evaluation between UNICEF and its government counterparts. During the 2015 UNEG Norms and Standards for Evaluation High-Level Group Event, the former United Nations Secretary-General recognised that “evaluation is everywhere and, at every level, will play a key role in implementing the new development agenda” (UNICEF East Asia & Pacific, 2017 June, p. 1). SDGs implementation and monitoring processes are expected to be country-led, resulting

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

447

in an increased focus on country-led evaluation (CLE)22 work. This is particularly true for the EAP Region, as countries in the region have progressively transitioned from Low-income Countries to MiddleIncome Countries (MIC). With some EAP countries being at the high end of the MIC status, development support by UNICEF is becoming more upstream and policy-oriented. Development evaluations, which were mainly donor-led, are progressively becoming country-led and not exclusively led by national governments but also civil society. UNICEF CO works with Governments, UN entities and other partners in the EAP Region to advance CLEs to support the achievement of the SDGs, joint and system-wide evaluations and national evaluation capacity development, recognising the benefits of promoting learning, shared accountability and reduced transaction costs. There are some notable country-led and joint evaluations supported by UNICEF, including Evaluation of Cambodia’s National Scholarship Programme of the Ministry of Education, Youth and Sport; Myanmar’s Formative Evaluation of the Maternal and Child Cash Transfer Programme in Chin and Rakhine States; Evaluation of the Thailand National Child and Youth Development Plan (2012–2015); Formative Evaluation of the TimorLeste Youth Parliament Programme (2010–2018), and Evaluation of the Action Plan to Prevent and Respond to Violence Against Children in Cambodia (2017–2021). In the ToRs of some CLEs and evaluations where government partners are involved, UNICEF requests that the selected external evaluation team deliver a two- to three-day training on evaluation basics (and linkages with monitoring) at the inception phase. This is in support of the national evaluation capacity development efforts of UNICEF.

22 In an important document on the subject produced by UNICEF in partnership with DevInfo, IDEAS, IOCE, World Bank, UNECE, and MICS, Country-led Evaluation (CLE) is defined as evaluation “in which the country which is directly concerned leads and owns the evaluation process by determining: what policy or programme will be evaluated; what evaluation questions will be asked; what methods will be used; what analytical approach will be undertaken; and how the findings will be communicated and ultimately used. CLE serves the information needs of the country and, therefore, CLE is an agent of change and instrumental in supporting national development results” (UNICEF, 2009). Country-led monitoring and evaluation systems: Better evidence, better policies, better development results, p. 9. http://mymande.org/sites/default/files/ima ges/Country_ledMEsystems.pdf. Accessed on 22 February 2022.

448

K. RAFFII AND X. X. YANG

4.3

National Evaluation Capacity Development

A national evaluation system (NES) is composed of the institutions, people and activities, as well as the regulatory elements and relationships, involved in a country’s evaluation demand, supply and use. National evaluation capacity development (NECD) is a key workstream under the REF. The degree to which evaluation is institutionalised varies among EAP countries. There may or may not be a national policy or other regulations governing national evaluation processes, and the standardisation and implementation of evaluation policies also vary. Given the diversity of country contexts, capacities, and varying approaches to NECD, it is difficult to distinctly categorise countries against NECD progress, or indeed be able to measure progress in a standardised manner (UNICEF, 2021, p. 3). To better understand the status of national evaluation systems and capacities, between 2017 and 2019, UNICEF and UNDP jointly conducted a Review of National Evaluation Systems and Capacities for Evaluating Progress towards the Sustainable Development Goals in Asia and Pacific Region (UNICEF/UNDP Joint Review, 2019). The objective of the Joint Review was to generate a body of knowledge to guide strategies in the region and beyond for NECD initiatives adapted and responsive to the 2030 Agenda. As part of this Joint Review, case studies were carried out in seven countries, namely Indonesia, Malaysia, Mongolia, Philippines, Sri Lanka, Thailand, and Vietnam.23 The key findings of this Joint Review are grouped under key headings, as shown below in Box 1 (Librado & MacLean, 2019, p. ii)24 :

23 Each of the case studies was conducted by national consultants with guidance from UNICEF and UNDP regional and country offices and senior international consultants, drawing on a shared conceptual framework and methodology. National peer review committees of UNICEF and UNDP country office staff and other experts provided additional input during case study drafting. 24 It should be noted that the COVID pandemic has further complicated the situation in the region. As a result of the bottlenecks that the pandemic created, a number of the above findings are negatively impacted.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

449

Box 1 Information paraphrased from the Joint Review On the Enabling Environment . There was a strong commitment to the 2030 Agenda and a growing commitment to evaluation, both for the SDGs and in general. . Increased demand for evaluation from citizens and mechanisms facilitating citizen engagement and progress (albeit mixed) on the openness of government to evaluations could be identified. . Evaluative thinking and learning culture were emerging. . There were insufficient financial resources for evaluation to support SDGs evaluation needs. . Feedback loops to ensure the use of evaluation results in support of the 2030 Agenda was weak. On Institutional Capacity . Key NES institutional structures and mechanisms supporting the 2030 Agenda were found to be in place. . Harmonisation and coordination of various NES efforts were limited. . Institutional capacity to produce high-quality, complete, reliable and disaggregated data for evaluation for the 2030 Agenda was weak. . Variable progress was found in promoting equity-focused and gender-responsive practice and evaluation. On Individual Capacity . Overall, individual capacity for evaluation of the SDGs was limited and uneven. . Opportunities for professional evaluation capacity development vary across and within the case study countries. Overall, these opportunities were limited but increasing. Source Own illustration, Librado and MacLean (2019, p. ii).25

The case study countries show evidence that the institutional building blocks for a well-functioning NES have been created. Taking national evaluation policy (NEP) as an example, the Philippines and Malaysia are 25 Please also refer to Polastro and Prokop (2017); Review of National Evaluation systems and Capacities for Evaluating Progress Towards the Sustainable Development Goals: Emerging Lessons Learned from the Asia–Pacific Region (https://nec.undp.org/ sites/default/files/2021-07/ReviewofNationalEvaluation2017.pdf. Accessed on 25 March 2022.)

450

K. RAFFII AND X. X. YANG

the only case study countries that have a formal NEP in place, while the five other EAP case study countries are without NEP. Nevertheless, all have well-established institutional and regulatory frameworks for evaluation in the public sector. However, the Joint Review also pointed out, “it is not always apparent that these systems intersect, or that the need for their intersection has received significant consideration. Further progress in this area is needed to position country-led evaluation to fully meet its potential to contribute to the 2030 Agenda” (Librado & MacLean, 2019, p. 19). Building national evaluation capacity continues to be an important element of the UNICEF’s response that is led by the CO, with support from the MCES and the REA. This support aims to “enhance the provision of timely evidence at the country level on the attainment of the Sustainable Development Goals and the fulfilment of the rights of all children” (UNICEF, 2018, p. 23). More concrete NECD initiatives will be discussed in Sect. 5, “Contributions to the professionalisation of evaluation” Our next section will focus on civil society’s role, particularly, the role of Voluntary Organisations for Professional Evaluation (VOPEs) in enhancing evaluation in the EAP Region.

5

Relationship with Civil Society

National non-governmental evaluation actors play important roles in both supply and demand sides of evaluation. UNICEF has a long tradition in supporting VOPEs. In 2012, UNICEF and the International Organization for Cooperation in Evaluation (IOCE) launched the EvalPartners initiative to build and strengthen VOPE networks and relationships worldwide. In November 2015, the EvalPartners launched the “Global Evaluation Agenda 2016–2020” to address priorities for evaluation during the first five years of the 15-year period of the newly introduced SDGs. Since 2010, the EAP Region has seen fledgling and emerging VOPE networks. By the end of 2021, 10 EAP-based VOPEs were registered as IOCE members, including one regional VOPE, namely Asia Pacific Evaluation Association (APEA), and nine national VOPEs from eight countries.26 Almost all these VOPEs were founded in or after 2009, 26 These VOPEs include Cambodian Evaluation Society, Indonesian Development Evaluation Community, Lao Evaluation Network, Malaysian Evaluation Society, Mongolia

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

451

except the Malaysian Evaluation Society (MES) that was established in 1999. They aim to promote both supply and demand sides of evaluation practice by strengthening individual capacity and developing VOPE institutional capacity (supply side) and promoting national evaluation systems and policies and the use of evaluations (demand side). The membership of majority VOPEs is open to any individual who pays dues and accepts to abide by the VOPE Ethics Code. As explained by the Mongolia Evaluation Network, “As we are a fairly new VOPE, we do not set strict criteria for membership. Anyone who is interested in evaluation, would like to use it in their work, and supports evaluation practice can join the VOPE”.27 As a result, the VOPEs bring together evaluation practitioners from various fields, including government, academia and non-governmental organisations and private sector. By advancing their missions, these VOPEs contribute not only to the benefits of their members but to the benefits of society at large as well. Acquiring adequate capacities to make significant contributions to the development of the NES and bring influence on public policies are the objectives of many VOPEs. Their current self-assessment in the IOCE directory shows that only MES claims that it is able to influence government policies and systems in a significant manner. The MES also admitted that “on evaluation being practised by private sector organisations …, there is a dearth of information to shed light on the state of affair. While it is safe to state that civil society organisations are more involved in the evaluation of funded programmes and projects as these evaluation exercises are mostly donor-driven [MES, 2016, p. 2]”. Other VOPEs are either carrying out advocacy in a modest way (six out of ten) or working on their capacity to develop effective policy advocacy strategies (three out of ten), including finding ways to mobilise resources for advocacy. Similarly, six out of ten of these VOPEs have initiated building a limited partnership with parliamentarians. Table 1 below presents information on VOPE in the region, including the year founded, legal status and a non-exhaustive list of member categories. The APEA is a regional entity founded in 2012, thus not included in Table 1.

Evaluation Network, Myanmar Monitoring and Evaluation Association, Philippine Evaluators for Development, Inc., M&E Network Philippines, and Vietnam Network for M&E. 27 Quote from a member of the Mongolia Evaluation Network.

Formal association/ society

Informal network Formal association/ society Informal network Informal network

2009

2016

2016

2016

1999

Formal association/ society

2009

Cambodian Evaluation Society (CamES) Indonesian Development Evaluation Community (InDEC) Lao Evaluation Network Malaysian Evaluation Society (MES) Mongolia Evaluation Network Myanmar Monitoring and Evaluation Association

Status

Year founded

1

3

13

5

20%

10

Gov’t

38

3

1

2

10%

22

CSOs

2

3

6

0

5%

2

Academia

Members include the following

National Representative VOPEs as an IOCE Member

VOPE

Table 1

1

7

6

2

65%

6

Private sector

Yes, but in a modest way Yes, in a significant way No, but we’re working on it No, but we’re working on it

Yes, but in a modest way

Yes, but in a modest way

Influence government policies & systems?

Yes, a little

No (at least not yet)

Yes, a little

Yes, a little

No (at least not yet)

Yes, a little

Partner with parliamentarians?

452 K. RAFFII AND X. X. YANG

Year founded

Source Own work

Philippine 2011 Evaluators for Development, Inc M&E Network 2011 Philippines Vietnam 2012 Network for M&E (VNME)

VOPE

3 20

50 5

Informal network Informal network

4

CSOs

10

Gov’t

5

1

9

Academia

Members include the following

Formal association/ society

Status

25

5

6

Private sector

No (at least not yet)

Partner with parliamentarians?

Yes, but in a No (at least not modest way yet) No, but Yes, a little we’re working on it

Yes, but in a modest way

Influence government policies & systems?

14 UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

453

454

K. RAFFII AND X. X. YANG

To generate a stronger impact, the VOPEs have formed partnerships among themselves. An ASEAN Regional Framework on Evaluation Standards (ARFES) was developed by APEA, MES, Cambodia Evaluation Society, Vietnam Network of Monitoring and Evaluation, Indonesia Development Evaluation Community, and Malaysia University of Science and Technology. ARFES was supported by a grant from EvalPartners’ P2P Programme, which is one of its flagship VOPE supporting programmes. ARFES aims at closing the demand–supply gap in evaluation caused by increasing demands for evaluation and the constrained supply of evaluation expertise and standards. This mismatched situation was caused by the absence of national evaluation standards among ASEAN member countries. As a regional framework, ARFES’ strategic intent is to offer a common generic and non-prescriptive framework with broad outlines to enable each country to draw up a respective national framework of evaluation standards for developing national evaluation capacity as well as for the professionalisation of evaluation (EvalPartners, 2016, p. 6).

6 Contributions to the Professionalisation of Evaluation 6.1

Global Framework

UNICEF is well known in the international evaluation community for its commitment to NECD and its support to international evaluation networks. UNICEF has served as an important member of the UNEG working groups on professionalisation and national capacity development for many years. UNICEF EO is committed to improving its decentralised Evaluation Function by addressing the three key pillars of the evaluative capacity of the organisation, namely (1) evaluation staff knowledge, skills and competences, (2) the enabling environment and organisational culture towards evaluation and (3) the facilitating institutional framework

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

455

(UNICEF, 2022b, p. 1).28 Various evaluation training courses are available on UNICEF’s AGORA platform29 to the general public, as well as tailored to the needs of UNICEF staff and our partners, including those in the region. Recently, a specialised blended course (online and face-to-face) has been developed by UNICEF. This Evaluation Blended Learning Programme, composed of courses at fundamental, intermediate, and advanced levels, aims to provide UNICEF staff and partners with the requisite technical skills and competencies, so that they can ensure the delivery of credible evaluation evidence in a timely manner, to help improve learning and accountability in UNICEF, or UNICEF counterparts. The Programme is open to partners (government counterparts, VOPEs, National and Regional Evaluation Associations, and academia), as it aims to foster a working partnership in evaluation with key government sectors and promote greater attention to evaluation in public administration. In collaboration with the non-profit organisation “Better Evaluation”, the UNICEF EO rolled out the course in April 2021 labelled “Intermediate evaluation course for staff and partners”. The April course trained the first cohorts of the course, the second cohort started the course in September of 2021. As of the writing of this article, twenty EAP Region evaluation focal points at the multi-country and country level have graduated from the course and are ready to share their knowledge in their respective countries. This is done through webinars; moreover, YouTube videos are developed and one on one support is provided to colleagues at the country level. This course continues to evolve and is increasingly adapted to the special environment with which UNICEF is currently confronted in the region.

28 Please note following text from UNICEF EO: “(i) evaluation staff knowledge, skills and competences, which are essential to perform tasks and manage processes and relationships; (ii) the enabling environment and organizational culture towards evaluation, which provides a context that fosters the performance and results of individuals and organizations; and (iii) the institutional framework which provides systems and structures in which individuals can increase their performance and whereby UNICEF can attain greater result collectively” (UNICEF, 2022b, p. 1). 29 https://agora.unicef.org. Accessed on 25 March 2022.

456

K. RAFFII AND X. X. YANG

6.2

Regional NECD Initiatives

In the EAP Region, UNICEF’s REF is dedicated to supporting the professionalisation of evaluation within and beyond the UN, at both institutional and individual levels. Within the UN system, UNICEF REA is an active member of the UN Evaluation Group in Asia and Pacific (UNEDAP), a regional inter-agency network aiming to promote an evaluation culture and contribute to UN coherence on evaluation. UNEADP’s annual course of “Unravelling the potential of Evaluation” is open to a limited number of evaluation focal points from different UN agencies, specifically those commissioning and managing evaluations. By combination of live sessions, collaborative and personal practical exercises, deep dives into rich and diverse resources and guidance from facilitators, this online course aims to enhance the quality and effectiveness of evaluations in the UN system.30 UNICEF REF also provides technical support to UN-wide evaluations in the region. During the past two years, the REA, supported by the MCES, has acted as UNEDAP focal point for joint UN evaluations in DPRK, the Pacific region and Thailand. Some MCES are members of the M&E groups led by the UN Resident Coordination Offices. The MCES involvement with joint UN activity ranges from the development of the UNSDCF results framework, UN joint work plans, providing support to evaluation coordination and technical support, and when requested, managing UNSDCF and joint UN evaluations. At the country level, REF directly supports NECD. Realising the elevated significance of evaluation function in measuring and achieving SDGs, the REF prioritises NECD initiatives that engage government and development partners at a decentralised level. In April and May 2021, the REA and MCES organised a series of five one-hour Learning Webinars on Evaluation. These sessions covered the topics mostly needed at the country level and included: Basics of evaluation, Evaluation management, Introduction to NECD, Use of evaluations, and Joint UN evaluation. Conducted in English with simultaneous translation to Mongolian and

30 Example of a UNEDAP 2021 training is open to the general public can be found using the following link: https://padlet.com/Animarnos/7x0u2wszg578y2n. Accessed 25 March 2022.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

457

Mandarin, these webinars attracted approximately two hundred participants from the region.31 The Webinars benefited from interaction and cross-fertilisation among participants who had different levels of knowledge and skills in evaluation. The above-mentioned Learning Webinars are available to the general public through Global Development Commons website.32 UNICEF is currently transforming the webinar sessions into a YouTube learning series in different local languages used in the EAP Region, and redesigned to support the training of young evaluators. Moreover, the REF provides technical support and facilitation to the individual CO on an ongoing basis. For example, in September 2021, the REA moderated a webinar with parliamentarians in the Philippines on Institutionalising a National Evaluation Policy Framework.33 In addition to using YouTube as a new learning platform, the REF is actively embracing social media as an innovative tool to promote timely, widespread dissemination of evaluation findings and recommendations. The ToR of many evaluations commissioned in the EAP Region has required the external evaluation teams to produce a short video reflecting evidence-based findings and recommendations as a key output of the evaluation assignments. Compared with traditional knowledge dissemination channels, LinkedIn, Twitter, and Facebook, as well as various UNICEF internal and external online platforms, help to further improve the effectiveness and efficiency of evaluation dissemination and use among internal and external stakeholders. Furthermore, the REF collaborates with CO to disseminate internally and externally emerging knowledge and good practices in evaluation from the region, through webinars; videos and by presenting these at international conferences (Polastro, 2022). Evaluate, a quarterly newsletter jointly produced with ROSA, and the Monday Morning Food for Thought messages, mainly disseminated among China, DPRK, and Mongolia CO, represent the past and current examples of these efforts. To contribute to

31 Among the participants 58.3% were government staff, 25% were from UNICEF country offices, 8.3% were from CSOs, and 8.3% were from academia. According to a post-training survey, all participants were satisfied with the session; specifically, 75% of participants found the training very useful and 25% fairly useful. 32 https://gdc.unicef.org/resource/unicef-east-asia-and-pacific-evaluation-learning-web inars. Accessed on 25 March 2022. 33 https://cpbrd.congress.gov.ph/events/189-2021-event-materials/1402-all-eventsarticle-9. Accessed on 25 March 2022.

458

K. RAFFII AND X. X. YANG

regional evaluation capacity building and the professional dialogue, the REAs and MCES are also regular speakers at the Asia Evaluation Week,34 a leading evaluation knowledge-sharing event in the EAP Region.

7

Future Vision for UNICEF to Strengthen Evaluation Function in EAP Region

As indicated previously, the UNICEF Strategic Plan 2022–2025 identifies data, research, evaluation and knowledge management as a key element of the nine game-changing operational strategies (UNICEF, 2022a, p. 17). Based on the 2017 Peer Review Report, “The potential and opportunity exist for UNICEF to better use the Evaluation Function for strategic corporate evaluations and partnerships that look beyond programme sectors and sections to support effective corporate governance, greater synthesis of knowledge and lessons (with research arm of the organisation), risk management (with audit section), and strategy and leverage of private sector lessons and investments” (DAC/UNEG Peer Review, 2017, p. 58). The EAP-REF is working on a proposal on NECD to further solicit donor engagement and support in this area. Realising that UNICEF’s support to NECD to date has often lacked an overall regional level strategy and has been driven at individual country level, the new proposal seeks to address these weaknesses through: . Adopting a regional approach that will attempt to find synergies and efficiencies through working with government and nongovernment organisations from multiple countries facing similar developmental needs; . Adopting a capacity-building approach that will focus more on an institutional, systems-strengthening initiative; . Seeking synergies and efficiencies through supporting cadres of trainees from multiple countries in training and support so that sharing experience can be emphasised and . Identifying and working with a small number of VOPEs to manage regional and in-country initiatives.

34 https://asianevaluationweek.org/about. Accessed on 10 April 2022.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

459

The five MCES in the EAP Region are going to be key leaders and champions with relation to the regional NECD initiative. At the global level, UNICEF held consultations with key headquarters and regional evaluation staff and drafted a new vision for a forwardlooking evaluation agenda. A white paper has been issued in March 2022 that provides for the following vision (see Box 2): Box 2 Quote from White Paper of UNICEF Evaluation Function (UNICEF, 2022c) The vision of the evaluation function is to consistently leverage rigorous, strategically prioritised evaluative evidence for UNICEF and its partners to realise the rights of every child in the Decade of Action. To realise it in a timely and comprehensive manner, we will need a multitude of approaches. Below are some of our preliminary plans that describe how we will “walk the talk” in bringing these principles to life in concrete ways in our day-to-day work. 1. Strengthened emphasis in outcome- and impact-level results measurement 2. Enhanced and better-prioritised evaluation partnerships 3. Better-coordinated approaches to national evaluation capacity development 4. More proactive communications and advocacy surrounding the evaluation function 5. More harmonised coordination and collaboration with complementary functions 6. Enhanced knowledge management to leverage evaluative evidence for children 7. Enhancements to our data and information management systems 8. Broader, more fit-for-purpose and representative bank of evaluation expertise 9. Strengthened guidance for consistent excellence of quality and utility in all evaluations 10. Together with management, more robust systems for ensuring follow-through on evaluation recommendations 11. Increased accountability within the evaluation function itself

460

K. RAFFII AND X. X. YANG

In summary, a corporate independent evaluation function is solidly anchored in the organisation. As confirmed in the 2020 MOPAN Assessment of UNICEF,35 “UNICEF has a strong evaluation function with a high degree of independence. Its evaluation plan is well funded and evaluation coverage has increased over time. UNICEF has a robust system to ensure the quality of its evaluations and has made efforts to increase the capacity of staff in evaluation” (MOPAN, 2021, p. 39). At the decentralised level, the EAP Region has successfully built up an evaluation structure customised to the needs of the region, composed of evaluation staff at the regional, multi-country, and country levels. The region has seen substantial progress in evaluation coverage, quality and budget use in recent years, though unevenness in these achievements still exists among CO. Guided by the above global vision, the REF is currently reassessing its regional strategy to further innovate and revamp the function. Cultivating an appetite and culture for evaluation among UNICEF staff, counterparts, and partners will still be a priority of the revised strategy. The REF will advance its two-pronged approach to strengthening both the demand and supply sides of evaluation through Country Led Evaluations, Seminars, Webinars and other NECD tools. Collaborating with other development partners, the REF will continue to monitor and adjust its direction to meet the evolving social, economic and political circumstances in the region and ensure “no one is left behind” in achieving the SDGs for children.

Appendix See Figs. 1 and 2.

35 https://www.mopanonline.org/assessments/unicef2020/. Accessed on 8 April 2022.

Fig. 1 Organisation chart of the evaluation function in EAPRO (Source Author’s own illustration)

14 UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

461

462

K. RAFFII AND X. X. YANG

Fig. 2 UNICEF strategic plan 2022–2025

References Canadian International Development Agency. (2006). Executive summary of peer review of evaluation function at United Nations Children’s Fund. https://www.alnap.org/help-library/peer-review-of-evaluationfunction-at-united-nations-childrens-fund-unicef. Accessed on 22 February 2022. DAC/UNEG Peer Review of the Evaluation Function of UNICEF. (2017). Final peer review document. http://www.uneval.org/document/detail/ 2045. Accessed on 22 February 2022. Economic and Social Council Official Records. (2002). Supplement No. 14. Executive Board of the United Nations Children’s Fund. Report on the first and second regular sessions and annual sessions of 2002. https:// www.unicef.org/executiveboard/media/1381/file/2002-8-Rev1-Board_rep ort_annex-Compendium_of_decisions_2002-EN-ODS.pdf. Accessed on 22 February 2022.

14

UNICEF SUPPORT TO INSTITUTIONALISING EVALUATION

463

EvalPartners. (2016). 2015–2016 Small Grant Award Winner reports. https:// evalpartners.org/sites/default/files/documents/EvalPartners-Grant%20U pdate%20Newsletter.pdf. Accessed on 22 February 2022. Librado D., & MacLean M. (2019). Review of national evaluation systems and capacities for evaluating progress towards the sustainable development goals in Asia and the Pacific. Synthesis Report of Indonesia, Malaysia, Mongolia, Philippines, Sri Lanka, Thailand, and Vietnam Case Studies. Malaysian Evaluation Society. (2016). Draft (revised) Malaysian Framework for Evaluation Policy and Standards (MyFEPS)—Evaluation for sustainability. https://www.evalpartners.org/sites/default/files/documents/p2p/SGP1510/MALAYSIAN%20FRAMEWORK%20FOR%20EVALUATION%20P OLICY%20AND%20STANDARDS%20-%20MyFEPS%20-%20Evaluation% 20for%20Sustainability.pdf. Accessed on 22 February 2022. Multilateral Organisation Performance Assessment Network. (2021). MOPAN assessment report: UNICEF 2020 assessment cycle. https://www.mopano nline.org/assessments/unicef2020/. Accessed on 8 April 2022. Polastro R., & Prokop, M. (2017). Review of national evaluation systems and capacities for evaluating progress towards the sustainable development goals: Emerging lessons learned from the AsiaPacific Region. https://nec.undp.org/sites/default/files/2021-07/Review ofNationalEvaluation2017.pdf. Accessed on 22 February 2022. Polastro, R. (2022). Anchoring UNICEF’s evaluation function in East Asia and the Pacific. Evaluation and Program Planning, 91, 102003. UNEG. (2016). Norms and standards for evaluation. Adopted in 2005, revised 2016. UNEG, United Nations Evaluation Group. http://www.unevaluation. org/document/detail/1914. Accessed on 25 March 2022. UNICEF Mission statement. (n.d.). https://sites.unicef.org/about/who/index_ mission.html. Accessed on 22 February 2022. UNICEF. (1991). A UNICEF guide for monitoring and evaluation: Making a difference? https://www.corecommitments.unicef.org/kp/a-unicef-guidefor-monitoring-and-evaluation-making-a-difference. Accessed on 22 February 2022. UNICEF. (2008) UNICEF evaluation policy. https://digitallibrary.un.org/rec ord/615249?ln=en. Accessed on 25 March 2022. UNICEF. (2009). Country-led monitoring and evaluation systems: Better evidence, better policies, better development results. http://mymande. org/sites/default/files/images/Country_ledMEsystems.pdf. Accessed on 22 February 2022. UNICEF East Asia and Pacific. (2017). Regional evaluation strategy 2018–2021 UNICEF East Asia and the Pacific. https://www.unicef.org/eap/sites/uni cef.org.eap/files/2019-11/Regional%20Evaluation%20Strategy%20and%20A ction%20Plan_latest.pdf. Accessed on 22 February 2022.

464

K. RAFFII AND X. X. YANG

UNICEF. (2017). Evaluation management response to the 2017 professional peer review. http://www.uneval.org/document/detail/2045. Accessed on 22 February 2022. UNICEF. (2018). Revised evaluation policy of UNICEF 2018. https://www. unicef.org/evaluation/documents/revised-evaluation-policy-unicef-2018. Accessed on 22 February 2022. UNICEF. (2021). Draft EAPRO NECD proposal concept note for discussion with MCES & Regional Office. UNICEF internal document. UNICEF. (2022a). Strategic plan 2022a to 2025; Renewed ambition towards 2030. https://www.unicef.org/media/115646/file/Strategic%20P lan%202022-2025%20publication%20English.pdf. Accessed on 22 February 2022. UNICEF. (2022b). UNICEF evaluation capacity development for countries (Presentation Series). UNICEF. (2022c). Leveraging better evidence for every child: The vision for a next-generation evaluation function for UNICEF in the Decade of Action.

CHAPTER 15

The Influence of Transnational Organisations: A Comparison of Effects in America, Asia and Europe Wolfgang Meyer

1

Introduction

The three published volumes of the GLOBE project on the Americas, Asia and Europe include six chapters on transnational organisations supporting national development of evaluation in the region. The selection of these organisations derived from discussions with national authors and suggestions from regional experts. This chapter provides a more systematic overview of the contributions from transnational organisations to the institutionalisation of evaluation in a global perspective. If one takes this viewpoint, United Nations Organizations come into focus as powerful actors with a global institutional network. Very close to this group of transnational organisations are the development banks, established to support national development in a broad variety of fields. Other actors like the Organisation for Economic Co-operation and Development

W. Meyer (B) Saarland University, Saarbrücken, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_15

465

466

W. MEYER

(OECD) or the European Union (EU) have also made important contributions. While the number of transnational organisations is huge and includes a broad variety of different types (e.g. governmental and nongovernmental organisations, associations and foundations, formal and informal networks), it is impossible to give a total overview on all types of organisations and all possible impacts. However, the starting point is an overview of main transnational actors and particular networks on evaluation build (and supported) by them. There are some organisations that are forerunners and key players in the field of evaluation, but their most important role is probably the interaction with other transnational, regional, and national actors for building up networks fostering evaluation. These more or less globally working networks will be described in the first part of this paper. The second part is on the transnational organisations themselves and the institutionalisation of evaluation within the organisational framework. Comparable to the analysis of the Nation states, both the structural implementation and the evaluation practices will be presented in a comparative way. The comparison is less targeting on an interregional than an interorganisational perspective because many of the transnational organisations are not limited to regional activities but act on a global level. The third part focuses on the activities of transnational organisations to support Nation states and national civil societies on behalf of the implementation and use of evaluation within this national framework. Many transnational organisations are explicitly active in Evaluation capacity development and support the professionalisation of evaluation in a more specific way. This part emphasises on National Monitoring and Evaluation Systems (NMES), Voluntary Organisations of Professional Evaluations (VOPES) and Evaluation Capacity Development (ECD) as main topics of these activities. The key objective of this chapter is to summarise the most important influences from transnational organisations on the institutionalisation of evaluation in a descriptive way. There will be no in-depth analysis why some transnational organisations or some activities causes effects in a particular country or region, nor will there be a comparison between the three regions America, Asia and Europe.

15

2

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

467

Evaluation Networks and the Role of Transnational Organisations

Globally speaking, the most important network of transnational organisations is surely the United Nation System. The United Nations is an intergovernmental organisation with 193 member states—almost all existing Nation States (there are only 10 exceptions of countries that are not acknowledged by all others like Taiwan, Palestine or Kosovo). It is composed of six principal organs (e.g. the General Assembly, the Security Council or the Economic and Social Council), seventeen specialised organisations (e.g. the Food and Agricultural Organization FAO, the United Nations Educational, Scientific and Cultural Organization UNESCO or the World Bank Group WBG), eight funds and programmes (e.g. United Nations Development Programme UNDP, United Nations Children’s Fund UNICEF or United Nations Environmental Programme UNEP), six research and training institutes (e.g. United Nations Institute for Training and Research UNITAR, UN Women or United Nations University UNU) and further related entities and bodies (e.g. United Nations Conference on Trade and Development UNCTAD, International Organization for Migration IOM or the World Trade Organization WTO). Although the UN-System is very heterogeneous and diverse in its organisational form, there is some good information on the institutionalisation of evaluation in the whole system available. This is the result of activities by the United Nations Evaluation Group (UNEG), a network built as Inter-Agency Working Group on Evaluation in 1984, including nowadays 45 evaluation offices as members and 3 more (including the World Bank) as observers. UNEG follows an own strategy (which is nowadays influenced by the Sustainable Development Goals, UNEG, 2019), developed several norms, rules and guidelines that are broadly used in the UN-system (e.g. UNEG, 2016, 2020a, 2021) and offers a wide spectrum of publications including evaluation reports and more (see for details http://www.unevaluation.org/document/library). Although UNEG is already quite old, it is not the first and not the only global network on evaluation. In development cooperation, the Organization for Economic Co-operation and Development and particular its Development Assistance Committee (OECD-DAC) played a leading role in building up standards and exchange platforms on evaluation. The attempt to rebuild Europe after the Second World War led to the OECD,

468

W. MEYER

formed in 1961 as a replacement for the 1948 funded Organisation for European Economic Co-operation OEEC, by twenty merely European countries and with the goal “to shape policies that foster prosperity, equality, opportunity and well-being for all” (https://www.oecd.org/ about/). While most of the donor states are members of the OECD, development cooperation became one of the main topics of the organisation and the Development Assistance Committee was even formed before the OECD in 1960. The main goal is the harmonisation of donor activities and “to promote development co-operation” (https://www.oecd. org/dac/development-assistance-committee/). From its very early beginning, the DAC organised high-level meetings and formed a network on development evaluation that includes besides the bilateral members nine multilateral agencies (DAC EvalNet, https://www.oecd.org/dac/evaluation/encouragingg oodpracticeindevelopmentevaluation.htm). Among the transnational members of this evaluation network are the independent evaluation units of the World Bank and the regional development banks (Asian Development Bank, Interamerican Development Bank, and African Development Bank). There are regular meetings of high-level representatives from evaluation offices at ministries and agencies that discuss important evaluation topics and provide certain publications and information on evaluation issues. The most important outputs of this network are the DAC Criteria for Evaluating Development Assistance (https://www.oecd.org/ dac/evaluation/39119068.pdf) and the Glossary of Key Terms in Evaluation and Results-Based Management (https://www.oecd.org/dac/eva luation/dcdndep/43184177.pdf). Both documents are developed originally in the 1990s. The last revision of the Evaluation Criteria has been approved in 2019. Both documents are still widely used in the field and accepted as key orientation for development evaluation. In 1996, the Evaluation Cooperation Group (ECG) was formed for “harmonizing evaluation work among multilateral development banks (MDBs) by working to strengthen the use of evaluation” (https://www. ecgnet.org/) and includes as founding members the African Development Bank (AfDB), the Asian Development Bank (ADB), the European Bank for Reconstruction and Development (EBRD), the Interamerican Development Bank (IDB) and the Independent Evaluation Group (IEG) at the World Bank Group (see for the development West Meiers, 2022, p. 431f.). The members established a secretariat and a rotating chairmanship with meetings twice a year. Two of the most important documents

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

469

published by ECG are the Big Book on Evaluation Good Practice Standards (ECG, 2012) and the Review Framework for the Evaluation Function in Multilateral Development Banks (ECG, 2009). There are several other networks on evaluation with the task to organise the exchange on evaluation between different actors that are initiated or at least supported by transnational organisations. Three of these networks should be additionally mentioned here: EvalPartners, GEI and CLEAR. Starting point of the formation of EvalPartners as a global partnership between Civil Society- and UN-Organisations was the initiative of six evaluation societies (American Evaluation Association, Associazione Italiana di Valutazione, Australasian Evaluation Society, Canadian Evaluation Society, Kenyan Evaluation Association, and the United Kingdom Evaluation Society) to form an International Organization for Cooperation in Evaluation (IOCE), which was formally endorsed in 2001 by 24 evaluation associations and networks. This organisation became the global branch of so-called Voluntary Organisations for Professional Evaluation (VOPE) and aimed “to increase public awareness and globally validate evaluation, and support VOPEs in contributing to good governance, effective decision making and strengthening the role of civil society” (https://ioce.net/). Together with the evaluation office of UNICEF, IOCE launched the EvalPartners initiative “to build and strengthen the global network of relationships between existing and emerging VOPEs” which was established in 2012 by IOCE, UNICEF and some other UN-Organisations” to influence policymakers, public opinion, and other key stakeholders into recognizing the need on effective evaluation as necessary to any important decision” (https://www.evalpartners.org/). EvalPartners organised several influential events like the Global Evaluation Agenda 2016–2020, the Chiang Mai Declaration and an UN-General Assembly resolution on country-level evaluations. Most recently in 2020, another Global Evaluation Initiative (GEI) has been started as “a global network of organizations and experts supporting developing country governments with strengthening monitoring, evaluation, and the use of evidence in their countries,” focusing “on efforts that are country-owned and aligned with local needs, goals and perspectives” (https://www.globalevaluationinitiative.org/who-we-are). Main drivers of this initiative had been the two independent evaluation offices of the World Bank Group (IEG) and UNDP (IEO). Today, 26 UN agencies (like UN Women or the World Food Program WFP), development banks

470

W. MEYER

(like the Asian Development Bank ADB or the Interamerican Development Bank IDB), global networks (like the International Initiative for impact evaluation 3iE or the International Program for Development Evaluation Training IPDET), national ministries (like the Federal Ministry for Economic Cooperation and Development of Germany BMZ or the Ministry of Foreign Affairs of Denmark DANIDA) and national development agencies (like the Swiss Agency for Development and Cooperation SDC) are part of GEI. The network focuses on 24 priority countries where partnerships with national authorities have been established (from Bhutan to Zambia) and the task is to help them to improve their National Monitoring and Evaluation Systems. One of the GEI-members is the CLEAR Initiative (the Regional Centers for Learning on Evaluation and Results ) and one of the six CLEAR Centers has been presented in the Latin-American Volume of the Globe (Maldonado Trujillo, 2022). The CLEAR Initiative was started 2010 by IEG and also a couple of other transnational organisations (again including other development banks and some of the partners of GEI like SDC). The partners decided to establish six regional centers in cooperation with local universities for evaluation capacity development in Brazil, Mexico, Senegal, South Africa, India, and China. These centre offer trainings, technical assistance, knowledge products, and knowledge sharing within the region. Until 2018, the CLEAR centers delivered capacitybuilding trainings to 28,277 individuals from 66 countries (further information can be found at West Meiers, 2022, 439ff.). Besides these networks, there are still more global, regional or sectoral networks on evaluation, like for instance 3iE, the Network of Networks for Impact Evaluation NONIE, the National Evaluation Center NEC, the Network of European Evaluation Societies NESE, the Organization for Security and Co-operation in Europe Evaluation Network OESCE, Network for Evaluation of One Health NEOH and others. In many of these networks, transnational organisations, particularly IEG and IEO, play an important role. To summarise, since many decades there are a lot of (global) networking activities in evaluation that are merely initiated, organised and driven by transnational organisations. The transnational organisations presented in the three GLOBE-Volumes can be identified as very influential in bringing different parties and actors together in this global network for developing evaluation (Fig. 1).

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

471

Fig. 1 Transnational Organisations and their involvement in (global) evaluation networks1

One may detect several reasons for developing such networks: firstly, ECG and UNEG where built for improving internal coherence in the UN-system and between the development banks respectively. OECD DAC and GEI are products of the attempt to harmonise development activities and include especially donor countries and the evaluation offices at national level. While DAC was formed as one of the first networks for donor homogenisation, the newly installed GEI is now trying to harmonise ECD and support activities on evaluation in the nations of the Global South. IOCE, NESE, NONIE, and 3iE are civil society networks which are acting independently but are primarily supported by transnational organisations. They are merely formed around the millennium. EvalPartners finally linked the civil society activities with the UN-system, forming a strong unit beyond state interests and pushed the diffusion of 1 Remarks The oval includes the transnational organisations presented in the three GLOBE volumes. The networks included are somehow mentioned in these chapters. The fat lines indicate membership relations, the dotted lines show observer state or supporting role.

472

W. MEYER

evaluation since its start in 2012. Almost at the same time, IEG and the World Bank started the first initiatives for evaluation capacity development IPDET and CLEAR on several levels (supporting governments as well as civil society actors) and they are built in cooperation with universities and evaluation researchers and trainers around the world. The only isolated actor in our books is the Council of Europe (and in total the European Union) because its attempts to be part of UNEG and OECD DAC at an observer state were refused due to formal reasons. However, CoE is an institutional member of the European Evaluation Society and uses the UNEG standards and principles in their own framework (Blomeyer & Eussner, 2020, p. 476). Although many activities of the networks aim on the coordination and homogenisation of members, there is at least an attempt to foster national developments. Some focus on governments, supporting them in implementing, improving and using national monitoring and evaluation systems (NMES). Others target on civil society organisations, particularly voluntary organisations of professional evaluation (VOPEs). And both are served by evaluation capacity development (ECD) initiatives, including a broad range of different forms of evaluation trainings and other forms of support for developing the evaluation culture. These three aspects will be discussed in more details in the final part as outcomes of transnational activities. But first the focus is set on the transnational organisations themselves, how the institutionalised evaluation within their organisational structure and how they use evaluation as an instrument to improve the quality of their own activities.

3

Institutionalisation of Evaluation in Transnational Organisations

Transnational organisations are not only active as facilitator or sponsor of evaluation activities done by national state or non-state organisations, but they are also doing evaluations by themselves for improving the quality of their development programmes, trainings or other activities within the framework of their own organisation. While they are transnational organisations, there is a more or less complex, merely globally installed decentralised organisation structure that acts in many cases rather independently and evaluation is institutionalised on several different levels. Therefore, the first look is on the institutionalisation of evaluation within

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

473

transnational organisations, the several forms existing today and the implications these forms imply for evaluation activities within the particular transnational organisation. As mentioned above, the United Nation System can be identified as the most important global player and its organisational forms cover a broad variety of possibilities to install evaluation as part of the activities. In 2007, UNEG published an overview on the institutionalisation of evaluation in the UN-System (Fig. 2). Roughly speaking, it is divided into three almost equally weighted groups. On top, there are 18 UN-Organisations that implemented evaluation as a function in oversight or even independent stand-alone units. Among them is one organisation that had been part of deeper investigations in the GLOBE project –the Independent Evaluation Group (IEG) of the World Bank Group (WBG) in the Americas Volume. Other Non-UNOrganisations (especially the Development Banks) followed this model and have also been added as examples (Asian Development Bank ADB, Inter-American Development Bank IADB). The next part of this chapter

Fig. 2 Institutionalisation of evaluation in the UN-System 2007

474

W. MEYER

will give an overview of these solutions, its development and impact within the organisational framework as described by the GLOBE authors. The second group of organisations in the UN-system located evaluation within several different units, merely in policy, planning or management units, but also in research or learning units. In these organisations, evaluation is closer to the operative activities and less independent. Examples for this structural model are the Food and Agricultural Organisation FAO, International Labour Organization ILO and UNICEF , which has been described in this volume (Raffii & Yang, 2023). The decentralised evaluation function of UNICEF and its specifics will be also mentioned in the next part, especially in comparison with Non-UN organisations—here the institutionalisation of evaluation in the EU-framework as described by Blomeyer and Eussner (2020) in the European volume of the GLOBE. Finally, there had been still one-third of organisations in the UNsystem without any dedicated evaluation units in the UN-system. Although this group shrank during the last 15 years, there are still parts of the UN-system which does not use Monitoring and Evaluation as a regular instrument (like the Department of Operational Support DOS or United Nation Department for Economic and Social Affairs UN DESA). As already stated, most of these UN organisations are also not participating in UNEG. Not surprisingly, none of these organisations can be found as influential for institutionalisation of evaluation on national level and therefore none has been added in the GLOBE project. 3.1

Implementation of Monitoring and Evaluation in Organisation

The first independent evaluation office was developed in the World Bank Group (WBG) and ended up with the Independent Evaluation Group (IEG), one of the most influential actors for the institutionalisation of evaluation not only within the World Bank framework but also—as already described—on a global and national level. Starting point was the decision of the 1944 founded World Bank to assess projects in an Operation Evaluation Unit in the World Bank’s Programming and Budgeting Department, influenced by President Robert McNamara who had been interested in evaluation during his time in the US government in the 1950s and 1960s. Primarily driven by internal organisational decisions, the first independent evaluation function within a financial institution was formed in the Operations Evaluation Department (OED) in 1973. While these evaluations first focused on the governmental sector, the evaluation

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

475

function diffused into the private sector branch and was compound to the IEG in 2006 (West Meiers, 2022, p. 420). The Asian Development Bank (ADB) was established in 1966. It supports the economic growth and cooperation and its Strategy 2030 (ADB, 2018), approved in 2018, focuses on a prosperous, inclusive, resilient, and sustainable Asia and the Pacific. While evaluation work was first performed by an evaluation unit under the Economics Office (1972), an Independent Evaluation Office was established in 2004 and is composed of a Thematic and Country and a Sector and Project division (Vijayaraghavan, 2023, p. 419). The other development bank presented in the GLOBE, the Interamerican Development Bank, started evaluation in the late 1990s and evolved into the Independent Office of Evaluation and Oversight (OVE) in 1999. It is separated from IDB management’s internal office of evaluation and reports directly to the Board of Executive Directors. It is responsible for evaluating the development effectiveness and performance of IDB Group’s Work (Linares et al., 2022, p. 404f.). Although all three evaluation units are independent, this does not mean that the organisational solutions are similar and equally constructed (Table 1). All three Independent Evaluation Units report to the board of directors in the organisation, they are seen as an important management tool, and they are obviously used to deliver information for decision-making. Institutionally and operatively are all Evaluation Units independent from the operational parts of the organisation as well from the decision-making boards. As soon as the work programme and the director are approved, the evaluation offices are free to act in their own responsibility. Nevertheless, there are some minor differences in the way of regulating reporting or cooperation between various evaluation units (some of the operative departments do also have evaluation departments). The differences are much more significant if one looks at the mandate and work programmes. All units are responsible for independent programme and policy evaluations, while there are some differences if it comes to self-evaluations. In general, self-evaluations are organised on behalf and by the operative departments but in some organisations, the independent evaluation units take over the quality control and a supervision position. In other organisations, they have only a consulting role and sometimes they play a role in particular forms. IEG, for example, mention mandatory self-evaluations, “prepared by the responsible operational units and are embedded in the project and program cycles… Behavioral independence is further

476

W. MEYER

Table 1

Comparison of independent evaluation units and its structural elements

Established

IEG (WBG) 2006

IED (ADB) 2004

OVE (IDB) 1999

Reporting

Reports directly to Board of Directors

Reports directly to the Board of Executive Directors

Institutional and operative Independence

Organisationally independent from operational units; Director General of Evaluation is responsible for all evaluation functions

Reports through the Development Effectiveness Committee to Board of Directors Organisationally independent from operational units; Behavioral autonomy, insulation from external influence, and avoidance of conflicts of interest Conducts evaluations to assess the development effectiveness of ADB’s operations; evaluates ADB policies, strategies, operations and organisational and operational effectiveness; provide feedback

Mandate and Work Responsible for the programme assessment of the relevance, efficacy, and efficiency of World Bank Group operational policies, programmes and activities, and their contribution to development effectiveness

Evaluation Approaches and Forms

Uses a range of approaches to assess outcomes, doing independent evaluations, mandatory self-evaluations and demand-driven self-evaluations

Evaluation Policy

The World Bank policy on access to information

Corporate and Thematic Evaluations, Sector-wide Evaluations, Country Assistance Programme Evaluations, Impact Evaluations, Project self-evaluations are conducted by operations departments ADB Strategy 2030

Organisationally independent from operational units

Evaluating the development effectiveness and performance of IDB Group’s work; Improve “operational performance, strengthen institutional learning, and achieve better results …” Independent and systematic evaluations of IDB Group’s strategies, policies, programmes, operations, activities, and systems

Evaluation Policy Framework

Source Extracts from Vijayaraghavan (2023), Linares et al. (2022), West Meiers (2022)

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

477

strengthened by IEG’s review and validation (sometimes on a sample basis)” (West Meiers, 2022, p. 426). Due to different degrees of decentralisation (and the attributed importance to this), not all UN-Organisations installed an Independent Evaluation Unit at the top of the organisation. UNICEF , for example, set up a decentralised evaluation function operating at three different levels—Headquarters, Regional and Country offices—and they are not connected by a formal reporting link. The structure of the UNICEF Evaluation Office (EO) is similar with the one in UNDP: it is headed by a director who reports to the Executive Director, and it leads and coordinates the evaluation function within UNICEF. Most of the evaluations are done decentrally in strong cooperation with the partner countries (Raffii & Yang, 2023, p. 438). Therefore, UNEG classified UNICEF as part of the third group “Entities with Evaluation co-located with Programme Policy, Management, Planning and/or Monitoring Units” (UNEG, 2007). However, the chapter in this book also mentioned several weaknesses of this approach, reported in the 2017 peer review of the evaluation system, particularly the tension between the independence of the evaluation function and the decentralised evidence-generation function. Regarding the recommendations, UNICEF tries to improve its system, but this fundamental antagonism will probably not be finally solved. Decentralisation is of course also a topic for the European Union and its evaluation function, but the task is still more complex. There is not one EU-Organisation with a homogenous evaluations structure, but three different actors that are working independently. Most of the evaluation are commissioned on behalf of the European Commission (EC), the executive body of the European Union, and about 80% of these evaluations are externalised. They are commissioned by different Directors General (DGs)—particularly the DG Budget and the DG Financial Control —and on behalf of their own interests and there is no overarching evaluation unit. Following the “Better Regulation Agenda” 2006, the number of impact evaluations increased and the Secretariat General (SG) became the lead unit responsible for evaluations. However, all EU-programmes must be evaluated, and this put—at least in the framework of the European Structural Fund—some pressure on EU-member states to develop an evaluation infrastructure on their side. Other parts of the EU-system like the Council of Europe and the European Parliament (EP) are commissioning evaluations on their own behalf and they are not linked to each

478

W. MEYER

other by a network like UNEG until now (Blomeyer & Eussner, 2020, 464ff.). 3.2

Monitoring and Evaluation Practice of Transnational Organisations

The transnational organisations under investigation do not only vary in the structure how they implemented the evaluation function within their organisational framework, but also in the evaluation practice. The following Table 2 focuses on objectives, evaluation tasks, forms of evaluations and its utilisation as far as it has been mentioned by the GLOBE authors. There are certain similarities in all transnational organisations under observation. The objectives for and use of evaluation target primarily on the development of the organisational system, internal reasons are the driving force for building up evaluation institutions. Therefore, differences between the institutionalisation of evaluation belong to the specifics of their objectives and activities. The large UN-Organisations are operative units, initiating and managing programmes and projects for a broad range of applications and countries. Evaluating the effects achieved by these activities and how they can be improved in effectiveness and efficiency are the main target and starting point of evaluation activities. While programmes and projects are done decentrally around the whole world, the framework conditions and cultural specifics are very important for its success. Therefore, on one hand side, there is some pressure towards decentralisation of the evaluation function to be closer to the demand from operative departments. On the other hand, the management of the transnational organisation is interested in controlling all activities and for decision-making, evaluation results are important and necessary. The evaluation function should be close to the top management and independent from operative units for delivering reliable and objective evidence. This tension between central and decentral interests is the main driver for the development of evaluation institutions and reveals a lot of dynamics over the years. Roughly speaking, the interest of top management on producing and using evaluation results focus primarily on independent ex-post evaluations, while the decentralised operative level is concentrating on formative, ongoing self-evaluations.

IED (ADB)

Within the WBG: Proofing expected results, evaluation-based recommendations, reviewing self-evaluations, regular reporting, uptake of evaluation findings, learning and accountability Outside the WBG: Encouraging and assisting member countries to build effective monitoring and evaluation partnerships, capacities and systems Within ADB Assess the development effectiveness, investment of resources, achievement of outcomes, policies, strategies, operations, providing feedback Outside ADB Developing member country stakeholders to improve future policies, programmes, and projects

Objectives

To articulate how self- and independent evaluation work together across the institution, principles that guide in complementary roles, ensure that self-evaluation instructions are in accordance with guidelines

All independent evaluation work, mostly ex-post, appraises the World Bank Group’s other evaluation systems and methods

Tasks

Evaluation Practice of selected transnational organisations

IEG (WBG)

Table 2

Independent evaluations, corporate evaluations, project evaluations, country evaluations and technical assistance (TA) evaluations

Major evaluations (covering broad themes, sectors and corporate processses); meso evaluations (‘just-in-time’ evaluations focused on specific issues); country programme evaluations; project performance assessment

Forms

(continued)

ADB-wide evaluation guidelines evaluation principles document, corporate results framework

Planning and Decision Making; World Bank Group Evaluation Principles and other standard evaluation products; Global agencies adopted IEG’s independent evaluation offices

Use

15 THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

479

Within IDB Improve operational performance, strengthen institutional learning, achieve better results Outside IDB Increase demand for evaluation, build evaluation capacities, support the creation/ consolidation of evaluation units

Objectives

(continued)

OVE (IDB)

Table 2

Ensure effectiveness in providing development results, adaptive learning, evidence of what does and does not work, foster accountability

Tasks Ex-ante evaluability assessment, self-evaluation of development interventions, validation of self-evaluations

Forms Evaluation Policy Framework, Final performance rating for corporate reporting

Use

480 W. MEYER

Within UNICEF Regional Evaluation Advisors (REA) report to the Regional Directors, with a technical reporting line to the Evaluation Director The evaluation function established multi-country evaluation specialists in all regions that had either direct or technical reporting lines to the REA Apply to all EC activities; reduce inconsistencies and burdens in the regulatory framework; gathering the best evidence for policy-making in the broadest possible way; produce policy learning, and not only financial and programme or project learning Systematic impact assessment of new proposals; “evaluate first”: states that all new activities must take place after the assessment of past actions; stakeholder engagement

Formal management response in guidance issued by the EO; the EO monitors and reports on the implementation status of every evaluation management response

Tasks

Sources Chapters on Transnational Organisations in the three Globe-Volumes

CoE (EU)

EO (UNICEF)

Objectives

Ex-ante evaluation; ex-post evaluation; the practice of ex-post legislative evaluation is relatively patchy and results rather from legislative obligations than DGs’ initiatives

Independent corporate evaluations, humanitarian evaluations, capacity development, partnerships and coherence, innovation, learning, and uptake

Forms

71% of evaluation recommendations fully implemented; the successful use of evaluation depends on involving senior and middle management from start to end

Evaluation reports; production of meta-evaluation; public Website, central database; follow up on the use of evaluation findings and recommendations; management response to the recommendations, tracking actions

Use

15 THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

481

482

W. MEYER

For sure, this has some impact on the evaluation practice and the coordination between both forms seems to be an issue until now. As one result, the evaluation offices at the top of the organisation produce an increasing number of guidelines or standards, particularly for regulating the evaluation roles and its differences within the own organisation. To a certain extent, these difficulties can also be found in the EU-system: while the majority of evaluations are commissioned by the European Commission, various DGs are responsible, and this influences the ways such evaluations—in most cases ex-post-evaluations—are following. Remarkably, the EU-system did not yet install a comparable strong centralised evaluation unit like the independent groups in the UN-system. Nevertheless, the tensions between summative ex-post evaluations and planning-oriented ex-ante evaluations seem to be a comparable important driving force for developing the evaluation function. The lack of institutionalisation on the top level may be explained by the isolation of the EU-system which is not part of the UN- and OECDnetworks as shown in the first part of this chapter. The development of independent evaluation units is clearly influenced by this exchange: IEG as the first of these units became a “blueprint” for the other banks as well as for UN-Organisations like UNDP or UNICEF. In general, the dynamic of developing the evaluation function is caused by the communication in global coordination networks like OECD DAC, UNEG and ECG. A systematic analysis of the UN-system stated, for instance, that the central evaluation function “has moved from a predominant role of oversight over and quality assurance of decentralized evaluations to focusing on supporting broad and strategic corporate-level decision-making” and UNEG “played a significant and highly visible role … in enhancing the professional development of the function and the harmonisation of evaluation methodologies, thus providing an effective platform for the advancement of the function” (Prom-Jackson & Bartsiotas, 2014, p. IV). Therefore, the harmonisation and coherence effects of networks are not limited to institutions but can also be found in evaluation activities of the UN agencies. A recent study of UNEG revealed four main trends of evaluation practice in the UN-system (UNEG, 2020b). The first trend is to ensure the relevance of evaluation to user knowledge needs and this is targeting on the scope of evaluations as well as on the criteria and methods used. The second trend is about the way and options of how UN agencies synthesise and communicate evaluation results. In general, the “evaluation offices appear to have understood the benefits of effective

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

483

communication to enhance the utilization of the evaluations.” (UNEG 2020b, p. 7) and they use a growing number of different communication products to achieve this objective. Tracking the use of evaluation recommendations and the implementation of measures by the users is the third trend identified in this report. All evaluation offices developed some strategies for doing so and the current practice already helps to understand the change generated by evaluations. Finally, there are more attempts to capture and track user opinions on evaluation. These feedback loops seem to be still at an early, merely informal stage and are not as robust as the information on the other three topics. While all these trends and developments are focused on internal processes of the transnational organisations, one has to recognise the significant attempts of at least some of these organisations to spread evaluation as a management tool beyond the borders of their own organisation—and even beyond the system-borders of UN and EU. The development banks and UNDP understand themselves as a driving force to foster improvements in partner countries and this includes the evaluation function. While being part of the UN-system, they focus primarily on national states and their development policies, but—as the next chapter will show—a lot of their activities target on civil society and the formation of a global evaluation culture. Evaluation is not only understood as a management tool for national elites, but also for democratisation and empowerment of civil societies.

4 Transnational Organisations Supporting National M+E Activities As shown in the first part of this chapter, transnational organisations are important players for the development, institutionalisation, and professionalisation of evaluation on a global level. The authors of the country chapters also mentioned the role of transnational organisations for developing evaluation on national levels (Table 3). There is only a small number of country chapters where no influence of transnational organisations has been recognised (Australia, Canada, China, Korea, Switzerland, Taiwan, the Netherlands, and the USA). For sure, this does not mean that transnational organisations do not have any influence in these countries, but the authors did not recognise them as important enough to be mentioned. In some cases (particularly Taiwan), this may be caused by the isolated role of the country in the international

484

W. MEYER

Table 3

Influences by transnational organisations mentioned in country reports

Countries

OECD

EU

Argentine Australia Bangladesh Belgium Bolivia Brazil Canada Chile China Columbia Costa Rica Czech Republic Denmark Ecuador Finland France Germany India Ireland Italy Japan Korea Latvia Mexico Nepal Pakistan Philippines Peru Poland Portugal Romania Spain Sri Lanka Switzerland Taiwan The Netherlands United Kingdom USA

World Bank

UN

Regional Banks

CLEAR

Others

1 0 4 2 2 7 0 3 0 5 1 1 1 5 2 1 2 7 3 1 1 0 4 1 3 3 4 6 2 4 4 2 5 0 0 0 1 0

2

3

2

3

2

3

N (total 38) 19 18 12 12 10 5 Other UN agencies: UNICEF 6, UNDP 5, UNESCO 2, IFAD, IMF, UNEG, WHO, non-specific 3 Others: OPAC, COEECI, EVALPARTNERS

n

3

88

Influence of TO mentioned on country level TO only mentioned on country level

Other UN agencies UNICEF 6, UNDP 5, UNESCO 2, IFAD, IMF, UNEG, WHO, non-specific 3 Others OPAC, COEECI, EVALPARTNERS

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

485

community. In other cases, the reason may be the pioneering role of the countries (Australia, Canada, and the USA) in which the national development had obviously been more important than external influences. In opposite, the development of evaluation in transnational organisations is mainly driven by the national evaluation culture already developed in these countries. At the other end of the scale, there are several countries where a lot of transnational organisations are named as important influence for the national development of the institutionalisation of evaluation (Brazil, Columbia, Ecuador, India, Peru, and Sri Lanka). These countries are main partners of development cooperation and therefore there are many programmes and projects run in international bilateral and multilateral partnerships. As already outlined, evaluation developed very early as part of programme management and was used in a more or less homogenised way (driven by OECD-DAC). These activities involved many national ministries and agencies in these countries, and they learned at least about the role evaluation played in donor organisations. In some cases, donors implemented special programmes to develop evaluation capacities—not necessarily transnational organisations: in Ecuador, for example, several initiatives engaged on different levels like the regional ECD-project FOCEVAL financed by the German government, university cooperations with French and German universities or single training courses offered by World Bank and IDB (Gross & Espin, 2022, p. 289ff.). From the perspective of transnational organisations, most countries mentioned OECD as a certain influence (19 countries). Merely, the OECD DAC evaluation criteria are recognised as part of development cooperation, but also other discussions and initiatives by OECD are reported as important aspects for developing evaluation in these countries like for instance the PISA-project in education or some debates on New Public Management approaches. Nevertheless, the effects are primarily indirect (colored yellow in the table) and not part of some kind of ECD programmes or other ways to support the institutionalisation of evaluation directly. This is different to the effect of the European Union, reported as very influential in almost all European countries and even some nonEuropean ones (Bangladesh, Pakistan). For the European member states, the EU rule for evaluating all European support programmes was important—particularly as drivers for the institutionalisation of evaluation in new member states from South (Spain, Portugal) and Central Eastern

486

W. MEYER

Europe (Czech Republic, Poland, Rumania). Due to its size, the large European programmes and its evaluation culture were and are one of the key elements to support the institutionalisation of evaluation in all countries involved. The third transnational organisation recognised as being important for developing national institutions for evaluation is the World Bank (12 countries). Its relevance lies somehow in between the EU and OECD— in difference to the OECD, the World Bank supported much more direct evaluation capacity development in several countries, sometimes via IEG, sometimes indirectly for instance through the CLEAR network. Compared to the EU, the demand for evaluation is less compulsory and therefore there is almost no pressure to institutionalise evaluation capacities offered or supported by the World Bank. Hence, the World Bank is still the most active and supportive transnational donor for ECD-initiatives and at least indirectly this work is relevant and well received. The UN agencies (12 countries) are also important players, but their support is more limited to certain regions or policy areas. Among the agencies, UNICEF (6 countries) and UNDP (5 countries) are most often mentioned and in some countries (Brazil, India and Sri Lanka) several UN agencies are very active. In general, the effects are linked to important ECD programmes (or corporation programmes with a significant evaluation component) and therefore there are more direct influences, sometimes limited to certain policy fields. Additionally, the regional development banks support ECD-initiatives and contribute to the institutionalisation of evaluation in many countries (10 countries). Besides special ECD-projects, the general evaluation practice in huge support programmes influenced the development of evaluation institutions in some countries. In Brazil, for example, the Inter-American Development Bank is emphasised as being central for the creation of an evaluation culture (Dolabella, 2022: 98). The technical support of the Asian Development Bank was very important “to strengthen the Post Evaluation System in the Ministry of Plan Implementation” in Sri Lanka (Kalugampitiya et al., 2023, p. 352). Like the regional development banks, the CLEAR Centre (5 countries) are only regionally active and not globally. In Latin America, there are two centres in Brazil (Sao Paulo) and Mexico (Mexico City), and in Asia, there are another two in India (New Delhi) and China (Shanghai). Compared to this, the reported influence of the CLEAR centres is rather limited.

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

487

Only in two of the four origin countries, India and Mexico, the effects on the national development of evaluation are recognised as being important. In China, the CLEAR centre is not even mentioned, in India, CLEAR is cited as a training provider—among others (Agrawal et al., 2023, p. 175). There are only two other country authors (Chile and Ecuador) that refer to CLEAR centres’, activities: in both countries, it is linked to the gLOCAL Evaluation Week which had been promoted by CLEAR and organised by IEG (Gross & Espin, 2022, p. 287; Manríquez & Fiscarelli, 2022, p. 188). To sum it up: at least in the subjective perspective of the authors, CLEAR centres do not have a relevant impact on the national development of evaluation in the Americas and Asia. Beyond these transnational organisations, the authors referred only to three other transnational organisations. Beside this, external influences come from several transnational networks—especially evaluation networks like IOCE, APEA, and RELAC or bilateral cooperations. Therefore, the selection of transnational countries covers obviously the most important external forces for developing evaluation in the selected countries. They offered a broad variety of support that cannot be mentioned here in detail. The following parts are giving a short introduction into three examples of such activities, not meant as a complete overview but just to illustrate some selected cases. The three topics are the development of National Monitoring and Evaluation Systems (NMES), primarily in cooperation with Nation states, the establishment of VOPEs together with civil societies, and Evaluation Capacity Development (ECD) activities that are targeting on both sides (demand and supply of evaluation) for to improve knowledge and competences for evaluation. 4.1

National Monitoring and Evaluation Systems

There are several initiatives from transnational organisations to support National Monitoring and Evaluation Systems (NMES). As mentioned above, this is even one of the main objectives of the new Global Evaluation Initiative (GEI). “In the late 1990s, and especially during the first decade of the 2000s, IEG staff authored and commissioned research to document the experiences of countries in setting up their M&E systems. Additional research activities involved developing publications on evaluation and results-based management methodologies” (West Meiers, 2022, p. 433). UNEG published most recently an overview on the UN-systems strategy for national evaluation capacity development and the evolution

488

W. MEYER

of national evaluation systems (UNEG, 2022). Hence, the following part will focus primarily on activities reported by authors of chapters on transnational organisations and give some examples what has been done on behalf of this topic. In general, the support of NMES is composed of institutions (e.g. national evaluation policies, strategies, and regulations), people (particular training measures which will be treated in the Evaluation Capacity Development part) and activities (e.g. supporting evaluation processes, developing guidelines etc.). Between 2017 and 2019, UNICEF and UNDP jointly conducted a Review of National Evaluation Systems and Capacities for Evaluating Progress towards the Sustainable Development Goals in Asia and Pacific Region (for further details see Raffii and Yang 2023, p. 448). This review revealed three regional priorities: the enabling environment, the institutional development and individual capacity. Among the measures recommended are the support of policy or legal commitments and guidelines, the engagement of parliamentarians, mechanisms to allocate specific resources for evaluation, the finalisation of monitoring and evaluation frameworks for SDGs, support policy coherence and multistakeholder involvement, and the development of standards, guidelines, and good practices (Polastro & Prokop, 2017, p. 33). UNICEF works together with seven countries (Indonesia, Malaysia, Mongolia, Philippines, Sri Lanka, Thailand, and Vietnam) and there, some first steps for building up a National Evaluation System have been undertaken. Two countries (Malaysia and Philippines), for example, installed already a National Evaluation Policy (Raffii & Yang, 2023, p. 449f.). In Latin America, the continuous work of the World Bank on Monitoring and Evaluation advisory must be mentioned. IEG provided such kind of advisory services and “the region is highly regarded in the evaluation community … of having the more successful experiences with establishing laws, policies, guidelines, processes, technical staff, leaders and so on”—particularly Chile, Colombia, and Mexico (West Meiers, 2022, p. 434f.). IEG produced research materials on M&E systems that are also translated into Spanish and these research activities can be seen as a strong influence due to the open access policy of the World Bank. Furthermore, the IDB Group developed a strategy on country systems and supported activities for strengthening agencies’ responsibilities for generating accurate, reliable, and timely statistics, and of agencies in charge of M&E systems (Linares et al., 2022, p. 406). These activities are merely financed by non-reimbursable technical assistance resources

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

489

and refer to all national norms, regulations, procedures, and structures for the management of the public sector. IDB reclaims an increasing demand for evaluation in governmental agencies and the creation and/or consolidation of evaluation units as key results of these activities. These are—as noted above—only a few examples for transnational organisations being successful in implementing National Monitoring and Evaluation Systems (NMES). Although there is a certain progress visible, most initiatives still wait for a widespread breakthrough and wellfunctioning NMES are still rare (not only in the Global South but also in Europe and North America). 4.2

Voluntary Organisations for Professional Evaluation

During the last three decades, the relationship has significantly changed between transnational organisations—particularly the UN agencies—and NGOs. While such cooperation had former been limited to the implementation phase of projects and to interactions with operational bodies of the UN, NGOs are nowadays more often integrated into policy-making forums and function as policy advisors or information provider (Martens, 2006, p. 696). Particular in the field of evaluation, the cooperation and interaction are remarkably high and intense between transnational organisations and NGOs. As already mentioned, UN agencies (e.g. UNICEF) formed a unique global partnership called EvalPartners together with the International Organisation for Cooperation in Evaluation (IOCE), the umbrella network of Voluntary Organizations for Professional Evaluation (VOPEs). This development was fostered by the post-2015 development agenda and led to the first EvalAgenda2020, encouraging the evaluation profession to take a more global approach. The agenda constituted three pillars, the building of individual evaluation capacity, strengthening institutional evaluation capacity and creating an enabling environment for evaluation (EvalPartners, 2015). While the first part is again on capacity development (and therefore will be treated later), the other two parts are targeting explicitly on institutionalisation. Strengthening the enabling environment for evaluation puts the focus on civil society’s role as a driving force for national development and wants to make civil society a “push”-factor for recognising the value of evaluation in all sectors of a society. The agenda seeks for including evaluation through governance and regulatory instruments such as national

490

W. MEYER

evaluation policies, an increase in civil society’s demand for and in evaluation and a recognition of evaluation as a profession—exactly the three topics investigated here in the GLOBE project. Saying this, the results provided in the three volumes of the Globe are not encouraging: civil society does not come even close to the demand of EvalAgenda and is by no way a driving factor for evaluation development on the national level. The dynamics recognised both in the political system and the systems of professions are not triggered by pressure from outside (particular civil society organisations and the social system), but by internal forces like public administration and academic actors. Up to now, civil societies failed to take the progressive role the EvalAgenda2020 wants them to have. EvalPartners is much more successful if it comes to institutional evaluation capacities, especially the support for founding and consolidating VOPEs. The EvalAgenda2020 stated for VOPEs four main goals, strengthening their capacity and role in influencing the enabling environment, disseminating the importance of evaluation, being strategic and inclusive, and engaging in peer learning. In general, the history of VOPEs is a great success story: between 1980 and 2015, the formation of professional evaluation organisations spread around the whole world, ending up today with covering almost every country (Graph 1). Hence, this diffusion process was almost finished before EvalPartners was founded and its role for developing new VOPEs is limited. The importance of IOCE and EvalPartners lies more in bringing VOPEs together on the global level and stabilising their existence by administrative advice and support. The support for VOPEs by transnational organisations began with IEGs engagement for building up the global International Development Evaluation Association (IDEAS), which started in September 2002 to represent the internationally active development evaluators. “In almost 20 years, the global evaluation community has changed greatly with many new and more robust national and regional associations” and IEGs originally very important role for bringing such initiatives on its way was significantly reduced (West Meiers, 2022, p. 441). As already mentioned above, UNICEF was an important actor to launch EvalPartners, but it also has a long tradition in supporting VOPEs particular in East Asia and the Pacific (EAP). “Since 2010, the EAP Region has seen fledgling and emerging VOPE networks. By the end of 2021, 10 EAP-based VOPEs were registered as IOCE members, including one regional VOPE, namely Asia Pacific Evaluation Association (APEA), and nine national VOPEs from eight countries. Almost all these VOPEs were founded in or after

cumulative

2012

2014

2008

2010

2006

0

2004

0

2002

20

2000

2

1998

40

1996

4

1994

60

1992

6

1990

80

1988

8

1986

100

1984

10

1982

120

491

cumulative

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

12

1980

absolute

15

absolute

Graph 1 Development of VOPES 1980–2014 (Source Meyer [2015, p. 232]; IOCE-Survey Data)

2009…” and they “bring together evaluation practitioners from various fields, including government, academia and non-governmental organizations, and private sector. By advancing their missions, these VOPEs contribute not only to the benefits of their members but to the benefits of society at large as well” (Raffii & Yang, 2023, p. 451). Finally, UNDP IEO is also active in supporting regional VOPEs, but it is more focusing on Africa than on Asia or Latin America. To sum it up: transnational organisations made a relevant impact on the development of VOPEs, merely in building up new capacities and global or regional networks. They provided and still provide opportunities for exchange and institutionalisation, although they were rarely involved in the formation and foundation of such initiatives. As a result, nowadays a remarkable and unique interlinkage exists between civil society-based professional associations and transnational organisations. It enables a homogenous and relatively coherent understanding of evaluation around the world. Compared to the huge differences of evaluation practice in

492

W. MEYER

various policy fields, the cultural and national variations seem to be not so large due to these transnational initiatives. 4.3

Evaluation Capacity Development

Most activities on transnational organisations are on evaluation capacity development (ECD) and again IEG and the World Bank took the initiative quite early. The objective of IEGs ECD efforts is “to foster international evaluation harmonization, to develop evaluation capacity in member countries and to encourage best practice in international development evaluation” (West Meiers, 2022, p. 423). Most important invention was the International Program for Development Evaluation Training (IPDET), founded in 2001 as the first regularly offered residential executive training programme in cooperation with Carleton University in Ottawa, Canada. In 2017, after having trained more than 3.500 participants from 125 countries, IPDET moved to the University of Bern and offers now three-week evaluation courses in a summer school together with the Center for Evaluation (CEval) at Saarland University (https://ipdet.org/). The new partners are among the most important demand-oriented evaluation training centers in Europe and Saarland University is currently offering a globally oriented Master of (Blended Learning) Evaluation (MABLE) with up today more than 100 students (https://fernstudium.rptu.de/en/management-law/master-ble nded-learning-evaluation-mable). Forced by the Covid-Crisis in 2020, IPDET invented additional e-learning offers and succeeded to survive two years without any opportunity for onsite meetings. The new IPDET programme does not only focus on providing trainings in Bern and online courses but includes an outreach strategy for offering special trainings together with partner organisations for selected target groups in other countries (e.g. courses for parliamentarians in Sri Lanka) and tries to include more intensively trainers from the global South into their own training activities in Bern, particular in cooperation with the Centers for Learning on Evaluation and Results (CLEAR centres). The CLEAR initiative was started by IEG in 2010 and includes six regional ECD centers in Anglophone and Francophone Africa, South and East Asia, Latin America, and the Lusophone Region (situated in Brazil). IEG shared forces with a lot of other transnational organisations such as development banks (ADB, AfDB, IDB), bilateral development agencies (Australian Government Foreign Affairs, Belgian Federal Service for

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

493

Foreign Affairs, UK Department for International Development DFID, Swiss Agency for Development and Cooperation SDC, Swedish International Development Agency SIDA) and private foundations (Rockefeller foundation). The CLEAR centres provide training, technical assistance, knowledge products and knowledge sharing and in 2018, more than 28.000 people from 66 countries had been trained on behalf of the CLEAR initiative (West Meiers, 2022, p. 439). Further details on the work of one of the CLEAR centres (Latin America and the Caribean) can be found in the Americas Volume of the GLOBE project (Maldonado Trujillo, 2022). Moreover, IEG initiated a French language version of the IPDET programme called PIFED (Le Programme International de Formation en Evaluation du Développement ), operating since 2011 in cooperation with the ‘École Nationale d’Administration Publique’ (ENAP) in Québec City with additional funding from the Government of Canada. IEG also published a lot of training materials, books, guidelines, blogs, etc. on evaluation in several languages and is nowadays—as part of GEI— also involved in the Better Evaluation Online platform, so to speak the “Wikipedia” for evaluation, providing insights into the World of evaluation by introductory articles written by well-known authors around the world (https://www.betterevaluation.org/). While IEG is without any doubt a strong actor in the field of ECD, there are also other transnational organisations very active, and their input is extremely powerful too. UNDP, for instance, published a handbook on planning, monitoring and evaluation in 1997 and keeps it updated since then. The 2009 edition received more than 10 million downloads until 2019 (UNDP, 2009). Additionally, the UNDP evaluation guidelines are published in five UN-languages, including Chinese (IEO, 2021). In general, all UNDP IEO evaluations and further activities in the field of evaluation are published on the website and can be easily traced in an evaluation resource center (https://erc.undp.org/). Another important UNDP activity for ECD is the NEC Conference series which also includes pre-conference training workshops like most of the evaluation conferences organised by transnational organisations and VOPEs. In Asia, the Beijing conference 1999 on Evaluation Capacity Development, organised by UNDP in cooperation with the Chinese National Center for Science and Technology Evaluation, proved to be relevant for the development of evaluation in East Asia (https://nec.undp.org/). The UNDP Evaluation policy from 2019 states that “the evaluation function is an

494

W. MEYER

effective country-led vehicle for greater citizen accountability that can accelerate progress towards national sustainable development goals” and “apart from the conduct of independent and decentralized evaluations of the work of UNDP, support to national evaluation capacity is embraced as a programmatic priority in its own right” (UN, 2019, p. 2). Individual capacity building is one of the three pillars of UNDPs capacity development framework, and these ideas are also shared with partners in the GEI-network or at UNEG. UNICEF offers a broad range of training courses on evaluation on its AGORA platform (https://agora.unicef.org/) and the evaluation blended learning programme “is open to partners (government counterparts, VOPEs, National and Regional Evaluation Associations, and academia), as it aims to foster a working partnership in evaluation with key government sectors and promote greater attention to evaluation in public administration” (Raffii & Yang, 2023, p. 455). Some of these activities target especially on East Asia and the Pacifics as described in the UNICEF chapter in this volume. The importance of transnational organisations can also be proofed by other regional activities on ECD and almost all of the mentioned actors are involved. For Latin America, for instance, there are not only the various activities of the CLEAR centers but also trainings provided by IDB to prepare for impact evaluations that have a certain effect for building up evaluation capacities in the region (Linares et al., 2022, p. 411ff., Maldonado Trujillo, 2022, p. 395ff.). As a summary, there is a huge variety of activities to develop evaluation knowledge and skills supported by transnational organisations particularly in the context of development cooperation. These activities are targeting on all kind of stakeholders in the field: from ministries to target populations, from experienced evaluators to new established staff in monitoring and evaluation units, from government to civil-society organisations. Some of these activities are tailor-made for the needs of transnational organisations or national authorities, others are more generalised and for forming an evaluation profession. The number of such kind of offers significantly increased and transnational organisations—particularly IEG and UNDP IEO are important drivers here. Moreover, there is a certain degree of harmonisation of evaluation understanding and standards how to do it produced by these activities from transnational organisations. They also stand for a certain quality of training that is at

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

495

least above global average. Training material is available in many different languages, and this also increased significantly during the last two decades. However, there is still a lack of demand for certificates achieved in these trainings: if it comes to tendering, none of the transnational organisations are offering a benefit for newcomers that applied successfully to training courses. For the acceptance of a bid, the key criteria are still the years of evaluation experiences and not academic titles or training certificates. While nowadays a lot of support is given to young and emerging evaluators, there is still a barrier existing they cannot overcome—even at the transnational organisations who are extremely active in empowering people for evaluation.

5

Conclusions

Transnational Organisations are without any doubt important driving forces for the development of evaluation and its institutionalisation on various levels. On global level, the most significant contribution was on building up global networks and interlinking them. Starting with activities for homogenisation of donors in development cooperation, the OECD DAC was the first organisation thinking about evaluation and discussing its use in programme and project management. Not long after its formation in the early 1960s, OECD DAC installed EvalNET, the first network on evaluation composed by bilateral and—in observer state—multilateral donor organisations. From these early beginnings, evaluation networks emerged and developed to a real global exchange platform not only for donors in development cooperation but for all kind of interest groups in evaluation. The perspective shifted from programme and project management coordination towards policy coherence and evaluation capacity development, including civil society in a unique form for transnational organisations like the one establishing the UN-system. The last step today is the Global Evaluation Initiative GEI, getting together several ECDinitiatives with a special focus on National Monitoring and Evaluation Systems. An important result of these global networking is homogeneity in the understanding of evaluation and the way how it should be used and trained. This is also true for institutionalisation of evaluation within the framework of complex, globally working transnational organisations. The Independent Evaluation Group IEG in the World Bank Group became

496

W. MEYER

a blueprint for several other UN agencies, particularly the development banks, but also UNDP and to a certain extend UNICEF. The key tension within the organisations is between the top-down-management structure and the effort to control and develop activities based on evidence, merely produced by ex-post impact evaluations, and the participative approach being closer to the “clients” and the operative parts in a decentralised self-evaluation function. This tension seems to drive organisational change and the institutionalisation of evaluation in transnational organisations, leading to certain back-and-forth decisions in the implementation forms. However, it is not a question, whether evaluation should be institutionalised, it is about how to do it. This is significantly different on the national level, where transnational organisations are doing their best to install national monitoring and evaluation systems (NMES) for many years. In general, there is little success and evaluation is only used regularly in a small number of states—merely not the ones development corporation targeted at. While many countries enact high-quality statistical systems and use them at least for some kind of monitoring state effects, institutionalisation and use of evaluation is still poor in the Global South. If it comes to the civil society, the result is even worse. Although particularly EvalPartners sees the civil society as an important driver for the institutionalisation of evaluation, the reality proofs this as “wishful thinking.” Success is merely limited to the consolidation of and exchange between VOPEs. Most of these organisations and networks were formed in the first decade after the Millennium and the high dynamics in this period is not caused by support from transnational organisations, at least not in a generalisable way. But evaluation capacity development programmes like IPDET, ENAP or the CLEAR initiatives may have attributed to the installment of a homogenous evaluation community with a certain indirect impact both on government and civil society side in target countries. Nevertheless, one may have some doubts if this really helps on a market that is still relying on seniority-based principles for commissioning. No doubt, transnational organisations did a lot and they come along a certain way. But there is still some way ahead.

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

497

References Agrawal, R., Rao, B., & Nandi, R. (2023). Evaluation in India. In R. Stockmann, W. Meyer, & N. Zierke (Eds.), The institutionalisation of evaluation in Asia (pp. 141–190). Springer. Asian Development Bank. (2018). Strategy 2030: Achieving a prosperous, inclusive, resilient, and sustainable Asia and the Pacific. ADB. Blomeyer, R., & Eussner, A. (2020). European Union. In R. Stockmann, W. Meyer, & L. Taube (Eds.), The institutionalisation of evaluation in Europe (pp. 463–482). Springer. Dolabella, J. V. (2022). Evaluation in Brazil. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The Institutionalisation of Evaluation in the Americas (pp. 93–141). Springer. Evaluation Cooperation Group. (2009). Review framework for the evaluation function in multilateral development banks. https://www.ecgnet.org/ document/review-framework-evaluation-function-multilateral-developmentbanks. Accessed om 21 September 2023. Evaluation Cooperation Group. (2012). Big Book on evaluation good practice standards. https://www.ecgnet.org/documents?f%5B0%5D=field_global_ document_type%3A14. Accessed on 21 September 2023. EvalPartners. (2015). EvalAgenda 2020. Declaration of support for the global evaluation agenda 2016–2020. https://www.evalpartners.org/global-evalua tion-agenda/declarations. Accessed on 21 September 2023. Gross, C. J. A., & Espín, L. C. M. (2022). Evaluation in Ecuador. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 267–296). Springer. Independent Evaluation Office of UNDP. (2021). UNDP evaluation guidelines. UNDP. Kalugampitiya, A., de Silva, S., & Senaratna, C. (2023). Evaluation in Sri Lanka. In R. Stockmann, W. Meyer, & Zierke (Eds.), The institutionalisation of evaluation in Asian and the Pacifik (pp. 349–382). Springer. Linares, A. M., Funaro Mortara, A., & Putic, M. (2022). Contributions of the Inter-American Development Bank Group (IDB Group) to the institutionalisation of evaluation in Latin America and the Caribbean. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 403–416). Springer. Maldonado Trujillo, C. (2022). CLEAR LAC—Centers for Learning on Evaluation and Results—Latin America and the Caribbean. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 381–402). Springer. Manríquez, D. O., & Fiscarelli, A. P. (2022). Evaluation in Chile. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 171–199). Springer.

498

W. MEYER

Martens, K. (2006). NGOs in the United Nations System: Evaluating theoretical approaches. Journal of International Development, 18, 691–700. Meyer, W. (2015). Professionalisierung von Evaluation: ein globaler Blick. Zeitschrift für Evaluation, 14(2), 215–246. Polastro, R., & Prokop, M. (2017). Review of National evaluation systems and capacities for evaluating progress towards the sustainable development goals: Emerging lessons learned from the AsiaPacific Region. https://nec.undp.org/sites/default/files/2021-07/Review% 20of%20National%20Evaluation%202017.pdf. Accessed on 22 February 2022. Prom-Jackson, S., & Bartsiotas, G. A. (2014). Analysis of the evaluation function in the United Nations System. Genf. https://www.shareweb.ch/site/Multil ateral-Institutions/Documents/JIU_REP_2014_6_English.pdf. Accessed on 21 September 2023. Raffii, K., & Yang, X. X. (2023). UNICEF support to institutionalizing evaluation in East Asia and the Pacific Region. In R. Stockmann, W. Meyer, & N. Zierke (Eds.), The institutionalisation of evaluation in Asia (pp. 435–464). Springer. United Nations. (2019). Executive Board of the United Nations Development Programme, the United Nations Population Fund and the United Nations Office for Project Services: The revised UNDP Evaluation Policy, Second regular session 2019 New York, Item 4 of the provisional agenda Evaluation DP/2019/29, http://web.undp.org/evaluation/documents/policy/ 2019/DP_2019_29_E.pdf. Accessed on 21 September 2023. United Development Programme. (2009). Handbook on planning, monitoring and evaluating for development results. UNDP. United Nations Evaluation Group. (2007). The role of evaluation in results-based management. UNEG. United Nations Evaluation Group. (2016). Norms and standards for evaluation. UNEG. United Nations Evaluation Group. (2019). UNEG strategy 2020–2024. UNEG. United Nations Evaluation Group. (2020a). Ethical guidelines for evaluation. UNEG. United Nations Evaluation Group. (2020b). Evaluation Use in Practice. A review of UNEG members’ practices to boost evaluation use. UNEG Interest Group on Evaluation Use. United Nations Evaluation Group/UN Development Coordination Office. (2021). Guidelines for the evaluation of the United Nations sustainable development cooperation framework. UNEG. United Nations Evaluation Group. (2022). United Nations contributions to national evaluation capacity development and the evolution of national evaluation systems. An overview of implementation of General Assembly Resolution 69/237 . UNEG.

15

THE INFLUENCE OF TRANSNATIONAL ORGANISATIONS

499

Vijayaraghavan, M. (2023). The Asian Development Bank and Evaluation in Asia and the Pacific. In R. Stockmann, W. Meyer, & N. Zierke (Eds.), The institutionalisation of evaluation in Asia (pp. 413–428). Springer. West Meiers, M. (2022). The Independent Evaluation Group (IEG) of the World Bank Group: Influences on evaluation structures and practices globally and in the Americas. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 417–447). Springer.

PART IV

Synthesis

CHAPTER 16

The Institutionalisation of Evaluation in Asia Pacific: A Synthesis Reinhard Stockmann, Wolfgang Meyer, and Niklas Zierke

1

Introduction

International comparative and comparative regional research frequently require the analysis of cases that exhibit significant heterogeneity in key theoretical dimensions, which is a fundamental methodological challenge in political science and sociological research.1 This is particularly true 1 Small-N comparative designs are naturally constrained by the limited number of cases under investigation, which in turn limits the range of theoretically possible variation that can be observed (see, Chapter 1, Part 4). Large-N studies, typically not comparative research with countries as the major unit of analysis, are necessary to observe a wider variation in many substantive aspects.

R. Stockmann (B) · N. Zierke Center for Evaluation (CEval), Saarland University, Saarbrücken, Germany e-mail: [email protected] N. Zierke e-mail: [email protected]; [email protected] W. Meyer Working Group Evaluation, Saarland University, Saarbrücken, Germany e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_16

503

504

R. STOCKMANN ET AL.

when analysing cases beyond subregions such as Western or Eastern Europe, South or Southeast Asia or Northern or Southern Africa, but if instead a larger world region such as Asia Pacific is considered, which covers various countries in Oceania, South, East and Southeast Asia as well as Australia. By definition, this regional summary encompasses a diverse range of subregions and countries. The diversity also becomes more than clear when looking at some of the basic characteristics of the country cases that were ultimately selected. First of all, the country selection for “Asia Pacific” comprises a total of considerable 45.7% of the world’s population, that is, 3,643 billion people (UN, 2022). This includes countries with very large populations such as China (1.425 billion), India (1.417 billion) as well as those with medium (Pakistan with 236 million, Bangladesh with 171 million, Japan with 124 million, the Philippines with 116 million), and relatively small populations (Nepal with 30.5 million, Australia with 26.2 million, Taiwan with 23.9 million, Sri Lanka with 21.8 million). There are similar vast variations in the territorial size of the countries examined. However, the conceptually relevant diversity of the cases begins when turning to the concrete historical, social, political and economic country contexts. To detail all these particularities would certainly go beyond the scope of this introduction, instead two indicators can be used to give an impression. The average national income per person (GNI per capita) ranges from countries such as Australia (GNI of US$56,760), Japan (GNI of US$42,620), and South Korea (GNI of US$34,980), which are among the wealthiest in the region, to those that are among the poorest in the world, such as Nepal (GNI of US$1,230), Pakistan (GNI of US$1,500), India (GNI of US$2,170), Bangladesh (GNI of US$2,620), and the Philippines (GNI of US$3,640) (World Bank, 2021).2 In addition to purely economic indices, looking at measures of social welfare such as the Social Progress Index (SPI), which takes into account social (such as nutrition and basic health care, access to education, health), legal–political (such as inclusion, personal rights, and freedoms) and environmental indicators, suggests the same conclusion: cases vary and are also subject to dynamic evolution. For example, countries such as Japan, Australia, and South Korea are among the twenty most socially progressive countries in the world, while the other countries are considered to have either a medium (China, Sri Lanka, the Philippines) or low (India, Bangladesh, Nepal, Pakistan) SPI (2022). If these two indicators are 2 Please note that it was not possible to report GNI for Taiwan as it is not covered in the referenced dataset.

16

SYNTHESIS

505

used and viewed over time, it becomes clear that many countries have experienced a significant and continuous economic and socio-political upswing over the last one to two decades. Moreover, the eleven countries studied differ in terms of their economic systems. According to Dongre (2020, p. 196), there are socialist market economies (China), development market economies (India, the Philippines; Nepal, Bangladesh, Pakistan, and Sri Lanka are also most likely to be classified here), coordinated market economies (Japan, South Korea, Taiwan) and liberal market economies (Australia). He emphasises the diversity and numerous special features of the economic systems and that there is no “one size fits it all” (ibid.). Furthermore, there are fundamental differences with regard to the political configuration of the countries studied. These differences extend across all three central objects of analysis of the political: polity, which concerns the central institutions and structures of politics; policy, as the content of politics; and politics, as the political processes (see Lauth et al., 2014). With regard to the institutional structures—the polity— it is clear that countries vary greatly in terms of rule-of-law principles, implemented forms of rule (democratic or autocratic) and established systems of government. The characteristics of polity also differ: political culture, such as political values/attitudes and political trust that are widespread in a society, as well as the entire field of political participation. Various political science indicators express this variance. For example, it is useful to look at the Economist Intelligence Unit’s Democracy Index, which expresses aspects of polity (e.g., functioning of government) and, more focused, those of politics (e.g., political participation, elections) in one measure, which are then classified into four groups based on calculated cut-off values: Out of 28 countries in the Asia/Australasia region, 18% are considered Full Democracy, 36% Flawed Democracy, 21% Hybrid Regime, and 25% Authoritarian Regime (EIU, 2021). After years of continuous increases in the average scores, the annually published reports show a stagnation from 2015 onward, which even resulted in a visible decline in the most recent report, which can partly be traced back to the socio-political impact of the Corona pandemic, negatively affecting a high share of countries but also socio-political changes and policy shifts in specific countries. Nevertheless, while the share of countries classified as authoritarian remains high, countries like Japan, Taiwan, and South Korea progressed—despite the worldwide and regional observable negative trend—and are now included (again) in the first category

506

R. STOCKMANN ET AL.

(see Sect. 3). In any case, a heterogeneous empirical situation becomes evident, which is subject to historical changes and some fluctuations. For the institutionalisation of evaluation, this represents a conceptually important explanatory factor, as it is plausible to assume that with such measured characteristics of the political system, the defined benefits and types of evaluation also vary which ultimately leads to specific practices and institutions. In addition to the differences in polity and politics, there are also differences in policies, which can be traced back, at least in part, to different processes and structures of the politics. The topic of “evaluation” as a policy content and thus subject of policy field analysis is no exception, as the results section of the synthesis chapter will underline. Good governance indicators can be used to numerically summarise and express various attributes in polity, politics, and policy, as well as other dimensions such as the economic system discussed above. Research has frequently underscored and empirically supported the link between good governance and evaluation (e.g., Dahler-Larsen and Boodhoo (2019) by focusing on the corruption perception index as a measure of one dimension of good governance). It is therefore plausible to assume that the observable variance in different good governance indicators in the Asia Pacific region, is expressed in differences in the institutionalisation of evaluation. Presumably, the individual dimensions of the complex good governance concept, such as political stability, government effectiveness, rule-of-law, voice and accountability, corruption control and regulatory quality (cf. Worldwide Governance Indicators of the World Bank) will correlate to different degrees with evaluation; conceptually, evaluation can in any case be seen as an instrument that can contribute to improving the various dimensions. Given this variability, it is an interesting question whether this translates into differences in the institutionalisation of evaluation. Cultural, socio-political or organisational practices that are assumed to diffuse and institutionalise globally are well suited to be studied empirically in a comparative way. Evaluation is such a phenomenon, very appropriate to inquire the global diffusion of institutions. There are many different variants of comparative social research— all with their own benefits and shortcomings. The comparative research approach of the Evaluation Globe project is based on a theoretically sound analytical framework as well as a coherent methodological design applied across the different volumes (see Chapter 1).

16

SYNTHESIS

507

The research approach is based on a uniform research guideline to ensure a high degree of comparability. All authors processed their articles using this structuring guide. In addition to the advantages for the analytical-empirical investigation of evaluation, some methodological challenges emerge that should be mentioned for a critical-realistic assessment of the findings: First, gaps in the analytical guideline were identified in some case studies, which were iteratively filled by multiple processing loops in coordination with the author teams. This was the basis for a cross-case comparison. Despite the elaborate review process, which pursued the greatest possible saturation of content, it was ultimately not possible to always obtain all information on every subcategory of interest in every individual case. The data collection instrument reached its natural limits at this point. Three case studies even had to be rejected due to a lack of content fit or incompleteness. The second problem, more serious than the first, is rooted in different conceptual understandings of evaluation. Despite various measures—such as the use of a glossary of terms—to base the contributions on a uniform basic understanding of the evaluation instrument in the narrower sense, there were conceptual deviations in the country case studies. Thus, empirically there are regional variances in the understanding of the term evaluation. For example, the term “evaluation” was often used relatively broadly and was not clearly distinguished from other terms such as audit, accreditation, monitoring, or impact assessment. In the course of the review, this was again explicitly emphasised in some places; in others, it was ensured that, apart from the examples mentioned, nothing was left out that concerns evaluation in the narrower sense. It turned out that it was not the authors themselves who arrived at this conceptual understanding, but rather that, from an empirical point of view, the conceptual nuances of the concept of evaluation have not been developed in many places and a fuzzy use of the term predominates. This was evident, for example, in many instances of reported evidence from media reports on “evaluation in the broader sense”. In many case reports, evidence was cited that, on closer examination and critical appraisal of the evidence, referred to accreditation, performance audits, or monitoring results. In the synthesis assessments, however, the focus was on “evaluation in the narrower sense”, which is why these examples did not carry any weight in the final rating.3 3 It should be noted that equity-based evaluation, gender-sensitive evaluation or indigenous evaluation, for example, can be subsumed under this; not, however, equity-oriented

508

R. STOCKMANN ET AL.

The results draw an overall picture with a certain degree of variation, which points to very country-specific institutions and practices in each case. Nevertheless, many cases are similar in the extent of institutionalisation, as the analysis will show. First, the institutionalisation of evaluation in the political system will be examined. On the one hand, the question of the legislative framework of evaluation will be addressed; on the other hand, topics such as the organisational anchoring in the political system, the significance of evaluation in the parliament, as well as the types and extent of the use of evaluations will be dealt with. The latter allows to address the question of whether the formal regulations are also translated into an ultimate practice, or whether they merely exist on the “formal structure”, to summarise it with the term neo-institutionalism (Boxenbaum & Jonsson, 2017, p. 78). This concerns the fact that organisations make adjustments on the formal level, while, on the other hand, the actual organisational practice remains unaffected. Reasons for this are seen in conflicting institutional and practice-related environmental conditions. For example, an organisation might formalise evaluations to generate legitimacy, while practically continuing to do “business as usual”. This is also referred to as decoupling between formal structure and organisational activity. This issue is revisited in the conclusions of the article. In addition, the institutionalisation of evaluation in the social system will be addressed, examining whether and how civil society actors use evaluations and are actively involved in them, whether they know and recognise the instrument of evaluation, to what extent public discussions about evaluations take place (media reception of evaluation), what the transparency of evaluation reports is like, and finally how different groups of actors’ demand evaluations. Subsequently, the results of the professionalisation system are presented. This concerns the dimension referred to in the literature as the “supply side”; this synthesis makes it possible to compare countries and examine whether the demands for evaluations can be adequately met. Finally, challenges in institutionalising evaluation are discussed, and the “driving forces” of institutionalisation are derived on the basis of the systematic comparison. Among other things, the information from the

performance audit, gender-sensitive monitoring or indigenous accreditation, for example. The assessment is thus concerned with the “type of organisational or policy practice”; and accordingly covers a variety of concrete applied evaluation theories, which are always welcomed reporting on in the case studies.

16

SYNTHESIS

509

case studies on the international organisations examined will be included in the overall analysis.

2 The Institutionalisation of Evaluation in the Political System If looking at the legislative structure in the countries examined here, it becomes clear that only very few countries have national (comprehensive for all departments and policy areas) legal regulations (cf. Table 1 columns 1 and 2). One exception is Japan, which adopted a Government Policy Evaluation Act in 2001, which obliges all sectoral ministries and governmental agencies to carry out policy evaluations. However, this is only understood as a “performance measurement”, which primarily serves the purpose of accountability (p. 196f., 203 and 206). Another exception is South Korea, where a Government Performance Evaluation Act (GPEA) was adopted in 2006, which orders all “central ministries: self-assessments and top-down assessments” (p. 220). In addition, the Ministry of Economy and Finance (based on the National Finance Act) has operated its own comprehensive evaluation system for budgetary purposes since 2006 (p. 221). The Prime Minister’s Office is required to create a medium-term evaluation policy at least every 3 years. The Ministry of Economy and Finance is required to create a medium-term evaluation policy every 5 years. In Australia, there is a Public Governance, Performance and Accountability Act of 2013 and a Commonwealth Performance Framework of 2015 which replace previous legislation. However, they do not constitute laws or regulations for carrying out evaluations, but rather are intended, among other things, to force administrations to produce “explicit links between activities and outcomes achieved” (p. 68). “Under the framework, all Commonwealth agencies must report on how performance will be measured and assessed” (p. 69). However, evaluations are by no means the only way of achieving this. It is emphasised, though, that the Act and the Framework “have triggered reforms to evaluation systems and processes that are taking shape across government” (p. 68). Australia takes a very decentralised approach to evaluation. Therefore, the individual departments are free “to formulate evaluation strategies and plans

Australia Bangladesh China India Japan South Korea Nepal Pakistan Philippines Sri Lanka Taiwan

National laws and strategies a 0 0 0 0 1 1 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0

National decrees b 1 0.5 0.5 1 0.5 0.5 0.5 0.5 0.5 0.5 1

Organisational embedding c

Institutionalisation of evaluation Role of court of auditors d 0 0 0 0 0 0 0 0 0 0 0 0.38 0.38 0.13 0.13 0.13 0.25

0.38

0.25 0.38 0.38 0.25

Mean

Legislative institutionalisation of evaluation and evaluation use

Criteria countries

Table 1

0 0 0 0 0 0 0 0 0 0 0

Role of parliamente 1 0.5 0 1 1 1 0.5 0 0 0 0.5

Sectoral spread f

Use of evaluation Scope of evaluation practiceg 0.5 0.5 0 1 0 1 0 0 0 0 0 0.5 0.33 0 0.67 0.33 0.67 0.17 0 0 0 0.17

Mean

0.36 0.36 0.21 0.43 0.36 0.5 0.21 0.07 0.07 0.07 0.21

Overall Mean

510 R. STOCKMANN ET AL.

(continued)

by the parliament (existing = 1, not existing = 0) b As national decrees, all regulations are considered which are not laws (as described under 1) but refer to the whole nation (i.e., not only to particular policy fields) (existing = 1, not existing = 0) c Hereby is meant, the existence of specific evaluation units in government institutions (such as ministries) and/or the existence of independent stand-alone bodies of evaluation (existing in both categories = 1, existing in one category = 0.5, existing in no category = 0) d National audit offices carry out not only performance audits (which are limited to the evaluation of goal achievement (effectively) and/or efficiency) but also carry out evaluations with a broader focus (yes = 1, no = 0) e Two criteria were evaluated: (A) Parliament has an evaluation unit at its disposal and commissions evaluations (B) Parliament regularly takes note of and discusses evaluation results (If both criteria are met = 1; one criterion = 0.5; no criterion = 0) f Degree of spread across sectors, number of sectors (policy fields) mentioned in the case studies in which comprehensive evaluations are carried out (above 7 = 1; between 6 and 5 = 0.5; below 4 = 0) g Intensity and frequency by which evaluations are carried out. Qualitative assessments according to case studies. Ratings in the text in brackets Source Author’s own work

a Here, only those laws were included that have a comprehensive national validity. As national laws, only such laws are considered, which were passed

Table 1

16 SYNTHESIS

511

512

R. STOCKMANN ET AL.

(or not) as they see fit” (Questions Annex). Accordingly, Australia does not have an “overall evaluation policy”.4 Nepal is an interesting case as “the only country in the Asia Pacific region [...] having evaluation directly embedded in the national constitution” (p. 240), which, however, has still not been transposed into law. Although the draft has already passed through the Cabinet of Ministers, it is still awaiting approval by the two Chambers of Parliament. However, some constitutional provisions regarding evaluation at a national level were implemented.5 Furthermore, there are sectoral regulations and provisions (p. 242f.). The situation in the Philippines is very similar. Although a “ResultBased National Evaluation Policy” is in progress, it has not yet been adopted by Parliament. The National Evaluation Framework developed by the National Economic and Development Authority, which covers all sectors and institutions in the country, could not yet be fully implemented. In addition to a number of other reasons, the political climate is primarily blamed for this (p. 329). In Bangladesh, the “Rules of Business” of 1996 make “evaluation of execution plans mandatory for government ministries in the country” (p. 98). IMED has developed a Monitoring and Evaluation Policy Framework which has the following objectives: . . . .

to ensure accountability to assist in ensuring results from development investments to support in the assessment of impacts of development interventions to improve effectiveness and results orientation in future strategies, policies and programmes by obtaining feedback from past learning (p. 102ff.).

There are sectoral regulations as well. In China, there are also no universal laws or regulations on evaluation (p. 128), but there are a series of administrative decrees and regulations 4 Since the Public Governance, Performance and Accountability Act and the Commonwealth Performance Framework does not constitute a national evaluation regulation, the criterion was evaluated as zero in Table 1. 5 Since the M&E paragraph has been added to the constitution but has not yet been incorporated into a law, its inclusion in the constitution (column 1) is rated as 1 and concerning the decrees (column 2) as 0.

16

SYNTHESIS

513

enacted and implemented by the central government and the Chinese Communist Party. These administrative decrees and regulations contain not only principles or guidelines for performance management “but more often than not, also very detailed evaluation measures” (p. 128). In Taiwan, while there are also no universally valid laws or decrees at the national level and no national strategy for evaluation (p. 394), “in a total of 800 laws, regulations, and statutes” evaluation is prescribed in one form or another (p. 393). In Pakistan, there is no national evaluation policy, but there are frameworks and guidelines “that help various departments assess programme evaluation” (p. 278). Due to the lack of general laws and decrees, some departments have developed their own systems to streamline evaluation processes (p. 279). In India (p. 150) and Sri Lanka6 (p. 357), there are no laws, policies, decrees or executive orders governing evaluation either at the national or at the sectoral level. In summary, it can be stated that there are very few legal regulations and administrative decrees in the countries examined in the Asia Pacific that apply universally to all departments in the political system at the national level. Organisational institutionalisation is also required for the planning and implementation of evaluations (cf. Table 1, column 3). The following structural solutions can be identified in the case studies: . In Australia, evaluation is organised in a decentralised manner within the political system, i.e., via the individual ministries and agencies. The Department of Finance has published an evaluation policy and a toolkit on its website to support evaluations. In addition, a number of specialised (semi-independent) public agencies have been established in Australia to carry out evaluations in selected policy areas (e.g., criminology, health, and welfare). . Although India does not have universal evaluation laws and a national evaluation policy, evaluation has been taking place here since the early 1950s. As early as 1952, a separate evaluation unit was 6 Although the Government of Sri Lanka endorsed a National Evaluation Policy in 2018 (and is thus the only country in South Asia), the National Evaluation Policy Framework necessary for its implementation has not yet been finalised and approved by the Cabinet (p. 357f.).

514

R. STOCKMANN ET AL.

established under the National Planning Commission. Today, this task has been taken over by the Development Monitoring and Evaluation Office (DMEO), which is mandated at a national level “To institutionalize application and use of monitoring and evaluation at all levels of government policy and programs” (p. 157). In addition, DMEO is also working on a national evaluation framework and standards, has advisory powers across the Ministries and Departments of the Union Government and has direct access to the government, which makes it a powerful organisation (p. 158f.). In addition, major ministries have their own evaluation units and practices. . In Nepal, the National Planning Commission (NPC) serves as a central agency for monitoring and evaluating development policy, plans and programs (p. 246). The so-called “development ministries” have M&E units, but their evaluation activities are dominated by the donor organisations.7 . In Taiwan, the National Development Council (NDC) plays an important role, “in providing impact evaluations of proposed policies and laws, and additionally …provides policy recommendations through process evaluations” (p. 392). In addition, the legislative Yuan disposes of a Research Bureau “which is responsible for evaluating laws and policies” (p. 392) and also “other branches of the national government as well as ministries and governmental bodies have set up their own units to evaluate policy and performance” (p. 396). . In Bangladesh, evaluation is centrally regulated in the political system. A department of the Ministry of Planning, the Implementation Monitoring and Evaluation Division (IMED) “is centrally responsible for monitoring and evaluation the performances of all ministries and divisions” (p. 100). These, in turn, are obliged to monitor and evaluate their activities themselves and to report to the relevant government bodies, including IMED (p. 100).

7 The Nepal case study says: “A significant share of the development budget in Nepal comes from donor agencies in the form of grant or loan projects” (p. 240). Moreover, elsewhere it states: “Generally, government institutions do not have sufficient capacity to conduct evaluations” (p. 248). Therefore, an overall rating of 0.5 (column 3 in Table 2) was given here.

16

SYNTHESIS

515

. In Sri Lanka, there exists only one Department of Project Management and Monitoring which performs evaluation tasks, among other things (p. 363). . The organisational institutionalisation of evaluation in Japan is characterised by the non-existence of independent national institutions that are specialised in policy evaluation. There are no independent national institutions specialising in policy evaluation. Instead, a number of ministries have established their own advisory boards to review their performance and evaluation results (p. 204). . The organisational institutionalisation of evaluation in Pakistan and the Philippines is defined as weak. The National Evaluation Policy Framework (NEPF) in the Philippines “was not yet legally established” (p. 329), and only a few ministries have M&E units (p. 334f.). NEDA, “the country’s primary socioeconomic planning body” is primarily concerned with “macroeconomic forecasting and policy analysis” (p. 331) and “is responsible for the national and sub-national coordination for the formulation of policies, plans and programs” (p. 331f.). In Pakistan, a number of activities are ongoing to establish an evaluation system (p. 289), but currently, only a few ministries have M&E units (p. 290). . South Korea also does not have a national evaluation body. Instead, evaluation committees consisting of external experts carry out the evaluations on behalf of the Prime Minister’s Office or individual ministries (p. 220). . In China, the evaluation system consists of a “performance appraisal and performance management system”, which, in turn, are controlled by two systems: the party system and the executive system (p. 131). This is conducted according to strictly hierarchical principles: “The central government ministries and agencies evaluates its line offices’ performance and upper-level government evaluates its direct lower-level government’s performance intermittently and annually according to the present performance indicators” (p. 131). In summary, it can be said that the organisational anchoring of evaluation is only very pronounced in relatively few Asian Pacific countries. Exceptions include Australia, India, and Taiwan where independent evaluation institutions exist and ministries have their own evaluation units. Most other countries rely on either a centralised (Bangladesh, Sri Lanka)

516

R. STOCKMANN ET AL.

or a decentralised form of organisation in which individual ministries have evaluation units (Japan, South Korea, Pakistan, China). However, it is regrettable that the aforementioned evaluation institutions are mostly not independent institutions, but rather departments of planning ministries or comparable institutions, which, in addition to other tasks, also perform monitoring and evaluation tasks, whereby with some monitoring predominates. If the evaluation function is generally carried out by the ministries, there are usually only a few ministries that perform this task, and the units are not well equipped and also carry out more monitoring or other tasks than evaluation tasks themselves. The role that evaluation plays in national courts of auditors is used here as an additional indicator of the institutional anchoring of evaluation in the political system (cf. Table 1, column 4). The analysis of the case studies shows that in none of the countries does the court of auditors carry out evaluations that go beyond finance audits or performance audits. That is why, for example, in Australia, in addition to the Auditor General, parts of the evaluation community also call for an “Evaluator General” (p. 72). While the criteria discussed so far are used to analyse the legal, institutional and organisational anchoring of evaluation, criteria that describe the sectoral breadth, the intensity and the extent to which evaluations are carried out as well as the use they provide will be examined below. These indicators are intended to show the extent to which evaluations are used in parliament and in individual policy areas. Of particular importance is the role of parliament, since evaluation results can be used, among other things, for legitimising political action, ex-post for controlling the government (accountability), ex-ante for legislative and budgeting processes and, in general, for making rational decisions in the sense of evidence-based policy. However, members of parliament can not only be users of evaluations but they can also influence the role of evaluations in the political system due to their law-making power by enacting evaluation laws, making evaluations mandatory in laws or even commissioning evaluations themselves. However, evaluation is rarely used for these purposes in the Asian Pacific countries. Although there is a standing committee for monitoring and evaluation in the parliament of Bangladesh, the case study states: “the role of evaluation reports in the parliamentary debates in Bangladesh remains unclear” (p. 98).

16

SYNTHESIS

517

In some countries, however, there do seem to be individual representatives and smaller parliamentary groups working to carry out and use evaluations. For example, in India (p. 153), Nepal and Sri Lanka. In Nepal, this has even led to the evaluation being enshrined in the constitution. A parliamentary forum comprising five major political parties contributed to this in 2013 (p. 244). However, the implementation acts necessary for carrying out evaluations have not yet been adopted, national policies are very rarely evaluated, and members of parliament do not play a role in this. In addition, the results of evaluations, even if they are of greater national interest, are rarely discussed (p. 244). Although there is a parliamentary committee for Implementation, Monitoring and Evaluation in addition to the parliamentary forum, which is still active, it does not seem that a broader circle of the members of parliament has been able to get enthusiastic about the subject. In Sri Lanka, too, there are parliamentary forces that advocate for the establishment of evaluation structures and have launched a National Evaluation of Development Interventions and Public Policies Bill. The Parliamentary Standing Committee on Evaluation (PSC), which was set up in 2019, was largely responsible for this. The draft bill has proposed to establish a National Commission on Evaluation with extensive evaluation tasks, but this has yet to be adopted. With the change of government in 2020, the Standing Committee was dissolved, which “interrupted the momentum of the legislative process” (Sri Lanka, p. 384). As early as 2016, a Parliamentarians Forum for Evaluation (SLPFE) was established in Sri Lanka, an informal network of like-minded parliamentarians with the aim “to advocate for the use of evaluation within parliamentary processes” (p. 362), which remains active. Here, South Korea is no exception. The case study highlights the fact that the National Assembly can commission the National Assembly Budget Office (NABO) and also external experts as well as research institutes to conduct an evaluation, but parliaments of other countries can also make use of this. In addition, the NABO can reject such requests coming from the National Assembly (p. 224). In summary, it can be stated that evaluation results in the parliaments of the countries examined do not play a particularly large role. As a rule, they are not discussed or used for political decision-making, if they are taken into account at all. The low level of interest or understanding of the potential use of evaluations is also expressed in the fact that there are no legal regulations in this regard in most countries.

518

R. STOCKMANN ET AL.

However, there are a number of countries in which parliamentary groups and commissions have been formed that are committed to advancing the issue of evaluation, particularly in Nepal and Sri Lanka. The next step is to examine the extent to which evaluations are being carried out. To do so, this paper will first describe the sectors (policy fields) in which evaluations are carried out frequently and with a certain regularity in order to determine, on the one hand, the range (sectoral spread) and on the other hand the intensity (the scope of evaluation practice) (cf. Table 1, column 8). As shown in Table 2, there is a wide range of sectors in which evaluations are regularly carried out. The sectors most evaluated are education, health, agriculture, and social development/welfare. The sectors with the least evaluations include foreign affairs, indigenous affairs, internal affairs, defence, and tourism. In most countries, evaluation is mainly carried out in the sectors that are financially supported by international donors and therefore have to evaluate “mandatorily” (Table 3). As in the previous evaluation globe volumes, it is difficult to say anything about the scope, frequency, type of evaluations and their quality. With regard to this criterion, we would like to remark self-critically that we have overwhelmed the case study authors, because these criteria vary too much across the individual sectors. But with the limited number of Table 2 Sectors where evaluations are conducted

Sector

Number of entries

Education Health Agriculture Social Development/Welfare Economy/Industry Science, Technology Employment/Labour Environment Justice Others8

9 7 6 6 5 4 3 2 2 8

Source Author’s own work

8 Only one entry for each.

16

Table 3 Degree of spread across sectors where evaluations are conducted on a regular basis (cf. Table 1, column 8)

SYNTHESIS

519

High (1)

Medium (0.5)

Low (0)

Japan India Australia South Korea

Nepal Bangladesh Taiwan

Philippines Sri Lanka Pakistan China9

Note (1) = spread in seven sectors or more (0.5) = spread in five to six sectors (0) = spread in four sectors or less Source Author’s own work

pages given to each case study, it was impossible to respond with the necessary differentiation. In the following qualitative analysis, however, an attempt will be made to extract the scope, frequency, and intensity of evaluation practice and quality assurance measures from the case studies (in alphabetical order) (cf. Table 1, column 9): . Although there are numerous ministries in Australia that have evaluation units, they do not have effective evaluation structures and processes everywhere. The case study states: “The frequency and extent of evaluation activity across different policy sectors in Australia ebbs and flows” (p. 75), depending on government investment in reform activities: “This is because increases in expenditures relating to new policies often include mandatory budget allocations for evaluation” (p. 75). Internal evaluations are often carried out; but also, external ones conducted by consulting companies. Overall, it can be said that there are no “dominant types of evaluation”. For years there has been an ongoing debate “about the need to enhance the general quality and methodological rigour of evaluation studies” (p. 75). Among other things, the low number of long-term and impact evaluations is criticised as is “the extent of evaluation and whether evaluation effort is being appropriately targeted” (p. 74), the lack 9 An assessment for China is difficult to be carried out, because although “evaluations”

are conducted on a regular basis in all sectors, the evaluations tend to be (performance) audits that meet strict requirements. Therefore, the rating here is “0”. In a different interpretation (which would contradict the definition of evaluation given in the glossary, which is binding for all), the value “1” could also be given. But as the case study authors themselves have written that evaluation is an “elusive concept” in China (p. 125).

520

R. STOCKMANN ET AL.

of application of scientific research designs and insufficient orientation towards the information needs of the user. Meta-evaluations for quality assurance of evaluation are rarely carried out. These shortcomings have led to a situation where “a highly negative assessment of evaluation institutionalisation within the APS [Australian Public Service] emerged” (p. 74) and evaluation results are not being used sufficiently (p. 76) (Rating: 0.5). . Bangladesh relies on a centralised evaluation approach, with IMED—the main player—acting as “Bangladesh’s apex body for monitoring and evaluation” (p. 98). Around 22 million euros were made available for the department in the Ministry of Planning for the year 2022/2023. These funds will be used, among other things, to carry out impact evaluations. Since 2009/2010, IMED has carried out 167 impact evaluations. These evaluations were mainly project/programme evaluations. IMED’s annual report, which is sent to the National Economic Council at the end of each fiscal year (p. 102), highlights above all “financial and physical progress”. IMED is also responsible for quality assurance and has carried out an internal project entitled “Strengthening M&E Capabilities of IMED” (p. 104) as well as meta-evaluations (follow-up surveys) (Rating: 0.5). . In China, only a few evaluations (in the sense definition used in this study defined here) are carried out. The established “evaluation system” is rather equivalent to an audit and performance management system, since no evaluation system has been established, but rather a performance management system that is mainly used to “stimulate public officials’ gaming behaviour” (p. 133)— without, however, achieving this goal. This is mainly due to the fact that the activities are combined with an economic responsibility auditing, which is used to check. In addition to various financial audits, economic responsibility auditing primarily contributes to the negative image of evaluations, which are, in reality, actually audits. Responsibility audits are primarily used to check the extent to which officials and managers perform their duties (p. 133). The evaluations carried out within the framework of the Targets and Responsibility System (TRS) are used “for the reward and punishment of local government leaders, agency heads, local government as a whole and individual administrative agencies and offices” (p. 131) (Rating: 0).

16

SYNTHESIS

521

. India not only has a Development Monitoring and Evaluation Office (DMEO), an attached office of the National Institution for Transforming India (NITI), but there are also a number of ministries that have their own evaluation units and regularly perform evaluations. Although there are no national evaluation laws, decrees or policies in India, evaluation is “widely practised in the whole country” (p. 163). The Programme Evaluation Organization (PEO), which was incorporated into DMEO in 2014, carried out a total of 236 evaluations between 1954 and 2016 (approx. 4 per year) (p. 161). In recent decades it has been observed – that the evaluations are no longer mainly carried out by internal agencies like PEO, but mostly by external independent research organisations or civil society organisations (p. 164) – that there was an additional methodological development, whereby not only quantitative methods such as RCTs are used, but also qualitative methods (mixed method approach) (p. 164) – that with the commitment to international goals, MDGs earlier and SDGs now, the spectrum of evaluation criteria has expanded – that the “credibility of evidence collected and reported” has increased, thanks to the commissioning of external evaluators (p. 164) For quality assurance purposes, only one meta-evaluation has been carried out in India so far, which included 110 evaluations, all of which were carried out before 2010. The results at that time were quite satisfactory: – in two-thirds of the evaluations, the quality methodology was rated as “satisfactory” – three-fourths of the reports were satisfactory in quality of conclusions, lessons learned and recommendations – and about half of the evaluations were rated satisfactory in assessing efficiency, sustainability, and impact (p. 167)

522

R. STOCKMANN ET AL.

As in other countries, it is regrettable that “the level of evaluation use is relatively low” (p. 165), whereby the implementation rate of recommendations cannot be determined due to a lack of implementation monitoring and surveys (Rating: 1). . In Japan, evaluation is largely the responsibility of the ministries themselves, but they are hardly in a position to do so because they do not have the appropriate units that have enough qualified evaluation experts (p. 201, 204). Instead, the assessments carried out internally by “ordinary government officials” consist of filling in anteand ex-post evaluation sheets in order to measure the performance achieved (p. 204). The highly standardised “evaluation reports”, as they are called, are then forwarded to the Ministry of Internal Affairs and Communications (p. 205). The evaluations carried out by the Ministry of Foreign Affairs (MOFA) are more rigorous and are based on ODA criteria and standards, but only concern international cooperation (p. 206) (Rating: 0). . The South Korea case study points out that “evaluation activities are embedded in almost every sector and evaluation results are used to justify policies or identify problems and improvement plans” (p. 226). Many different types of evaluation are used. While the Prime Minister’s Office (PMO) and the Ministry of Economy and Finance (MoEF) mainly carry out impact evaluations,10 the sectoral ministries focus on formative evaluations in order to improve programme development. The PMO mainly uses the results “to improve organisational performance and policy effectiveness” and the MoEF, for budget formulation. The main focus is on measuring effectiveness and efficiency (pp. 226f). The sectoral ministries sometimes also carry out internal evaluations “to justify their programmes”. The quality of the evaluations is ensured by the commissioners. Sometimes meta-evaluations are also carried out. Since South Korea has “many qualified researchers and strong public research institutes”, “the qualification of evaluators is usually guaranteed” (p. 227) (Rating: 1).

10 The PMO has conducted 2 to 4 policy evaluations since 2017 and the MoEF has conducted around 8 in-depth evaluations (follow-up surveys).

16

SYNTHESIS

523

. In Nepal, the National Planning Commission (NPC) manages the overall M&E system and “facilitates evaluations, engaging third parties hired through competitive processes” (p. 247). Each year, the NPC selects programmes and projects “using specific criteria received from line ministries” (p. 247). Steering committees are set up for the individual evaluations to approve the terms of reference, to select evaluators and to ensure the quality of the entire evaluation process (including the report). Although all ministries are supposed to evaluate and sometimes even have their own M&E divisions for this purpose, “the government institutions do not have sufficient capacity to conduct evaluations” (p. 248). There is also a lack of funding. In the 2021 budget, with a total budget of 16.47 billion, a budget of only 1.49 million NPR was earmarked for M&E, around 0.09%. Most ministries are therefore limited to monitoring. Evaluations are only carried out for megaprojects and above all for donor-funded projects. This is why they account for a large proportion of the evaluations carried out in Nepal (p. 248). The types of evaluations vary widely, but mostly focus “on the results of processes and delivery of outputs rather than on outcomes and impacts” (p. 251). To ensure that the results are useful, Management Response Plans are supposed to be created after each evaluation. However, this rarely happens: “Both the demand and supply sides of evaluation have limited capacity to facilitate, conduct, and use outcomes of rigorous evaluations” (p. 249). Therefore, the case study concludes “that use of evaluation is very low” (p. 253). The donor dominance in the evaluations is particularly evident from the statement: “In many cases, evaluations are carried out to fulfil the donor’s interest but they are hardly used for improvement, accountability and learning” (p. 253). One positive aspect, however, is that all evaluation reports are made public and are easily available on government websites (p. 252) (Rating: 0). . Although some ministries in Pakistan carry out evaluations, the focus is largely on monitoring (p. 280). Although the evaluation processes are well-designed from a technical standpoint (p. 281), they are “seldom used” (p. 283). Instead, forms of the responsible Planning Commission (PC) “are used more as bureaucratic, box-checking exercises rather than as tools to improve and rethink mechanisms and systems for robust evaluation” (p. 283). In addition, it must be taken into account that the M&E framework of

524

R. STOCKMANN ET AL.

the PC was not linked to capacity building measures. This situation ultimately depends heavily on the financial allocation for M&E, which amounted to 0.0008% of the total budget at the federal level in the 2021–2022 financial year (p. 281). The Prime Minister’s Inspection Commission has therefore criticised “the lack of a proper M&E system” and “demanded the immediate revival of Evaluation Section of Planning Commission with adequate manpower and conducting post evaluation of at least 10% completed projects in a financial year” (p. 284). The PC itself is dissatisfied with the current situation but points out that executing stakeholders are not very interested in changes. This attitude within the government and bureaucracy is explained by the fact that the focus is on the use of the allocated funds and there is little interest in the outcome or impact. In addition, it is pointed out that even “prominent M&E experts in the country believe that evaluations are typically carried out to fulfil requirements of donor funded projects” (p. 16). A problematic aspect listed is that Pakistan’s M&E culture is overwhelmingly focused on accounting and not on accountability. There is no kind of quality assurance available or even provided for in the Pakistan’s M&E “system” (p. 16). Still, reform efforts on the part of the government can be observed in the strengthening of the M&E department in the PC, the use of new technologies to strengthen the monitoring function and the contacting of external companies to carry out evaluations (p. 14) (Rating: 0). . In the Philippines, there is a National Economy and Development Authority (NEDA), but it does not fulfil its evaluation tasks (p. 331 and p. 332). Although there are some ministries that have M&E units, many barriers hinder their work, “such as financial barriers and varying levels of evaluation capacities across agencies and levels” (p. 335). If evaluations do take place, they are output-oriented and results-oriented evaluations are hardly ever carried out. The results of evaluations are “severely undocumented or inaccessible” for the public to view (p. 336). In addition, often only monitoring is carried out. In order to eliminate the shortcomings, UNDP has launched a strategic M&E project with NEDA, which includes the following components:

16

SYNTHESIS

525

(a) managing the conduct of independent evaluations (b) supporting the implementation of the National Evaluation Policy Framework (c) providing learning opportunities on evaluation (p. 337) UNDP has also commissioned a number of independent evaluation studies on behalf of NEDA. In addition, UNDP supports the Philippines in the implementation of the National Evaluation Framework, which has hardly been applied in practice so far (p. 337f., 347f.) (Rating: 0). . About 50 evaluations were carried out in Sri Lanka between 1994 and 2007, supported by UNDP. However, nowadays “it is not uncommon that government agencies undertake project reviews that are less costly and less time-consuming as a substitute arrangement for evaluation” (p. 366). Currently, a total of about 10 original project evaluations are carried out annually by the Department of Project Management and Monitoring (DPMM), which is responsible for evaluations. Since around one-third of the capital budget in Sri Lanka is foreign-funded, most evaluations are donor-driven and follow donor requirements. One of the biggest challenges is “the non-existence of allocated budgets for evaluations” and the lack of evaluation expertise. There is also no indication that this could change in the foreseeable future (p. 366f). According to public officials, the “use of evaluation is limited or not evident at all” (p. 368). The reasons given for this are: – – – – – – – –

small number of evaluations conducted non-implementation of National Evaluation Policy lack of evaluation capacity resulting in low-quality evaluations lack of understanding of the difference between monitoring and evaluation lack of management response to evaluations limited dissemination of evaluation findings socio-political influences resistance to donor-driven or independent evaluations (p. 368f.) (Rating: 0).

. In Taiwan, evaluations are conducted within a number of sectors. While evaluations are “omnipresent” across sectors, the frequency and intensity vary widely. Most evaluations are internal evaluations

526

R. STOCKMANN ET AL.

(p. 396). Since evaluation is part of Control Yuan, which is responsible “for auditing government agencies, censuring civil servants, and impeaching public officers” (p. 392), evaluation is perceived as a control instrument and focuses strongly on efficiency criteria (p. 399). Even if many evaluations are carried out in Taiwan “there is not yet an intrinsic culture of evaluation” (p. 399) (Rating: 0). When the degree of legal and organisational institutionalisations, as expressed in the criteria (1) existence of national laws, (2) national degrees, (3) organisational embedding in governmental organisations, (4) role of evaluation in audit courts in Table 1, are compared with the criteria that say something about the use of evaluation such as (5) role of evaluation in parliament, (6) degree of sectoral spread and (7) scope of evaluation practice, this yields the following results shown in Fig. 1. First of all, it is striking that almost all countries have a very low degree of institutionalisation (0.13–0.25). Only South Korea, Japan, Bangladesh, and China are slightly above concerning this criterion. This is mainly due to the fact that only a few countries have nation-wide evaluation laws and decrees. Furthermore, the fact that in all countries the court of auditors 1

0.9

0.8

Institutionalization

0.7

0.6

0.5

0.4

CHN

BNG JPN

NPL

KOR

0.3

AUS

TWN

IND

0.2

PAK PHL SLA

0.1

0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Evaluation Use

Fig. 1 Institutionalisation and evaluation use in Asia–Pacific (Source Author’s own work)

16

SYNTHESIS

527

does not carry out evaluations (in the definition on which the analysis is based) also contributes to the overall low assessment of the degree of institutionalisation. Only in the case of the criterion “organisational embedding of evaluation” could all countries score points, since they have at least rudimentary evaluation structures in the form of evaluation units in ministries and authorities or in the form of evaluation institutions or both. Although the application and usage dimension also has an overall low rating, the spread (from 0 to 0.67) is significantly more pronounced than in the institutionalisation dimension (from 0.13 to 0.38). The criteria “sectoral spread” and “scope of evaluation practice” vary particularly widely. On the other hand, there are no differences with regard to the “role of parliament” criterion, since the parliaments in all the countries studied make little or no use of evaluation results. Looking at the relative position of each country in Fig. 1 it is noticeable that there are no clear champions, i.e., those who have both a high degree of institutionalisation and a high implementation profile. In the majority of countries, those with a very low degree of institutionalisation (0.13–0.25) have, at the same time, such a low degree of utilisation that they could only be given a rating of 0–0.17 (China, Pakistan, Philippines, Sri Lanka, Nepal, Taiwan). However, there are also countries such as Australia and India which have a similar low institutionalisation value (=0.25) or, such as South Korea have a slightly higher use value of 0.38, which then have a significantly higher utilisation value (Australia: 0.5, India: 0.67, South Korea: 0.67). This means that some countries (Australia, India, South Korea), despite a relatively low degree of institutionalisation, have managed to achieve a degree of application that falls at least in the middle. It appears that a low legal and organisational anchoring of evaluation does not automatically prevent its application and use. A closer look also shows that there is no difference in the extent to which evaluations are implemented and used (measured by sectoral spread and scope) whether the evaluation function is organised primarily centrally (as in Bangladesh, Sri Lanka, Taiwan, the Philippines) or decentrally (as in South Korea, Japan, Pakistan). This makes it clear that both forms of organisation can lead to the use of evaluations. With regard to the use dimensions, apart from the principal lack of use of evaluation results by the national parliaments, it can be stated that

528

R. STOCKMANN ET AL.

evaluation is mainly used for accountability purposes. For many countries, the criticism formulated in the Pakistan case study is symptomatic: “Many in the government continue to use evaluation as a tool to highlight success rather than use it as a scientific method to assess gaps and redesign policies for the future” (p. 308). This preference is likely mainly due to the fact that many evaluation activities are linked to budgeting and financial control processes (cf. e.g., Australia, South Korea, Philippines, Taiwan) and are used for control purposes by planning ministries (cf. e.g., Bangladesh, Pakistan, Nepal, Sri Lanka, Taiwan).

3 The Institutionalisation of Evaluation in the Social System 3.1

Introduction

The “social system”, as it will be referred to in the following, is receiving increasing attention from a multidisciplinary scientific community.11 This relates to the continuously increasing practical relevance and increasing activities, both in historically differentiating civil societies and the third sectors.12 While the success story of the third sector has been interpreted in the context of social diagnoses of a “decline of the state” or the “triumph of the market” (Ghosh, 2009) which strengthened new forms of non-profit coordination, civil societies’ differentiation is frequently discussed in the context of democratisation processes and the rise of 2007) with civic engagement democratic governance (Wnuk-Lipinski, ´ being the central category of analysis (cf. Wagner, 2012). Besides these general explanatory factors, complex interactions with (national) state policies as well as regulations and concrete institutional structures need to be considered (cf. e.g., DiMaggio & Anheier, 1990); which also come to bear in historical terms as own traditions and approaches (cf. Salamon & Anheier, 1998; Salamon et al., 2017). As discussed in Chapter 1, civil society and third-sector actors fulfil various societal functions, among others, they are important forces in

11 Please note, that this societal system covers both the rather economic concept of the “non-profit sector” and the rather political concept of “civil society”, which must be analytically distinguished (cf. Wagner, 2012). 12 For the development of the non-profit sector, see e.g., Anheier and Ben-Ner (1997). For the history of civil society, see, e.g., Kaviraj and Khilnani (2001) and Kocka (2006).

16

SYNTHESIS

529

addressing social challenges—this is true for both the Global South as well as the Global North. The “social system”, often understood as an intermediate sphere, sits alongside state and market coordination instances, has complex interactions with them and yet follows its own systemic logics. In the Evaluation Globe research project, the social system is accordingly not considered in the sense of the “welfare state model”, but as a systemic entity in its own right, in which social interactions, agency, coordination as well as behaviour of various third sector and civil society actors take place. The social systems of the Asia Pacific region can be considered heterogeneous in many respects. Different political cultures, fundamental differences in recent contemporary history, governance systems13 that diverge from each other and widely varying social structures and mechanisms contribute to this diversity of social systems. One of the historical features that contributed to the plurality of systems can be seen in the context of democratisation processes in which civil societies played a significant role. For example, the “third wave of democratization” (Huntington, 1993) in some countries in the region, such as the Philippines (Silliman & Noble, 1998; Yeung, 2002, p. 230), South Korea (Kim, 2000) and Taiwan (Tak-Wing, 1993; Tien & Chu, 1996), led to entirely new models of political and social order. For the first time, the proportion of democratic or liberal societies in the region outweighed those that were authoritarian or repressive.14 Nevertheless, the proportion of

13 For example, the different governance systems as well as other characteristics of the political system likely contribute to different forms of participatory evaluation. Depending on the concrete setting, a varying contribution to strengthening deliberative democratic dialogue is likely. For example, in an autocratic political context participatory politics is unlikely to be widespread, which should also be reflected in the respective approach to participatory evaluation, e.g., expressed in different “gradations to participation”—if these are practised at all (Cullen & Coryn, 2011, p. 32). 14 Comparative research on democracy has revealed, that civil society can be characterised as an important enabling condition for democracy (Merkel, 2016, p. 467); but it does not necessarily imply democratisation processes—it is a necessary but not sufficient condition. This means that democratisation is a multifactorial social phenomenon with many intervening variables. In the case of the Asia-Pacific region, active civil societies in the context of the third wave of democratisation represent a decisive explanatory factor (e.g., for the South Korean case see Ku-Hyun & Inchoon, 2001). To use Robert Putnam’s terminology, different civil society actors succeeded in “building bridges” (Putnam, 2000) and thus in creating strong “social capital” (Putnam, 1993) that provided the seeds of change.

530

R. STOCKMANN ET AL.

“hybrid regimes” (Diamond, 2002) in which autocratic elements play a role as well as authoritarian regimes remains high in the region (EIU, 2021). In China, for example, a special form of “state-led civil society” (Frolic, 1997) has emerged as a result. Consequently, historically determined socio-political contextual factors essentially contribute to the structuring of social systems. The result is a colourful patchwork of social systems in the region, each of which provides a very specific framework for the operation of civil society actors. This variation naturally poses a challenge to a systematic comparison of social systems. Perhaps as a result, there is limited previous international comparative research on the institutionalisation of evaluation in social systems. Furubo et al. (2002) and Jacob et al. (2015) have addressed the subcategory of “media discourse” in their comparative study; however, it wasn’t a multidimensional approach to civil society as a functional differentiated societal subsystem. Apart from that, comparative studies on the use of evaluation in organisations in selected countries can be found (e.g., Kang et al., 2012; Mitchell, 2014; Mueller-Hirth, 2012; O’Leary, 2017; Wahlén, 2014). Given this, it seems a reasonable and relevant question from a comparative research perspective to ask whether the vast differences in social systems affect the institutionalisation of evaluation in the same. Moreover, the descriptive inventory of the state of institutionalisation of evaluation in social systems holds great added value for the study of evaluation. Both will be addressed below. 3.2

Results

In many countries of the Asia Pacific region, evaluation does not play a prominent role in the dialogue between the state or administration and civil society, whereas the functions of external and internal accountability and, in some cases, evidence-based organisational development and management are much more pronounced. Overall, the anchoring of evaluation in the Asian civil societies is relatively low, as can be seen along almost all categories that were relevant for the analysis of the social systems (see Table 4, column 7). That’s why—in comparison to the results on the other societal systems—evaluation can best be characterised as a decision-making tool, with historical origins in the global academic discipline of “evaluation science” (see next Sect. 4), mainly and historically rooted in the public administrations of the national political systems as well as in organisational routines, regulations or requirements of donor

16

SYNTHESIS

531

and international organisations (see Sects. 2 and 5)—and from there started to diffuse into Asian Pacific civil societies. Independent receptions of the instrument of evaluation within civil societies, driven by a specific subset of CSOs are at a very early stage, however are beginning to emerge in some countries. In the future, this may lead to a more interlinked development between the societal subsystems, allowing for increased knowledge spillover and shared learning about evaluation concepts, theories and best practices. Thus, in some cases and for some indicators, there is evidence for a certain degree of institutionalisation and a dynamic of development, but this should not overshadow the overarching finding that to the present time, the anchoring of evaluation in civil societies is not very advanced in terms of the criteria examined. Moreover, the cases studied demonstrate limited variation, with most data accumulating in the lower to mid-range. Surprisingly, the specifics of civil societies in the region previously described are only slightly reflected in the results on the institutionalisation of evaluation in the social systems. Thus, while on a general level, there is something like a “convergence of Table 4

Institutionalisation of evaluation in the Social Systems of Asia Pacific Institutionalised

Public perception and discussion

use of evaluations

of evaluation

Civil societies’ demand of evaluations

Country

General Use of Eval. in CS

Knowledge

Public

about

discussion/

Evaluation

media

Availability of Reports

Civic Demand

Mean

Australia

0.5

0

0

0.5

0.5

0.30

Bangladesh

0

0

0

0.5

0

0.10

China

0

0

0

0

0

0.00

India

0

0

0.5

0.5

0.5

0.30

Japan

0

0

0

1

0

0.20

South Korea

0.5

0

0

1

0.5

0.40

Nepal

0

0

0

1

0

0.20

Pakistan

0

0

0

0.5

0

0.10

Philippines

0

0

0

0

0

0.00

Sri Lanka

0

0

0

0

0

0.00

Taiwan

0.5

0

0

1

0.5

0.40

1 = high spread, 0.5 = medium spread, 0 = low spread, n/a = not answered Source Author’s own work

532

R. STOCKMANN ET AL.

the findings” with regard to the Asian social systems, some particularities can be found with regard to civil societies’ use, publication practice of reports, and civil societies’ demand. 3.3

Institutionalised Use of Evaluation by Civil Society

The first dimension, “institutionalised use of evaluation”, examined first how citizens and civil society organisations participate in evaluations and second how they use evaluations for political or organisational purposes. This dimension is thus captured in the form of a summative indicator.15 The findings of the cross-sectional analysis suggest that initiatives aimed at broader civil society participation in evaluations are relatively rare. Where they are found, they often stem from government efforts to establish participatory governance mechanisms along the entire policy cycle—including participatory policy or programme evaluations. In some countries studied, an interest in and a—to different degrees—regularly implemented application of participatory evaluation practices is documented, like in Australia (p. 79ff.), Bangladesh (p. 108ff.), Pakistan (p. 297f.), and South Korea (p. 228f.) and Nepal (p. 256f.). However, a systematic and widely formalised involvement of citizens and CSOs cannot be concluded from this even in these cases. In addition to the participatory elements of policy evaluation, in many countries, the subset of CSOs that regularly perform evaluations, such as research think tanks, sometimes adopt participatory evaluation approaches—especially when there are regulations to do so. Moreover, findings indicate that participatory evaluation practice is substantially influenced by the particular policy field under consideration. For instance, in Bangladesh (p. 255f.), development cooperation and poverty reduction are fields of action that are distinctively characterised by participatory evaluation approaches. The overall picture, in contrast to these examples, is that in most countries, participation appears to be either rare or relatively limited in 15 Please note that here, as in sum, individual best practices of pioneering organisations, which fortunately exist in many cases, do not yet lead automatically to a different assessment, as it is a matter of taking stock in the breadth of CSOs; and here the regularity, type of application as well as occasions are decisive. This is consistent with the general methodology of the case studies (see Chapter 1), in which assessments on qualitative indicators should be made across a breadth and variety of cases, rather than focusing on a few salient cases. In this way, overgeneralisation based on a few positive examples can be limited or avoided.

16

SYNTHESIS

533

scope, especially when it comes to implementing “participatory evaluation designs” in the narrow sense. Thus, in many counties of the Asia Pacific region, there appears to be no substantial regular or institutionalised application of “participatory evaluation”, instead rather a dominance of “expert-based evaluation styles” without participatory elements is observable (Sager & Mavrot, 2021).16 Accordingly, CSOs participation in various countries is often considered only in terms of including them as respondents in surveys, expert or focus group interviews and thus tends to be a specific source of data collection rather than an involvement at various stages of an evaluation project. This finding is not surprising, given a widely documented rudimentary civil society use, low demand and media discussion and knowledge about evaluation. The following gives an overview of the individual cases in this regard: . The participatory approach in Australia (p. 79f.) is viewed comparatively very pronounced and developed, as evidenced by both extensive participatory evaluation practice in many sectors and, in some cases, sectoral institutions that mandate participatory evaluation approaches (e.g., the inclusive evaluation framework of the National Indigenous Australians Agency or calls for tenders that require participatory approaches). . In Bangladesh CSOs “do not follow any specific participation procedure for evaluation”, but do sometimes “collaborate with the Bangladesh government to conduct evaluations” (Bangladesh, p. 109). . In China, evaluation is almost exclusively a state-centred and government internal activity—participatory evaluation approaches are possible in principle and to some degree can be observed, even when their scope is very limited here (China, p. 136). . In India, “there are limited organised spaces for citizens to participate in the M&E of the government programmes” (India, p. 174). At the same time, policies like the Right to Information (RTI) and Public Interest Litigation (PIL) have a positive impact on the 16 The statement is based on the implemented country selection and on the information provided in the case studies. New Zealand, for example, would be another case, which can be considered as a global pioneer in participatory evaluation practice; but also, in valuesensitive and indigenous evaluation approaches (Cram & Chouinard, 2023; Rogers & Davidson, 2013).

534

.

.

. .

. .

.

R. STOCKMANN ET AL.

application of participatory evaluation approaches. As in many other countries, in India participation focuses on the data collection stage. Likewise in Japan, where various participatory institutions exist, for example within the Diet’s Standing Committees, “the use of participatory approaches in the evaluation and monitoring system […] remains limited” (Japan, p. 211). In South Korea, where at least a partial participatory evaluation practice is reported, the vast majority of evaluations are planned and carried out by experts. CSOs regularly participate in advisory panels for evaluations (South Korea, p. 227). In governmental evaluations in Nepal, CSOs and citizens regularly participate as “respondents in data collection and in some cases participation in evaluation reference groups” (Nepal, p. 259). In Pakistan, a variety of CSOs is “actively involved in evaluations” and CSOs and research think tanks regularly work together in the context of impact evaluations (Pakistan, p. 299). Thus, when these organisations are entrusted with evaluations, they often also rely on participatory evaluation approaches. In the Philippines, “participation is limited to information provision rather than in key steps of an evaluation including planning, designing, analysis and dissemination” (Philippines, p. 342). The case study on Sri Lanka states that overall participation is “minimal”; as far as participation is concerned, it is mostly found in the case of donor-funded project evaluations where civil society groups are involved as stakeholders, mostly in pre-evaluation consultations. However, this does not reflect the involvement of the “general public” or individual citizens (Sri Lanka, p. 375). The Taiwan case report highlights that “civil society does not play a major active role in the internal evaluations of government agencies or the evaluations performed by the CY” (Taiwan, p. 403). Even if these institutions enable the participation of citizens and CSOs in principle.

In addition to the not particularly widespread use of participatory evaluation procedures, there are also no formal participation mechanisms across countries that include evaluations as standard. Even, for example, the participatory governance instrument of “Public Inquiries” in Australia which “include[s] formal participation processes to capture views about

16

SYNTHESIS

535

the adequacy of government policies from individuals, civil society organizations, community groups, unions, academic experts, private enterprise, and various other actors” (Australia, p. 81), it cannot be said that evaluations are implemented in this context as a rule. Although these can certainly be said to involve evidence-informed political practice, they do not always include evaluation in the narrower sense. For instance, they may be followed by evidence-based investigations that also may cover evaluative evidence, however without always empirically examining the worth or merit of programmes by systematically applying evaluation criteria in the narrow sense (cf. Annex 1.1, glossary). Moreover, it is important to note here that evaluations take place in this institutional context with a certain frequency—however, not as institutionalised prerequisite. The situation in other countries with robust participatory institutions is comparable. Accordingly, even in regional pioneering countries with relatively long traditions of citizen participation, there is still potential for the use of evaluation as a standard instrument for the dialogue between the state and civil society. The results are similar across the region, where evaluation is likewise not used institutionalised as part of participatory governance procedures. Overall, with regard to participation, it can be summed up that while public policy evaluations that apply participatory methods are found in some countries and are more pronounced in some countries and sectors than others, there is no evidence that they are widely formalised in any of the national evaluation systems studied. Nor are there formal participation procedures that include evaluations as a matter of routine. No evidence was found in the case studies of Asian civil societies demanding greater participation in evaluations. With regard to the use of evaluation by civil society organisations, there are indications of a similarly weak institutionalisation—at least when the use of evaluation for political purposes is considered. The country comparison points out that there is only a few institutionalised uses of evaluation by civil society in the context of political decision-making processes. Citizens and civil society organisations do not systematically and also not frequently resort to the instrument of evaluation within political negotiation processes—even where it would be possible for them to do so within the framework of existing special steering bodies or advisory councils (such as in Australia, India, Japan, Nepal, the Philippines, South Korea, and Taiwan). Only in a few countries, at least a partial use of evaluations

536

R. STOCKMANN ET AL.

by civil society within political decision-making processes becomes visible in the case studies: . In South Korea, civil society evaluations are also used “for public debate and raise issues for government policies” (South Korea, p. 227). Moreover, CSOs are sometimes invited to advisory panels of evaluations (p. 227). . In Taiwan, in certain sectors such as human rights and healthcare, evaluation reports are used to influence decision-making processes (Taiwan, p. 400, 403f.). The case report states that “the release of these reports is tied to concrete demands targeting the government” (400). In addition, NGOs or CSOs in certain sectors are regularly entrusted with the evaluation of government policies. This allows them to control government action to a certain degree (ibid., p. 403). . In half of the country case studies, it is documented that civil society actors are entrusted with evaluations, which gives them an opportunity to hold state actors accountable (Bangladesh, p. 109; India, p. 168f.; Taiwan, p. 399f.; South Korea, p. 299; Pakistan, p. 296). These examples show that in some countries CSOs use evaluation to a certain degree for political decision-making processes, which contributes to the control function of evaluation by holding governments accountable. However, on the basis of the case studies, a largely institutionalised and regularly implemented practice on a broad civil society base cannot be derived from this. Based on these findings, most countries are rated with a “low spread” on the first indicator, which combines the criteria of “use of evaluation in civil society” and “participation of civil society in evaluation” (Rating: 0). Only Australia, South Korea, and Taiwan are rated with “medium spread”, as they have some significant and regularly implemented practices across the different subcategories that differ from the other cases, as outlined above (Rating: 0.5) (see Table 4, column 1). The situation is somewhat different with regard to the use of evaluations for purely organisational purposes. For all countries, at least minor evaluation-related activities by some civil society organisations are reported. At the same time, the findings suggest that despite some evaluation practices in the region’s CSOs, evaluation is still experiencing

16

SYNTHESIS

537

marginal or limited importance when considered across a broad variety of organisations. The latter is reflected in a weak institutionalised use of evaluation even for inherently organisational reasons. For example, it is indicated that other practices such as “annual reviews” (Nepal, p. 258) and “assessments or performance audits” (the Philippines, p. 340) outweigh evaluation practices. Based on the examples of evaluation practices in CSOs reported in the Taiwan case study, it can be seen that these are also often other management tools such as accreditation (Taiwan, p. 394) and do not involve evaluation in the narrower sense. Even in South Korea where civil society actors are relatively strong and broadbased and make at least sometimes use of evaluations in the context of political negotiation processes, “their role as producers of evaluations is limited” and evaluations “are the exception rather than the rule” (South Korea, p. 227). While in China, where evaluations are mostly seen as “government internal activities” (China, p. 134), to a limited extent and scope emerging “third-party evaluations” are important to mention, they are facing huge challenges in their organisational anchoring mainly because of their documented lack of independence. Thus, the degree to which they can be regarded as independently anchored in the organisational structures tends to be relatively low. In countries where CSOs use evaluations at least partially, they are often implemented for legitimacy, external accountability, and reporting reasons. Either requirements mandated by donor organisations are met through evaluation (Japan, p. 208f., Philippines, p. 340f., Nepal, pp. 254ff., Sri Lanka, p. 372, India, p. 169, Pakistan, p. 295), or the desire to publicly legitimise organisational activities “in order to garner support within a competitive funding landscape” and to “fostering donor and public confidence in the organisation” (Australia, p. 79) is in the foreground. This contributes to a situation in which evaluation in many CSOs runs the risk of becoming a “tokenistic activity”, as noted in one case study (India, p. 169). The Pakistan case study underlines that evaluation is mainly being defined as a policy instrument and not as an organisational management tool (Pakistan, p. 308).17 This can be seen as a generally important explanatory factor for the overall intra-regional finding that evaluation is rather anchored in the political structures than in civil society organisational contexts. Based on the findings of numerous 17 At this point, it must be said that there is currently insignificant evidence for an exact assessment, as the authors of the case study emphasise.

538

R. STOCKMANN ET AL.

case studies, evaluation has not yet fully established itself as an independent, evidence-based tool for organisational management within many CSOs, both in terms of its application frequency and the range of types utilised. Nevertheless, it should not be ignored that in the subset of CSOs (and also companies) where evaluation is successfully applied, the great steering benefits of this instrument are appreciated. Accordingly, in such cases, evaluation is perceived as a useful management tool for evidence-informed organisational development, strategic decision-making and programme improvement (e.g., Nepal, p. 254f., Taiwan, p. 404, Bangladesh, p. 109, India, p. 170, South Korea, p. 227f., Sri Lanka, p. 370f.). Elsewhere, it “is yet to be seen” whether national evaluation regulations will lead CSOs to “eventually [use evaluations] for decision making” (Pakistan, p. 297). However, as previously mentioned, the finding must be put in context with the fact that across the region and considered in the breadth of organisations, many CSOs still do not regularly or independently use evaluations. According to the case studies, in Asia Pacific civil societies, evaluation is more likely to be implemented and anchored in the “large” and “well-resourced” CSOs, as, for example, in some NGOs and local sections of INGOs, as the case reports on Japan (p. 208), Australia (p. 79), Bangladesh (p. 108), Nepal (p. 254f.), and Sri Lanka (p. 370, 372) underscore. Organisational awareness of the benefits of evaluation is an important prerequisite for its use. Yet many case reports point to existing challenges that contribute to low use in CSOs. These include various resource constraints like expertise constraints18 and low organisational and technical evaluation capacity (Australia, p. 79, India, p. 169, Sri Lanka, p. 371f., Japan, p. 208f., Pakistan, p. 296f.). Some country reports (e.g., Sri Lanka, p. 370f.) also mention a lack of evaluation policies that could promote evaluation practices and structures in CSOs. Overall, the observable examples of use of evaluation for organisational purposes do not change the overall assessment of the summative indicator, as—first and foremost—they do not apply to a broad variety of CSOs and

18 Please note that in organisational studies the term “resources”, used in the case studies contains various dimensions. It is a summary of aspects like skilled labour and expertise, infrastructure, facilities and equipment (technical resources), funding (economic resources), network contacts (social network resources), and technology (technological resources) among others.

16

SYNTHESIS

539

previously mentioned aspects were assigned a stronger weighting. If low spread was reported for all or the most other criteria of the first category, an identifiable (institutionalised) use for organisational purposes did not “turn the tide” and accordingly the assessment of “low spread” persisted (Rating: 0). The case study assessments need to be critically contextualised for the overall regional picture. While a correlation between organisational size and regular evaluation practice may be observable in some countries, a generalisation based on the rather qualitative findings and the qualitative-comparative assessment of important factors is not indicated, since conceptually and empirically many reasons speak for a more complex empirical interrelationship, as can be illustrated by the studies in the fields of “organizational learning” (Argote, 2012; Argyris & Schön, 1996) and “organizational sociology” (Pohlmann, 2023). For example, in their organisational sociology-based study on development cooperation in Germany, Borrmann and Stockmann (2009) showed that regularly used organisational evaluation practices and well-designed organisational evaluation systems are not primarily related to organisational size or resource endowment. Instead, other factors play a role for evaluation use in organisations, which is also reflected in the explanations of the case studies. The first thing to look at is the network contact of CSOs. It is often the case that embeddedness of the organisation in a network of international organisations and donor agencies contributes significantly to the organisational evaluation culture. In particular, regular collaboration with IOs or donors plays a crucial role in such organisational learning processes. On the one side, they get to know the instrument because of external donor requirements on project and programme evaluation, on the other hand they get the opportunity to participate in the global discourse on accountability and learning which makes active learning more likely, resulting in organisational awareness for the concept of evaluation. This can be summarised as “interorganizational learning” (e.g., Holmqvist, 2003) or “network learning” (Knight, 2002). Another factor, derived conceptually, is the general organisational learning culture. In “learning organizations” (Easterby-Smith et al., 1999) there is a much higher probability to develop an understanding of the concept of evaluation and its various purposes and then finally to apply it. In addition, the sector of the respective organisation is another crucial explaining factor. Discourses on accountability and learning have sector-specific emphases, which are reflected in the learning opportunities for CSOs in particular sectors

540

R. STOCKMANN ET AL.

and thus in the organisational evaluation knowledge that is ultimately encountered. Such a network embedding allows in relation to organisational behaviour in specific organisational settings learning processes on individual, team, and organisational level—moments or opportunities for reflection may arise at the various organisational levels (Moon, 1999). This expresses the idea of pragmatic learning based on experiences in reflective moments or irritating action situations that lead to practical problem solving, which John Dewey (1938, 1980, 1988) set forth as a social science theory of education and knowledge in its highest intellectual flowering, at the organisational level (Elkjaer, 2021; Gillberg & Vo, 2014). In summary, it is therefore important, especially for the crossregional view, which is accompanied by a certain generalisation, not to focus on individual factors internal to the organisation, such as organisational size, or environmental factors, such as evaluation regulations, as explanatory factors, since these are not sufficient conditions for evaluation use in organisations. Instead, the interplay of various factors in a more complex model of relationships must be brought into focus. As outlined aspects of social network embeddedness play a crucial empirical and conceptual role in organisational learning (cf. Grundmann, 2001).19 The overall picture is one of rather weak institutionalised use of evaluation by most Asian Pacific civil societies. It is also evident that where evaluation is practised by civil society actors at some level, it is mostly not very systematic or not formally anchored and thus might be volatile. However, there are some exceptions, such as in Australia, South Korea, and Taiwan where evaluation regularly serves a dialogue and control function, as well as many promising examples across the region where participatory evaluation is frequently applied or where evaluation is often used at least for organisational purposes in several CSOs, whereby participatory approaches to evaluation can also be found in some CSOs.

19 For a more accurate assessment of the influence of various theoretically plausible

and empirically cited factors on evaluation use in CSOs, further research is needed using other research designs such as large-N organisational studies that allow correlative and statistically robust conclusions to be drawn about the significance of explanatory factors with the inclusion of control variables, as well as QCA studies that reveal the precise interplay of theoretically relevant factors.

16

3.4

SYNTHESIS

541

Public Perception and Discussion of Evaluation

In terms of public perception and discussion of evaluation, three subquestions were of interest: knowledge about evaluation in civil society, media reception of evaluation, and transparency of evaluation reports. These aspects were not included in the results in the form of a single summative indicator, as was the case with the first dimension, but were each covered by their own specific indicator expressing a distinct theoretical subcategory. The results for each subcategory, represented by dedicated indicators, are presented in sequence, one after the other. Relatively fragmented public perception and weak knowledge about evaluation in civil society emerged as a central commonality among cases in the cross-sectional comparison (Table 4, column 3). Generally speaking, the case studies do not report widespread and developed general knowledge about evaluation in civil society. While there is a significant awareness of evidence-based policy and programme design—a socially shared sense of the value of evidence—in many places, this does not translate to evaluation-specific knowledge, as the majority of case studies indicate that civil society is not familiar with the concept in the strict sense. As a result, citizens and CSOs do not yet widely grasp the concept of evaluation and its added value for society. Instead, across the region the concept is known predominantly in a few special professional circles; for example, in India within the professional community of development evaluation like in the Evaluation Community of India (ECOI) (India, p. 169) or in Japan in the field of international cooperation (Japan, p. 210). Beyond these specialised communities of practitioners and academics, evaluation does not meet with widespread social recognition and has not yet established itself as a separate and autonomous profession in the eyes of civil society. For example, this is expressed in the Australian case study, in the fact that “evaluators are frequently confused with auditors, social researchers, and management consultants” (Australia, p. 79). That accurately sums up the situation in many other countries. In some sectors, such as higher education, evaluations are reported to be somewhat better known than in others (e.g., China, p. 136, Taiwan, p. 404). Overall, these findings lead to the assessment of “low spread” across all countries (Rating: 0). The existing knowledge resources with a sectoral reference or in special social circles documented in some country case studies do not change this assessment, as it does not reflect a broader

542

R. STOCKMANN ET AL.

civil society’s awareness of the concept and the benefits of evaluation in the narrower sense. This overall weak public perception and awareness implies that evaluation findings are generally not used very much by civil society actors to initiate public policy debates, even when evaluation results are published (e.g., Sri Lanka, p. 374, Nepal, p. 257, Japan, p. 209f.). Overwhelmingly, civil society groups do not recognise evaluation as a means for social advancement and political empowerment—again considered in terms of the range of CSOs and in terms of the regularity of application.20 To some degree the low public awareness of evaluation in civil society is likely to be related to a relatively low level of media attention. In some countries, evaluation results are reported in the media at certain intervals and often with some sectoral differences. In most of the cases studied, the media coverage of evaluation is rather unbalanced or selective21 and also not systematic and rather not regular. In-depth discussions and detailed debates on evaluation are not documented. Some country reports underscore that evaluations are mainly referred to when they concern “controversial projects” (Australia, p. 79; South Korea, p. 228; the Philippines, p. 342). Other countries show a generally low media reception of evaluations (Nepal, p. 260, Sri Lanka, p. 374) or low but quantitatively increasing one (Pakistan, p. 299). Reasons for this general situation are seen for example in “fast-paced news”, the “commodification of news” (Taiwan, p. 403), and the major challenges of a “post-truth era” in which the search for truth seems less weighty than populist slogans and personal opinions (Australia, p. 80). This may contribute to public

20 As there are different methodological traditions with specific evaluation designs in the field of evaluation (Meyer & Zierke, 2022; Stockmann, 2024), it is worth noting that some methodologies and applied evaluation theories (often purposefully) imply a level of political agency (see Vestman and Conner (2006), who generally discuss the relationship between evaluation and politics and the role of value propositions in evaluations; moreover, see Stockmann (2024) for the conceptual distinction between agreed evaluation criteria and unequivocal valuations). Such types of evaluations that follow a political goal affect the policy processes purposefully “from the inside” of an evaluation; moreover, evaluations and their results can be used “from the outside” by civil society groups in order to be useful for their political goals. The latter is the focus of the consideration presented here. 21 This becomes evident when the case reports show that evaluations tend to be more received in the media within certain topics, such as quality of life (China, p. 136), environmental protection (China, p. 136, Philippines, p. 342, Japan, p. 210), education and food safety (Taiwan, p. 402).

16

SYNTHESIS

543

discourses in which evidence-supported rational analyses of the effectiveness and impact of public policies and programmes—the “details of policy” (Schwandt, 2019, p. 320)—are in decline (Australia, p. 80). In another case, they are not yet developed (Pakistan, p. 298f.). Furthermore, low public awareness combined with low mass media interest in evaluation contributes to the situation (cf. Sri Lanka, p. 374f.) as well as specific patterns of agenda setting in which the evidence itself is given secondary relevance (ibid., the Philippines, p. 342). Lastly, in many cases, “evaluations” are reported and discussed in media in a very broad sense, so that in fact monitoring, accreditation, audits or assessments are often meant and there is no specific notion of evaluation behind it (e.g., Bangladesh, p. 108f., Taiwan, p. 402, China, p. 136, Japan, p. 210). These findings lead to the fact that 10 of the 11 countries examined are assessed with “low spread” for the indicator “media reception” (Rating: 0). In other words, across the region the role of media to support or initiate public debates on evaluation and its results is not substantially anchored and thus rather weak (see Table 4, column 4). Nevertheless, it must be said that many case studies provide examples that point to data-driven media coverage, i.e., an evidence orientation, which has the potential to bring evaluations more into focus. In slight contrast to the other cases, in the inventory on India, media is described as an important “source of demand” for evaluations (India, p. 176) and there is also regular—but as in the other cases not systematic—reporting on evaluations with varying focal points (e.g., Sustainable Development Goals, labour market, evaluation policy, social policy) in some of the larger daily newspapers accordingly (ibid.). However, also here not always “evaluation in the narrower sense” is covered. Besides that, India is the only country in the sample in which a media initiative to strengthen the public perception of evaluation is reported (the media platform India Development Review, India, p. 172). Because this can be seen as an institution as well as the regular media reporting on evaluation-related topics, India is assessed as the only country with a “medium spread” of this indicator (Rating: 0.5). But with regard to this subcategory, too, it is striking that it is a policy field-specific question to what extent media discusses evaluation results; it usually correlates with the sectors in which evaluation is comparatively strongly anchored and practised. In the example of India, the development and international cooperation sector proves to be particularly relevant.

544

R. STOCKMANN ET AL.

This is a notable finding, since in some of the countries there is in principle a culture of transparency on the part of the government, as will be shown in the following. This may indicate that media perception of evaluation widely corresponds with the results on use, demand and knowledge of evaluation within civil society. Remarkably, in many cases, the publication practice of evaluation reports does not contribute to the two previously discussed findings on media reception and civil societies’ knowledge about evaluation (Table 4, column 5).22 Worldwide, the idea of transparent governments and open access to information is becoming increasingly important.23 FOI laws are a typical example of the “open government” concept. Governments around the world have adopted Freedom of Information (FOI) laws, which give citizens—to varying degrees—the right to receive information about certain government operations. It can be seen as a “legal backbone for creating and safeguarding a basic level transparency” (Grimmelikhuijsen et al., 2018, p. 2). Viewed in an international comparative perspective, the “right to know” seems to be a legislative success story. All 11 Asia Pacific countries examined here have now passed such a law. Empirically, individual states vary in the precise design of their respective open government approaches. At one time, it goes hand in hand with truly open data legislation and far-reaching digitisation (this relates to the broader policy context of FOI laws); at another, it merely encompasses the right to receive information about certain processes upon request (this covers the content of FOI laws in the narrow sense). Not surprisingly, the handling of evaluation reports, as an example of the documentation of government action, differs between the individual nation states studied. Comparable institutions have not emerged from civil society, but some CSOs do publish their evaluation reports regularly (Bangladesh, p. 108, Taiwan, p. 400f., Japan, p. 210); elsewhere, internal use of CSOs’ evaluation findings prevails (Nepal, p. 257, the Philippines, p. 341). The “third-party” evaluations that have been emerging in China are more likely to publish their results and methodology than not, for example,

22 Nevertheless, it cannot be considered supportive if there is extensive non-transparency (cf. Sri Lanka, p. 372f., China, p. 136). 23 For example, in the context of the 16th Sustainable Development Goal, the United Nations is advocating for public access to information to ensure accountable institutions (UN, 2015, SDG 16, Target 10).

16

SYNTHESIS

545

university evaluation reports in the higher education sector (China, p. 138). With regard to the publication of evaluation reports three groups of countries can be differentiated: a first pursues the practice of transparency in evaluation reporting (Japan, South Korea, Nepal, Taiwan), a second group publishes its evaluation reports on a partial basis (Australia, Bangladesh, India, Pakistan) and in a third the practices of public transparency in the field of evaluation are weak or not developed at all (Philippines, Sri Lanka, China). The first group was assessed as having a “high spread” of transparent publication practice (Rating: 1), the second group was assessed as having a “medium spread” (Rating: 0.5) and the third group was assessed as having a “low spread” of public transparency of publication reports (Rating: 0). Among the first group, there are some countries like Taiwan where the transparency culture is based on a legal framework, the “Freedom of Government Information Law” from 2005, which intends to “facilitate people to share and fairly utilise government information, protect people’s right to know, further people’s understanding, trust and overseeing of public affairs, and encourage public participation in democracy” (Taiwan, p. 401). This law thereby also includes evaluations. In other countries in the first group, evaluation reports are usually published, but the case study authors do not point out the particular relevance of the FOI or open government legislation to this practice (South Korea, Nepal, Taiwan). For instance, in South Korea, “almost all the evaluation results are open to the public, although there is a possibility that the official version of evaluation results might be watered down” (South Korea, p. 228). The second group of countries has FOI or transparency laws (Australia, Bangladesh, India, Pakistan), which is also highlighted by the authors in one case (Pakistan), but this does not imply the publication of all evaluation results. As in Pakistan, for instance, where the “Right to Access to Information Act” from 2017 covers both audit and evaluation reports (Pakistan, p. 297), although the practice differs somewhat from this: there are only “several national programmes which have provided public access to their evaluation reports” (ibid.) and not all provincial-level evaluations are published; however, this does not mean that reports are not made available in response to public demand as the report highlights (Pakistan, p. 298). In India, similarly many evaluation reports are published, although there are also some intra-regional differences (India, p. 173). The last group of countries (Philippines, p. 342,

546

R. STOCKMANN ET AL.

Sri Lanka, p. 375, China, p. 134) indicates that despite existing FOI laws, evaluation reports may rarely or never be published. For example, in China, “there is also an increasing trend for civil society to request government information”, while “still a long way to go for the open government information movement”, clearly evidenced by the fact that “internal performance appraisal and management results are kept within the government and considered confidential” (China, p. 134). Overall, in view of these findings, no clear link can be identified between transparency practices and the use and institutionalisation of evaluation in the political system. When evaluation is not substantially anchored and only sporadically practised, as described in Sect. 2 on the institutionalisation in the political systems, however, it is not surprising that there are cases where evaluation results are not published and are not subject to the national transparency claims. 3.5

Civil Societies’ Demand for Evaluations

The result described above on public perception and media discussion of evaluation probably strongly influences the results on the last dimension of the social system: the civil societies’ institutionalised demand for evaluation. This examined whether citizens or CSOs request evaluations, how regularly and in how far institutionalised this happens, and whether there are constraints in doing so. For this, the Evaluation Globe analytical guideline provides for an assessment across the breadth of CSOs rather than focusing on selected pioneers and best practices of the field. The results show that the Asian Pacific civil societies predominantly do not demand evaluations. If they express demand, it is not regular, not systematic and not based on any institution. This is true for individual citizens as well as the media and civil society organisations (Table 4, column 6). Generally, with few exceptions, there is no significant pressure from citizens or civil society organisations to encourage the state to conduct more evaluations. Only in four countries institutionalised opportunities to address evaluation-related public requests to governments are mentioned. The first three of these are the “Public Inquiries” in Australia (p. 81), the “Citizens Proposal” on the online portal of the President’s Office in South Korea (p. 230) and the “Research, Development and Evaluation Commissions” on local level in Taiwan (p. 392). It is important to note that these procedures do not necessarily concern evaluation in the strict

16

SYNTHESIS

547

sense24 and are consequently not limited to this form of evidence; but unlike, for example, the Public Interest Audit Request System in South Korea (p. 229), they do not only involve audit information but may indeed be followed by evaluations or often entail evaluative components. Thus, for evaluation, on the one hand, it is an institutional possibility to express civic demand for evaluation; at the same time, in practice, there is rather a general use in the sense of demand for evidence, which, however, regularly covers evaluative aspects, e.g.,—as an ideal type—the demand for assessing the effectiveness, sustainability or efficiency of a programme or if citizens ask to investigate the results of public policies. This is one of the reasons for the rating given to these three cases (“medium spread”) (Rating: 0.5). Besides these three cases, the “Citizen Portal” of the Prime Minister’s Performance Delivery Unit (PMDU) in Pakistan is a potential institutionalised way for individual citizens and CSOs to request evaluations or evaluative evidence, although in the case study, no assessment is being made of the actual content and extent of use for this purpose (p. 301), which is why this example does not affect the rating. Many of the case reports identify low awareness for and understanding of evaluation as a reason for low civic demand (Sri Lanka, p. 376, Pakistan, p. 298, 302, Nepal, p. 259). The aspect of low organisational evaluation capacity related to limited international network embeddedness expressed in rare cooperation with international organisations, minor participation in global discourses on learning and accountability, resource constraints like limited human resources (e.g., MEAL expertise) or the general organisational learning culture in many Asian CSOs discussed earlier expresses this at the organisational level. Due to that, the organisational setup for identifying and processing evaluation-related information needs is lacking. This also becomes visible when, as in the case of Japan, the Japanese Non-Profit Network promotes “Seven Necessary Conditions for Reliable Non-Profit Organizations”, taking performance monitoring into account but disregarding evaluation (Japan, p. 211). As a result, the independent articulation of demand for evaluation is very unlikely. Similarly, there are cases where a need for information that could be 24 For this, please refer to the definition of evaluation contained in the glossary in the Appendix 1.1 of this volume: “Evaluation is the systematic investigation of the merit or worth of an object (program) for the purpose of reducing uncertainty in decision making” (Mertens, 1998).

548

R. STOCKMANN ET AL.

met through evaluation is documented, like in the context of the desired improvement of development projects (Philippines, p. 342f.) or in the corporate sector (Bangladesh, p. 110f.), but this is not translated into an actual demand for evaluation in the narrower sense. In consequence, this means that if the understanding of evaluation were better in different fields of practice, such as at the project level or in the corporate context, there probably would be a greater awareness that different information needs and purposes can be met through the tool of evaluation, which would consequently result in an increased demand for evaluation in the narrower sense. But as has already been shown, in many areas of the civil society sector there is a lack of awareness and knowledge about evaluation, which inevitably affects demand. This concerns almost all country case studies, which is why 8 out of 11 nation states are assessed with a “low spread” of civil society’s demand for evaluation (Rating: 0). Low demand for evaluation may also be attributed to a potential lack of trust in the government or evaluation results, which may exist among certain segments of the population (Australia, p. 82, South Korea, p. 228). Furthermore, national evaluation policies may often not substantially strengthen civic demand for evaluation (Nepal, p. 259, Sri Lanka, p. 372), as the Australian case study puts it, often there is a “limited effort [on part of the state] to stimulate public interest in evaluation” (Australia, p. 82). In China, the specific approach of “performance-based legitimacy”—resulting in high degrees of political trust in government— combined with a “traditional culture of rule by ‘secrecy’” leads to the fact that the population does not demand evaluations “as long as the Chinese government can continuously improve their wellbeing” (China, p. 136). In the latter case, the political institutional framework that largely restricts evaluation to a governmental internal subject hinders the effective facilitation of civic demand for evaluation. Thus, as the last four examples show, in addition to internal constraints that can be traced to civil societies themselves, strengthening civil societies’ demand also faces challenges of varying degrees in the political environment. Consequently, the external drivers for the institutionalisation of evaluation in the social systems are also subject to limitations. At the same time, it is clear from a large proportion of the case studies that when evaluation is called for, external factors are regularly the main drivers of demand. In particular, five countries emphasise donor requirements (Sri Lanka, p. 376, 19, Bangladesh, p. 296, India, p. 169, Nepal, p. 258, Philippines, p. 340f.) and three studies underline

16

SYNTHESIS

549

specific reporting requirements of CSOs and NGOs to the government as central drivers of demand (Nepal, p. 254f., Pakistan, p. 295, Philippines, p. 341, India, p. 168f.). It is important to note, that there are also shifts over time. For example, India has increasingly moved from donor-based evaluations to such that are country-led and internalised in the planning stage in recent years (India, p. 175f.). Consequently, as these cases document, external driving factors play a strong role in civil societies’ demand for evaluation. Overall, the case reports consider civic demand for evaluations relatively low and usually driven by specific professional groups—like “independent research organisations” (India, p. 164), “a selected group of professionals/intellectuals” (Pakistan, p. 298), “the fora of academics and professionals”, namely, “several groups such as professional organisations, universities, private sector actors, non-profit organisation, and some international organisations” (Sri Lanka, p. 376), and civil society organisations like public philanthropies (Australia, p. 87). In Pakistan, research-oriented actors of the third sector like research think tanks “have played a pivotal role in underscoring the importance of impact evaluations and evidence driven policy making” (Pakistan, p. 303). Thereby, they “have paved the way to scale up the [evaluation] work and also created demand for M&E training at all levels of federal and provincial ministries” (303f.). Moreover, in India, the media is referred to be a “constant source of demand” (India, p. 176), because in some cases it requested evaluations. When there is a demand for evaluation from civil society organisations, it usually refers to such non-profit or non-governmental organisations that regularly work together with international and donor organisations, participate in global and local discourses on “accountability and learning” and thus are more likely to be aware of the various benefits of evaluations as is the case with BRAC in Bangladesh (p. 108f.) and CERP in Pakistan (p. 288f.). In other countries, individual social groups associated with political activism and policy advocacy frequently demand evaluation for political purposes as two case reports underline (India, p. 175, 177, Philippines, p. 342f.). In Australia, also individual citizens ask for evaluations in the narrower sense, although this is rather rare (p. 81), moreover, demanding forces can be seen in such large CSOs and NGOs that follow an intrinsic motivation in evaluation practice such as fostering confidence (p. 79) or who stimulate the evaluation market through the public tendering of external evaluations. Moreover, in the previously mentioned instrument

550

R. STOCKMANN ET AL.

of the Public Inquiries one frequently can find the use of evaluations or topics like addressing the quality of evaluation, which amounts to a metaevaluation—thereby diversified stakeholder groups come into contact with evaluation in an institutionalised framework (p. 81f.). In Taiwan, there is a strong commitment to evaluation within several sectors, such as human rights, the legal system, and parliament. A general public demand can also be observed, which is related to the civil society culture and is expressed in the fact that “citizens wish to hold state actors accountable” (Taiwan, p. 405). Moreover, several participatory political institutions exist for addressing such demands. Finally, it can also be said for South Korea that the existing participatory institutions with which civic demands can be addressed to the government “often include evaluation components” (South Korea, p. 229). For the last three countries, it is more likely that a general demand for evidence is expressed, which then regularly extends to evaluations or essential aspects of evaluative practices. An ideal example of this is when citizens or civil society actors ask for evidence of the effectiveness or efficiency of public funded programmes. Due to this situation, the three cases have been assessed with a “medium spread”, as there are institutions for processing civic demand for evaluative evidence and they are also at least partially and in at least some respects practically used for the demand of evaluations (Rating: 0.5). The situation is somewhat different in India. Here, the case study does not highlight any particular political institution by which citizens frequently demand evaluations. However, a demand that can be identified across a relatively large number of civil society actors (media, CSOs, political advocacy, independent research organisations) is presented. This leads to the assessment of an existing “medium spread” of this indicator (Rating: 0.5). In sum, Australia, India, South Korea and Taiwan show at least partial regular demand for evaluation among individual citizens or several types of civil society actors, which is reflected in the rating: thus, based on the aforementioned examples, which indicate institutionalised options that are used for demanding evaluations or a comparatively broader civil society base, these four can be assessed as having a “medium spread” of civil society demand (Rating: 0.5) (see Table 4, column 6). Finally, it is important to note that specifics of certain policy fields shape the civic demand for evaluation: In some fields of action civil society is usually more involved and therefore more likely to call for the instrument of evaluation (e.g., the development sector in Bangladesh (p. 105f.)

16

SYNTHESIS

551

and India (p. 176f.), or the areas of justice, parliament and human rights in Taiwan (p. 405). This, however, can lead to a “fragmentation of demands” (Taiwan, p. 405f.), which results in the absence of an overarching instance that calls for evaluation across sectors. Although this task would be well served by VOPEs, they only take a significant role in articulating the demand for evaluation in some countries (see next section).

4

Summary

The institutionalisation of evaluation in civil societies across the Asia Pacific region can be characterised by the following key findings: . First, it should be noted in general that compared to the political system and the systems of professions the institution of evaluation is considerably less shaped by the activities of the various civil societies in the Asia Pacific region. Although there are certainly some promising best practice examples in terms of evaluation practices. . The Asia Pacific countries studied vary considerably in terms of strength and tradition of their civil societies. However, this is only slightly reflected in corresponding differences in anchoring evaluation in their social systems. Australia, Taiwan, and South Korea, three of the highest-rated countries (see Table 4, column 7), are among the top performing in the Asia Pacific region in terms of good governance (World Bank, 2021). . Only in Australia, South Korea and Taiwan at least a partial use of evaluation in civil society can be observed—expressed either in a somewhat more visible use of participatory evaluation approaches or in a regular use of evaluations by a variety of civil society actors in the context of political negotiation processes and to influence policy outcomes. Furthermore, for Australia, India, South Korea, and Taiwan it is possible to conclude that evaluations are more frequently and rather institutionalised demanded by individual citizens or several civil society actors. . However, across different categories, it becomes evident that the emergence and historical development of institutions of evaluation is to a large extent policy-driven, as also expressed, for example, by the great importance of government regulations and donor requirements for the use of evaluation in CSOs. Even in the countries

552

R. STOCKMANN ET AL.

with more advanced or differentiated civil societies, there is little institutionalisation stimulated by them. . An exception to this trend is the institutionalisation of evaluation in a specific subset of CSOs (like think tanks, research organisations or charities with awareness of evaluations, various benefits due to (i) frequent cooperation with international and donor organisations, (ii) organisational orientation to learning, (iii) embeddedness in local and global discourses on accountability and learning and (iv) organisational focus on sectors with strong evaluation culture) and multi-nationally operating INGOs (International NonGovernmental Organisations). In the first case (local and national CSOs), external accountability measures by donors or state authorities are especially important, but in some countries, a number of CSOs also autonomously begin to see the added value of evaluation for various other organisational purposes. In the latter case (INGOs), developments are additionally often characterised by transnationalorganisational dynamics, which require that a regular evaluation practice is also established locally—i.e., in the national or local branch of an INGO. . In terms of the transparent handling of evaluation reports, there is a variation between the Asia Pacific countries studied. In Japan, South Korea, Nepal and Taiwan, the reports are highly transparent. In Pakistan and Taiwan, Open Government and Freedom of Information decrees explicitly include evaluations; although in Pakistan not all evaluation reports are published, they would likely be released in response to public requests. Comparable institutions have not emerged from the third sector itself. . The low level of public discussion of evaluation findings in particular and the low public impact of evaluation, in general, contribute to the weak institutionalised use of evaluation in the social system. In the countries of the Asia Pacific region, hardly any balanced media coverage on evaluation is reported. It is therefore not surprising that the general public knowledge about evaluation is repeatedly reported to be low. In addition, there is a limited level of awareness of and interest in evaluation among most civil society actors, as well as many CSOs, especially those with specific organisational characteristics like low embedding in social networks in which there is a discourse on learning and accountability, weak contact to international organisations and donors, as well as associated low organisational awareness

16

SYNTHESIS

553

for the benefits of evaluation for this purposes which leads to low interest and sustainable investment in evaluation capacity; in some cases, this is also reflected in the fact that evaluation is often seen as an instrument that is implemented due to external requirements or for public communication, and often not as an instrument for evidence-based decision-making.25 The overall result is that for Asian Pacific civil societies there is still a journey to be taken in fulfilling the various systemic functions of evaluation; while in countries where evaluation is being used by civil society the internal control or external accountability function of evaluation is particularly emphasised, the insight, the political control, and dialogue function are less central, although as described above in some countries examples point to regular practices that contribute to these purposes.

5

The Institutionalisation of Evaluation in the System of Professions 5.1

Introduction

The state of professionalisation of evaluation in Asia is very heterogenous and it lacks behind the global development in several regions. From theories of the profession, four main dimensions are derived (Abbott, 1988; Meyer, 2016; Meyer et al., 2020, 2022) and analysed in detail from the authors of the country chapters. For a profession, the existence of some kind of training courses for teaching the essentials of the profession is key. While evaluation is an academic endeavour, including scientific instruments from various social sciences, inclusion into the national academic system of universities, academies, research institutes, training courses, and programmes is an important step for becoming a generally accepted profession at least in the country level. Certainly, education of evaluation is not limited to the academic system, and one must take other offers from private or state organisations into account. The transfer of evaluation skills and knowledge to beginners is just one part of communication on evaluation. In the academic system, the

25 Thus, the organisational strategic benefits of “measuring social change” (Ebrahim, 2019) are not being realised in a variety of CSOs in many countries.

554

R. STOCKMANN ET AL.

exchange of research and experiences is primarily organised via scientific journals and the existence of such a journal is key for academic disciplines and their establishment as an academic profession. However, journals are nowadays not the only opportunity for discussing research results and sharing experiences about new approaches and research methods. Particularly new media like blogs, e-mail lists, video channels, newsletters and more are used and replace traditional printed media like for instance preprints from research institutions. Most of such communication media are edited and published by scientific organisations that are formed to support the development of the profession. In the evaluation community, there is a global effort to help evaluators build up such kind of organisations of evaluation, summed up as voluntary organisations for professional evaluations (VOPEs) and registered at the International Organisation for Cooperation in Evaluation (IOCE), the global umbrella organisation. For sure, not all networks and activists for organising evaluation are catched-up here, but because no fees for registration are charged and some services are offered, this option is widely used at least as a clear statement of willingness to be part of the global community of evaluators. Hence, being registered at IOCE as a VOPE does not say much about the form and role of the organisation within the country. The range of VOPEs lasts from huge and powerful formalised associations like the American Evaluation Associations to very small and open networks composed by a small number of members. The final aspect of professionalisation investigated here are norms of evaluation. Building up a profession is also a closure process by defining what belongs to the profession and what not. Such definitions provide rules, guidelines, and ethical principles which are more or less obligatory for becoming a member of the evaluation community and be respected as an evaluator. While in some countries registration systems exist, the need to follow such restrictions is much higher than in the vast majority of countries which barely have no such norms or only voluntary systems of standards or codes of conduct offered by VOPEs or research institutes. These four dimensions of a profession are used as a Guttman Scale for defining the degree of being a profession in a comparative way (see Table 7 for the scale at the end of this part). All four components are equally weighted and can be both used for comparing the countries as well as the four dimensions in a regional perspective.

16

5.2

SYNTHESIS

555

Results

As an overarching result, one must state that only one country has established evaluation as a profession in a way that is comparable to the leading countries in North America and Western Europe. This country is Australia, which belongs to the forerunner countries in the global development of evaluation. One very good example of this statement is the role evaluation played at Australian Universities, particularly at the University of Melbourne that offers graduate courses on evaluation since the early 1980s and established a Master of Evaluation in 2011 (Australia, p. 83). There is no other country in Asia that has a comparable long and ongoing history of academic courses on evaluation to Australia. In Sri Lanka, there are several offers of evaluation courses at universities, including a Master of Evaluation that is not continued and a new Master programme which is not yet established (Sri Lanka, p. 377f.). Although these efforts had been strongly supported by UNICEF, they cannot yet be seen as being successful. However, there is a post-graduate diploma in Monitoring and Evaluation offered by the University of Sri Jayewardenepura, providing a university degree in Monitoring and Evaluation. No other country provides university programmes specialised in evaluation at the graduate level. However, one can find evaluation as a topic on a broad variety of other study programmes in several countries. Most advanced seems to be Nepal where several universities offer short-term courses (2 or 3 credits) and included them in different disciplines like Agriculture or Education (Nepal, p. 260f). In South Korea and Taiwan, some elective Monitoring and Evaluation courses can be found in a few other study programmes like Public Administration or Development Cooperation on a regular basis (South Korea, p. 230f., Taiwan, p. 406f.). While these courses are part of established study programmes and regularly offered, in almost all other countries university courses are exceptions and provided sporadically (Bangladesh, China, India, Japan, Pakistan). In the Philippines, only some traces of evaluation can be found in courses on other topics like auditing, economics, environmental sciences, public administration, and public management (Philippines, p. 343f.). One interesting detail is the role research institutes play not only in the field of doing evaluations but also in training (and further communication activities). Several countries reported of such training offers in climate change (Bangladesh, e.g., International Centre for Climate Change and

556

R. STOCKMANN ET AL.

Development, p. 112), economics (Pakistan, e.g., The Pakistan Institute for Development Economics, p. 302), planning (Nepal, e.g., The National Planning Commission, p. 262), public health (India, e.g., Tata Institute of Social Sciences, p. 108), and public policy (Australia, e.g., Australian Consortium for Social and Political Research Inc., p. 84). Some of these organisations are state-owned and run by ministries, which sometimes also by themselves offer courses for their own staff or even for a broader public. Nevertheless, these are not regular offerings as also the courses are organised by VOPEs like the Japan Evaluation Society (JES, Japan, p. 212), the Community of Evaluators and the Nepal Evaluation Society (NES, Nepal, p. 262), the Sri Lanka Evaluation Association (SLEvA, Sri Lanka, p. 379), the Taiwan Assessment and Evaluation Association (TWAEA, Taiwan, p. 407), and—with more regional importance—Australian Evaluation Society (AES) and Asian Pacific Evaluation Association (APEA). Same is true for courses provided by transnational actors (e.g., UNICEF in Sri Lanka) in corporation with several national organisations, but not on a regular basis. In general, the supply of evaluation training in Asia is—except for Australia—very arbitrary and a clear indication of a lack of becoming a profession. The picture is getting better if one puts the focus on communication about evaluation. In the academic system, the existence of scientific journals with a clear focus on the professions’ topics is of certain importance for the discipline and its establishment at national level. In four countries, the exchange on evaluation is merely organised through a particular journal. The Australian Evaluation Journal of Australasia (EJA) belongs to the oldest and well-established Evaluation journals in the world, although it has gone to some very difficult phases and seems to be stabilised after its move to SAGE a few years ago (Australia, p. 85). The journal is edited and published by the Australian Evaluation Society AES. In Japan, there are two evaluation journals available, the Japanese Journal of Evaluation Studies (published by the Japan Evaluation Society) and Evaluation Quarterly (published by the Institute of Administrative Management). Moreover, other Journals like the Journal of International Development Studies take on evaluation topics occasionally (Japan, p. 213). Taiwan reports three evaluation-focused journals, the Journal Evaluation Bimonthly as a more practitioner-oriented journal, the Journal of Educational Administration and Evaluation, an academic journal published in Chinese and the Journal Higher Education Evaluation and Development published in English by one of the major academic

16

SYNTHESIS

557

publishing house Emerald (Taiwan, p. 407). In China, there are two journals specifically focusing on evaluations, the Journal of Higher Education Development and Evaluation and Appraisal Journal of China. The former is an academic journal in the field of education, while the latter is a practice-oriented professional journal published by the China Appraisal Society, in the area of financial assets management. In the two countries, communication on evaluation does not work in specialised scientific journals but in other regularly published media. The trial to establish a scientific journal on evaluation failed in India after the third issue in 2012, but there is a quarterly newsletter issued by the Evaluation Community of India, informing on Monitoring and Evaluation activities and several academic journals published papers on evaluations (India, p. 183). The Pakistan Journal of Educational Research and Evaluation is an open access, peer-reviewed journal published by a university, but it does not have a primary focus on evaluation. Nevertheless, this situation in Pakistan is quite close to China and not far from the leading countries (Pakistan, p. 305). Communication on Evaluation in other countries is difficult and not much developed. The VOPEs in Nepal provide some options for informing on evaluation via newsletters and list servers, conclaves and special issues of journals. However, there is no regular medium for communication on evaluation in Nepal (Nepal, p. 263). The evaluation association in Sri Lanka is also very active and is collaborating with a lot of actors both national and international. Although there are many different activities, the Sri Lanka Evaluation Society does not provide regular communication channels or platforms (Sri Lanka, p. 380). In the Philippines, only the Philippians Monitoring and Evaluation Society (PMES) has a website that serves as an exchange platform, the other two VOPEs only use Facebook for communication. No journals or list servers exist, but the M&E Network Philippines telecasts an annual conference on Facebook. However, compared to other countries, this is a very poor communication infrastructure (Philippines, p. 345). In Bangladesh, the formation of an appropriate association started in 2012 and was finalised in 2018 by being officially registered as a non-profit organisation under the society’s registration act. Like other Asian Evaluation organisations, the Bangladesh Evaluation Society (BES) was involved in several regional meetings and hosted the fifth Evaluation Conclave of the Community of Evaluators-South Asia, a well-established opportunity for knowledge exchange and capacity building in South Asia. Up

558

R. STOCKMANN ET AL.

to now, BES is not offering a regularly published journal or newsletter but provides an informative website (https://evaluationsociety.org.bd/, Bangladesh, p. 114f.). Hence, from a communication perspective, the offers are still at a low level. In South Korea, there are no VOPEs, and communication on evaluation is not organised by any specialised network. Instead, several academic disciplines like economics, education, public health, public policy, and others are responsible for communication on evaluation, and research institutes play a remarkable role in providing information (South Korea, p. 232). There are three countries with large evaluation societies and still increasing membership. The oldest one is the Australian Evaluation Society (AES), founded in 1987 with today more than 1.000 members merely from Australia. The AES is one of the oldest and well-known evaluation societies in the World (Australia, p. 86). Compared to this, the other two associations, the Japan Evaluation Society (JES) founded in 2000 in Japan and organising nowadays 500 members (Japan, p. 197), and the Evaluation Community of India (ECOI), established in India in 2015 with more than 200 members (India, p. 184), are quite young. These organisations cover the whole range of evaluation communities and are solely representing their interests. Two other countries (Bangladesh and Sri Lanka) are making some progress in developing an organisational infrastructure for evaluation. Both the Bangladesh Evaluation Society and the Sri Lanka Evaluation Association are member organisations and organise communication, exchange, capacity development, publications etc., particularly for their members like the others mentioned above. Compared to AES and JES, these activities are rather limited but increasing in numbers and size. In difference to this, evaluation interests are divided into various organisations in three countries (Nepal, Pakistan, and the Philippines). There are three VOPEs on the Philippines with different alignments. While the informal M&E Net Philippines is composed of evaluation practitioners, most of them from the government sector, the second VOPE, the Philippians Monitoring and Evaluation Society (PMES) focuses on capacity building and the Philippine Evaluators for Development (PHILDEV) is a professional organisation closely linked to international organisations (including the APEA) (Philippines, p. 346f.). The three VOPEs in Nepal are all registered with the government and listed on the IOCE website. In difference to the other two organisations, the Community of Evaluators

16

SYNTHESIS

559

Nepal (COE-Nepal) does not conduct project evaluation but is engaged in organising conclaves, conferences, and workshops. COE-Nepal has 40 members, Nepal Evaluation Society (NES) 36 and Society of Monitoring and Evaluation SOME 30 members (Nepal, p. 263f.). In Pakistan, the Pakistan Evaluation Network (PEN) aimed on the use of aid money and technical use until 2015. The Pakistan Evaluation Association (PEA) is still promoting the evaluation culture in the country and preliminary coordinates the exchange with other countries. Both organisations are open networks with a decent number of followers and activities (Pakistan, p. 306f.). The other countries—China, South Korea, and Taiwan do not have a VOPE or any specific network/organisation for evaluation. There are several academic networks including evaluation topics in these countries, but without a specific professional organisation, evaluation is not able to become a profession on its own. As seen for education and communication, the role of VOPEs is very important in many fields, particularly for bridging the gap between different policy fields and discipline-oriented evaluation practices (Table 5). The role of VOPEs is also essential if it comes to the last point highlighted here, the development of norms and rules on how evaluations should be done. In most countries with such kind of norms, the national VOPEs have developed them and are responsible for their distribution (Australia, India, and Japan). The Australian Evaluation Society (AES) developed several documents guiding the use and implementation of evaluations, particularly the “Code of Ethical Conduct ” and the “Guidelines for the Ethical Conduct of Evaluations ”. In 2001, a draft set of standards for evaluation was developed but yet not approved. Two other frameworks, the “Evaluators Professional Learning Competency Framework” and the “First Nations Cultural Safety Framework” were settled in 2013 and 2021, respectively (Australia, p. 87ff.). Most recently, the Indian, the Japanese and one of the Nepalian VOPEs (ECOI, JESs and COEN) developed Ethical Guidelines and published them on their websites (India, p. 186ff.; Japan, p. 214; Nepal p. 265). Compared to the AESGuidelines and Frameworks, these new documents of smaller and less developed VOPEs seem to be yet little known. Hence, all these documents are not mandatory or include some form of self-commitment for evaluators or commissioners of evaluations. But even this voluntary state seems to include hurdles for some VOPEs in other countries—particularly in the Philippines and Sri Lanka: they are thinking about issuing

1000 900 800 700 600 500 400 300 200 100 0

Overview of VOPEs in Asia Pacific

Source Author’s own work

Table 5

VOPEs in Asia

560 R. STOCKMANN ET AL.

16

SYNTHESIS

561

guidelines or standards for quite some time but did not realise to finalise them yet (Philippines, p. 347f; Sri Lanka, p. 383). Not such activities are reported from the VOPEs in Bangladesh and Pakistan (Bangladesh p. 114f., Pakistan p. 307f.). Other more obligatory forms of regulations of evaluation can be found in China and Taiwan. In China, “there is a myriad of laws, ruling party guidelines, government decrees and implementation methods on evaluation activities including performance appraisal, performance management, and performance audit, at the Central government and local government levels in China. These standards and guidelines are imposed and enforced by the Chinese government and the ruling party” (China, p. 140). This strict regulation seems to be less associated with the political system than with the specific culture because they can also be found in Taiwan, which is a liberal multi-party democracy: “Both the internal evaluations conducted by government agencies, as well as the external evaluations conducted by entrusted NGOs or the Control Yuan follow regulations. These regulations determine the breadth and depth of the evaluation, its contents, and qualification requirements for evaluators… Compliance to these standards and quality obligations is enforced by penal provisions. In other words, independent evaluation institutes not adhering to the regulations may no longer be recognised by the respective executive-branch ministry or agency” (Taiwan, p. 410). For sure, these are the strictest regulations on evaluations in Asia and there are not many countries with such obligatory rules on evaluation in the World.

6

Summary

The state of institutionalisation in the system of professions differs significantly between the countries (Table 6)—not only due to the degree of development but also caused by several forms of professionalisation. Australia is for sure the most advanced country in professionalisation of evaluation on behalf of the four aspects in focus here. One of the main driving forces is the well-developed VOPE and the strong roots of the academic system at the University of Melbourne. India and Japan seem to follow this track most recently, although the institutions are not settled in a similar form. However, both the scientific anchorage and the development of the VOPE seem to be in a good way.

562

R. STOCKMANN ET AL.

Professionalisation score

Table 6

Country

Education

Australia

Communication Organisation

Norms

Mean

1

1

1

0,75

0,94

Bangladesh

0,5

0,25

0,75

0

0,38

China

0,5

1

0

1

0,63

India

0,5

0,75

1

0,75

0,75

Japan

0,5

1

1

0,75

0,81

South Korea

0,5

0,25

0

0

0,19

Nepal

0,5

0,5

0,75

0,5

0,56

Pakistan

0,5

0,75

0,25

0

0,38

Philippines

0,25

0,5

1

0

0,44

Sri Lanka

1

0,25

0,75

0

0,5

0,75

1

0

1

0,69

Taiwan Source Author’s own work

Table 7

CPEI professionalisation scale 0–1

Scale

I1 Education: Study Programmes

I2 Communication: I3 Focused Exchange Organisation: VOPE

I4 Norms: General Agreement

0.00

No offer available

No offer available

0.25

Only non-academic offers Only single academic courses Minor subject courses

Exchange in other discipline media Exchange in open media Exchange in regularly published media Exchange in academic journals

No VOPE existent Open network without duties Formalised network Small formalised organisation

No rules existent Informal agreements Self-commitment to internal rules Endorsed general rules

Large formalised organisation

Obligatory rules and certifications

0.50 0.75

1.00

Major subject courses

Source Author’s own work

The road of development in China and Taiwan seems to be quite different: there is almost no sign of establishing an open culture of evaluation profession in the civil society and the driving force is primarily the state. With obligatory rules on how to run evaluations, both countries are

16

SYNTHESIS

563

largely decoupled from the global evaluation community and its development. However, this does not mean a low degree of integration into the system of professions—there is a certain degree of communication organised by academic providence and the development is guided by strict political steering. The separation may also be increased by the language hurdle that is probably higher in this cultural entity than in South Asia where the majority of countries are allocated. Hence, South Asia is not homogenous in its state of institutionalisation of evaluation in the system of profession. Most countries—Nepal, Pakistan and Sri Lanka—show at least some progress in establishing evaluation as an identifiable part of the system of professions, but they also lag behind the development in India to a certain extent. Difficulties seem to be caused both on the academic side (a lack of established evaluation structures at universities) and the formation of a professional organisation for advocating evaluation (the forces are divided into several small networks with low numbers of members). Compared to this, Bangladesh is still at a very low level with the weakest score on professionalisation scale. The difficulties in the other two countries seem to be similar to the one described here. South Korea is without any doubt one of the most developed countries in the region, but regarding the institutionalisation of evaluation in the system of professions it is lacking significantly behind. Here, evaluation is still a part of other academic disciplines and almost no institutions for evaluation as a profession on its own are developed. In difference to South-Asia, research institutes seem to play a minor role in establishing evaluation research and trainings. In the Philippines, the difficulties are much more comparable to South Asian states. On one hand, there is a strong but divided evaluation society and some progress can be seen caused by this institutional framework. On the other, there is a lack of institutionalisation in the academic system, dominating the opportunities for becoming a profession on its own. Although there are still a lot of difficulties in the institutionalisation of evaluation in the system of professions, one has also to state a certain progress during the last decades. There are many efforts taken in the region and a lot of engagement—especially in contact with transnational organisations like the IOCE or UNICEF—helped to install the cornerstone for evaluation in most of the countries investigated here (Table 7).

564

R. STOCKMANN ET AL.

In difference to the results for the Americas (Stockmann & Meyer, 2022, p. 492) and Europe (Stockmann et al., 2020, p. 513), there is a certain correlation between the institutionalisation of evaluation in the political system and the professionalisation visible (see Fig. 2). Hence, this is merely a result of the poor institutionalisation in the political system—compared to some forerider countries in the Americas—particularly, the USA and Canada—and Europe—especially Switzerland—the most advanced countries like South Korea, India, Japan or Australia do not achieve comparable high scores on the political dimension. In general, four clusters of countries can be found in Asia. The first group (Australia, India, and Japan) reveals a high ranking on the inclusion into the system of professions and belongs to the leading countries in the region by institutionalisation in the political system—although this is still on a low level. Regarding the development of the profession, these countries already achieved a fair degree of integration in the academic system and the Australian Master of evaluation belongs to the best-established programmes in the world. Compared to this, the second group (China, Nepal, and Taiwan) is still on a much lower level of institutionalisation in the system of professions. Particularly in China and Taiwan, this difference belongs to the low degree of organisation of evaluation. In the political system, the deviation is caused by the lack of use of evaluation: while in Australia, Japan, and India some traces of evaluation practice can be found in a variety of sectors, evaluation is almost not used as an instrument in China, Nepal, and Taiwan. It seems, that evaluation is primarily an academic encounter in these countries and is driven by some disciplines like public administration or education. The third group (Pakistan, Philippines, and Sri Lanka) is still lacking behind the second group, merely caused by differences in the political system. While China, Nepal, and Taiwan developed institutions like laws and degrees or specific evaluation units in governmental organisations in a comparable way like Australia, India, and Japan, the countries in the third group do not have such kind of institutions. The differences in the system of professions are less important, but still exist. Although the three countries do not have many similarities in the development of institutions within their system of professions, the role of evaluation networks and organisations seems to be a bit more important than the influence of academic disciplines compared to China, Nepal, and Taiwan.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0

0.1

Pakistan

Philippines

Sri Lanka

0.2

Nepal

China

Taiwan

0.3

India

0.5

Korea

0.6

Institutionalisation in the Political System

0.4

Bangladesh

Japan

Australia

0.7

0.8

0.9

1

Fig. 2 Correlation between the institutionalisation of evaluation in the political system and the professionalisation (Source Author’s own work)

Professionalisation

1

16 SYNTHESIS

565

566

R. STOCKMANN ET AL.

Finally, there is a fourth group (Bangladesh and South Korea) that must be assigned as “outliers” regarded the correlation described above. Compared to the other Asian countries in this volume, South Korea achieved the highest institutionalisation of evaluation in its political system—although it is still on a low level—, but the poorest one in its system of professions. There are only minor traces of evaluation at universities and no national VOPE exists. Evaluation services are merely offered by public research institutes, a unique solution compared to other countries (South Korea, p. 226). In Bangladesh, the results for the system of professions are like Pakistan, the Philippines, and Sri Lanka. Differences can be found in the political system because evaluations are both formalised by institutions and used in practice in Bangladesh—at least to a higher degree than in the other countries mentioned here.

7

Challenges and Driving Forces 7.1

Challenges

The primary use of evaluation for accountability purposes promotes the image of evaluation as a control instrument. This can be observed particularly drastically in China, where evaluation is not only used “to inspect and examine the fiscal budget and financial accounts and monitor the executing of the fiscal budget” and to discover “suspicious misuse of the fiscal budget” (p. 130), but also to monitor persons. The Target Responsibility System (TRS) allows the party and the government offices to “evaluate the performance of its subordinates according to the extent to which they meet the present performance targets” (p. 126). Since this hierarchically organised form of “evaluation” involves rewards and punishment, it violates fundamental evaluation principles. In a number of other countries where evaluation is mainly used for control purposes, a negative impact on the understanding of evaluation can be observed as well. In some countries, organisational fragmentation is criticised as an obstacle to the further development and use of evaluation. In Bangladesh, “the lack of coordination with the executive branch of government” is criticised (Bangladesh, p. 115) and in Nepal, it is noted that the evaluation was entrusted to the line ministries, but these are not well linked with the major national development agenda and even contradict it in some cases (Nepal, p. 267).

16

SYNTHESIS

567

Only in a few countries in the Asia Pacific region is there a broader range of methodological evaluations carried out on a continuous basis in a larger number of sectors, using different types of evaluations (e.g., Australia, India, South Korea). Evaluations are often carried out internally (e.g., in China, Japan, Nepal, South Korea, Taiwan), which results in complaints about a lack of independence (China, p. 134, Australia, p. 74). Some of the internal evaluations are carried out by government officials who are not well-trained (Japan, p. 215). Impact evaluations26 are rather rare, even in Australia and India, which are characterised by a higher degree of use compared to other countries. Monitoring dominates in many countries where M&E structures have been established. For example, the Nepal report states: “Monitoring has become compulsory but not evaluation” (Nepal, p. 248). Or in the Pakistan report: “Ministries are still largely inclined towards project monitoring and there has been limited focus on evaluation of programmes” (Pakistan, p. 280). This is also the case in Sri Lanka (pp. 366 and 384) and the Philippines (pp. 335). Especially in countries where performance monitoring is anchored in sometimes very elaborate Management Information Systems, evaluations are neglected (India, p. 146f.). In some countries, methodological progress is explicitly reported (e.g., in India), in others, conflicting processes are reported, in which evaluation is rather a “bureaucratic box-checking exercise” carried out by the administration (Pakistan, p. 283). The lack of quality in the evaluations is an important issue in many case studies. In addition to inadequate financial resources (Nepal, p. 249; Pakistan, p. 294; Philippines, p. 335), which do not allow for methodologically demanding evaluations, the poorly qualified evaluators are primarily held responsible for this (Nepal, Pakistan, Philippines, Sri Lanka). The lack of quality assurance measures is another reason for the lack of evaluation quality. In very few cases, independent meta-evaluations are commissioned. In Australia (p. 75) it is said that this is “not common practice”, and in India, only one meta-evaluation (2013) has been carried out in the first place and it is lamented that: “there are no institutional mechanisms to assess the quality of evaluation” (India, p. 167). In other

26 This means methodologically demanding evaluations such as randomised control trails among others.

568

R. STOCKMANN ET AL.

countries, quality control lies with the responsible ministries or commissioning organisations themselves. However, it often cannot take place within these ministries and organisations at all, because the necessary qualified specialists are lacking. The criticism that evaluation results are poorly used by decision-makers is as old as the evaluation itself. Almost all case studies (see, for example, India, p. 165; Australia, p. 76f.) complain about this. However, it is also pointed out that the assessment of the degree of use depends heavily on how comprehensively use is defined (India p. 165; Australia p. 75ff.). In a number of countries in the Asia Pacific case studies, the fact that the results are not used is blamed not only on the lack of political will but also on the lack of technical skills and competencies. Thus, the Philippines report points out “that government agencies do not have the capacity to conduct M&E and use evaluation results” (Philippines, p. 339). In Sri Lanka, a “lack of implementation capacity” is regrettable (Sri Lanka, p. 383). Technical shortcomings exist when the evaluation products are of low quality and are not very suitable for implementation. For example, the Nepal case study criticises the fact that “recommendations were insufficiently based on rigorous analysis” and that the overall quality of the evaluations leaves something to be desired (Nepal, p. 251). But even for Australia, one of the few countries surveyed that has at least an average degree of use, it is often stated that the view prevails “that evaluations were often not valuable due to their narrow focus, weak methodology, and limited data” (Australia, p. 74). The Japan report criticises: “Most evaluation reports are written using official language and are difficult to read and understand” (Japan, p. 215). Conversely, poor evaluation products do not exactly foster demand or the political will to make more staff and finances available for evaluations. In addition, there is little knowledge and understanding among decisionmakers in many of the countries studied about the possible applications, the performance profile and the potential uses of evaluations. The low use of evaluations for policy-making is partly due to the fact that the necessary political culture is lacking. Evaluations are often regarded as a donor matter, which is carried out to “fulfill requirements of donor-funded projects” (Pakistan, p. 293; Philippines, p. 335). It goes on to say and is also illustrative for other countries: “Pakistan’s political hierarchy creates an environment where individual self-interest is more

16

SYNTHESIS

569

likely to determine policy makers’ decisions than empirical data and evaluation” (Pakistan, p. 308). Taiwan also complains that “there is not yet an intrinsic culture of evaluation” (Taiwan, p. 399). However, even for countries with a long-standing evaluation tradition, this does not mean that the standard once achieved can be continuously maintained. Australia, whose evaluation roots date back to the mid-1970s, states: “This golden age of evaluation institutionalisation in Australia did not last” (Australia, p. 68). At the end of the 1990s, mandatory requirements for evaluation in the policy planning process were lifted and evaluation budgets were significantly reduced. But this was also a turning point. In an independent review of the federal public sector (called the Thodey Review) commissioned by the government in 2013, optimisation of the use of data in decision-making and the implementation of targeted evaluations are urged to improve the efficiency and effectiveness of policies and service delivery (Australia, p. 69f.). The long-term decline in research and evaluation expertise is not only attributed to a lack of skills but also to the cultural practices that have evolved within the public service sector. The recommendations of the Thodey Review, which could also be useful for other countries, are: . Embed a culture of evaluation and learning from experience to underpin evidence-based policy and delivery . Develop an Australian Public Service (APS)-wide approach to build evaluation capability and ensure systematic evaluation of programmes and policies . Establish a central enabling evaluation function to support APS evaluation practices and expertise . Develop in-house evaluation functions and annual plans, and publish evaluations, unless exempted by the Cabinet and . Amend budget requirements to establish a systematic approach for formal evaluations (Australia, pp. 70) In India as well, which has an even longer evaluation tradition and where a Programme Evaluation Organization (PEO) was founded back in 1952 under the National Planning Commission and in the next two decades a series of evaluations were carried out, evaluation practices gradually became weakened in the 1970s. In recent years, the state evaluation

570

R. STOCKMANN ET AL.

system has been strengthened through structural reforms and the establishment of the Development Monitoring and Evaluation Office (DMEO) (India, p. 157). The same is reported in Japan. The case study states: “in the past twenty years, policy evaluation has become a routine task, is losing its substance, and may not be considered seriously even among government officials” (Japan, p. 215). In a publication cited in the case study, three reasons are given “why the government policy evaluation system is not functioning as intended”: (1) because the organisations lack professional evaluators and because such positions are not very attractive for career development (2) because there are no “perfect” evaluations and therefore there will always be dissatisfied users (3) because in the current policy evaluation system the connection between evaluation and planning is not well institutionalised (Japan, p. 215). The greatest challenge in terms of institutionalisation in the social system can be seen in relatively weak endogenous civil society processes in relation to the formation of evaluation, despite the fact that there is a long tradition of civil society as a central driving force of social development in many countries. To be sure, there are promising initial situations in this regard, be they media initiatives (e.g., in India), strong participatory institutions between the state and civil society (e.g., Australia, South Korea, and Taiwan), the increasing anchoring of evaluation in numerous CSOs—mostly the subset of CSOs and NGOs which frequently work together with international or donor organisations, that are wellembedded in international discourses on learning and accountability and that have a strong learning culture and awareness for the benefits of the instrument of evaluation (Australia, Bangladesh, India, Japan, Nepal, Pakistan, Sri Lanka, Taiwan), or the regular application of participatory evaluation procedures (e.g., Australia, Bangladesh, Nepal, Pakistan, South Korea), but a concept of evaluation shaped independently by civil society as well as a corresponding autonomous institutionalised and frequently implemented evaluation practice across a broad range of actors in civil society organisations has largely not yet emerged from this. The latter is expressed, for example, in the case studies by the frequent reference to

16

SYNTHESIS

571

“donor requirements” or “government regulations” as major reasons for evaluations in civil society organisations. There are two aspects of professionalisation processes that produce challenges in different forms in Asian countries. The first aspect is linked to the academic system of professions and how universities are handling evaluation. On behalf of this, some progress can be seen in Asia, although there is still room to improve. Many universities from different disciplines are offering frequently at least some courses on evaluation, merely integrated into policy studies, public administration, programme management, education or other related contexts. This practice is widespread, and one can find such courses almost everywhere. However, specialised study programmes on evaluation are still missing, with the exception of the well-established Master courses in Australia and the new attempts in Sri Lanka. The situation is quite similar if one looks at the communication platforms. Particularly in East Asia, some scientific journals on evaluation—merely in areas like education or health—exist and a lot of printed and non-printed media are used for communication on evaluation. Only in Australia and Japan, highly specialised journals are published, focusing on evaluation in a cross-disciplinary way. In most countries, communication on evaluation is institutionalised outside the academic sector and in the form of newsletters, blogs or other kinds of media organised by evaluation networks. This last aspect relies on the second challenge, the building of regular collaborations on evaluation and this is still a missing step in some countries. With the foundation of the Asian Pacific Evaluation Association (APEA) ten years ago, the formation of VOPEs got a push in the region and some of these groups achieved a fair number of members. Compared to the big Australasian Evaluation Society, most VOPEs are still small or open, not formalised networks. Having in mind, that there are the biggest countries of the World—China and India—among the cases observed, the size of VOPEs and their degree of formalisation are still at a very low level in Asia. While the development in the most advanced countries (particularly in Australia and Japan) seems to stagnate, some countries with a high dynamic in developing evaluations are coming into serious political or economic trouble and this is also hindering the development of evaluation (cf. Sri Lanka). No doubt, there are some encouraging developments in the region and there is some hope that the challenges mentioned here can be overcome.

572

R. STOCKMANN ET AL.

7.2

Driving Forces

In the Asia Pacific region, four driving forces, in particular, can be identified that have led to institutionalisation and implementation of evaluation: (1) The review of the programme and development objectives set by planning institutions (such as Ministries of Planning) (2) The use within programme planning and budgeting systems (e.g., by finance ministries) (3) The use of donor evaluation practices for the development of own M&E systems (4) The review of the achievement of Sustainable Development Goals (SDGs) (5) Evaluation in the context of organisational development Evaluation in the Context of Planning Processes27 India is the country with the oldest tradition of evaluation in the Asia Pacific region, dating back to the 1950s. To measure the programme and performance of the development goals set out in five-year plans, 1952 the Planning Commission has set up a special Programme Evaluation Organization (India, p. 2 and p. 8). Even today, “evaluation is being driven mainly because of the value planners, policy makers, and programme implementers are placing on it as an instrument for accountability and improvement of public programmes” (India, p. 164). The international donor practice of evaluating projects and programmes has also proved to be a driving force in India. As an independent system, however, it no longer serves as a response to donor requirements “but tend[s] to be driven by an urge to learn from evidence and enhance programme performance” (India, p. 163). The situation looks different with regard to civil society participation in evaluations. While national policies like the Right to Information and Public Interest Litigation have strengthened the application of participatory approaches, viewed overall, there is still “limited organised spaces for citizens to participate in the M&E” activities (India, p. 174).

27 The country’s order is based on the historical development—from the front runners to the late comers.

16

SYNTHESIS

573

Neither the long-lasting tradition nor the intensive international support has led to an outstanding degree of institutionalisation in the system of professions. Without no doubt, the academic sector is welldeveloped in India, but there are still no established Master’s courses or other kinds of full-time evaluation programmes. Most of the activities towards establishing evaluation are quite young and obviously merely driven by the international development and not because of the national demand for evaluation (India, p. 188ff.). The beginnings of M&E in Pakistan are housed under the Economic Affairs Division with the establishment of a National Planning Board in 1953. The system was advanced in the 1960s and 1970s through the Harvard Advisory Group, which advised the Planning Commission on M&E issues (Pakistan, pp. 276). Further impetus was given to the M&E system when a Results Based M&E System was set up in the wake of the Paris Declaration on Aid Effectiveness. In the 2010s, M&E became a major function of the Planning Commission and thus is strongly linked to the tradition of performance measurements of accountability and accounting (Pakistan, pp. 276 and p. 291). As part of these government evaluations, the participation of civil society organisations takes place on a regular basis (Pakistan, p. 299f.), however, it is not formalised. There are participatory governance institutions, however, as there is limited evidence available, it is not possible to say how far they are being used for evaluation-related processes. Overall, evaluation is seen as a policy-making tool rather than an evidence-based decision-making and management tool for CSOs. Compared to India, the development in the academic sector is almost similar: although there is a huge and well-developed system of universities, there are still no trials to build up a fully equipped study programme on evaluation. The 2010 reform does not yet lead to a push for evaluation and networking is on a lower level than in India, maybe because of fewer international initiatives in the country. South Korea set up a policy evaluation system in the 1960s to review the programme of the Five-Year Economic Development Plan, as was done in India (Korea, p. 220). The system focused heavily on economic factors. With the adoption of the Government Performance Evaluation Act (GPEA), a system was established on the basis of which the ministries carry out a “self-assessment” that “covers the assessment of major policies, budgetary programmes, and administrative capacity”. These assessments are carried out by a self-assessment committee, which

574

R. STOCKMANN ET AL.

consists of ten to 30 external experts and internal members. There is also a top-down assessment, which is the responsibility of the Prime Minister’s Office. In addition, there is an independent performance management and evaluation policy of the Ministry of Economy and Finance (MoEF) for budgetary programmes. This means that the origins of this threestakeholder system (ministries, PMO, MoEF) were mainly used to review targets and budgets. Meanwhile, the spectrum has become broader and the evaluations carried out by the ministries are more formative for developing programme improvement plans. PMO and MoEF carry out evaluations primarily with the intent “to examine whether policies and programmes meet their intended goals” (South Korea, p. 226). Participatory approaches are sometimes used within the framework of these evaluations, and civil society participation regularly takes place in the form of evaluation advisory boards (South Korea, p. 227f.). In the South Korean civil society, CSOs regularly use evaluations for organisational purposes and sometimes to “raise issues for government policies” (South Korea, p. 227), even if viewed in comparison with government evaluations, “their role as producers of evaluations is limited” (South Korea, p. 227). Some degree of institutionalisation can be seen in the form of participatory governance instruments that, while not specific to evaluation, encourage participation in evidence-based practices and then regularly find expression in the use and demand of evaluative evidence. On the supply side, almost no institutions are built for the system of professions. Evaluation is primarily done by public research institutes and therefore “the market for well-trained freelancers is limited due to well-established public research institutes and universities” (South Korea, p. 232). Since the establishment of the State of Bangladesh in 1971, the Monitoring and Evaluation Division (IMED), which is located in the Ministry of Planning, has been implemented. It is centrally responsible for monitoring and evaluating the performance of all ministries and divisions (Bangladesh, p. 99f.). Its results are mainly used by the Ministry of Planning and the National Economic Council. However, there is still enormous development potential to establish “a [more] collaborative and synchronised system through appropriate and efficient use of skills, resources, and technology” in order to overcome the nascent stage of evaluation (Bangladesh, pp. 116). This is also true for the potential of civil society participation, as the participatory approaches used in the country are particularly evident and

16

SYNTHESIS

575

frequently conducted in certain policy areas such as development cooperation (Bangladesh, p. 108). Within civil society, evaluation is regularly used in some CSOs for organisational purposes as a management tool (ibid., p. 108f.). The situation on the supply side is comparable to the other countries mentioned here. Evaluation is primarily a task of think tanks and research institutes, it is not established in the academic system as a topic on its own and the formation of a network for evaluation is still at an early stage, although some important steps have been taken during the last years (Bangladesh p. 111f.). In Sri Lanka, evaluation is a rather new phenomenon that has its roots in the establishment of a Ministry of Plan Implementation (MPI). The beginnings of professional evaluation are dated to the late 1990s when the Asian Development Bank and the United Nations Development Program (UNDP) provided support for a Results Based Monitoring and Evaluation System (Sri Lanka, pp. 356 and p. 363). The system is very focused on monitoring and is used primarily for control purposes. Accordingly, with regard to civil society, participatory evaluation approaches are sometimes applied—mostly due to donor requirements—, but they do not define the overall picture and thus are considered to rather have a “minimal” significance (Sri Lanka, p. 375). Like in other countries, “evaluation is not yet professionalised” in Sri Lanka because there “was no demand for an evaluation profession” (Sri Lanka p. 377). Due to the increasing support of UN agencies, there are a lot of activities for professionalisation of evaluation including initiatives at the universities as well as for networking. Japan is also one of the latecomers in evaluation. This is mainly due to the translation of the word “evaluation”. Because this is usually translated as “Hyoka” in Japanese. But as in English, where the word “quality” is usually automatically associated with the meaning of “good quality”, so it is in Japanese with the word “Hoyka”, which is equated with “it is positively evaluated”. If something has been negatively evaluated, this is described with the phrase “it is criticised”. This notion—according to the case study— “has been a barrier to promoting any type of evaluation for a long time” (Japan, p. 196). The breakthrough occurred in the 1990s following new approaches brought in by the OECD and the introduction of the concept and implementation of New Public Management. This development was supported by concepts from the development sector, in which the Japan International Cooperation Agency (JICA) introduced

576

R. STOCKMANN ET AL.

the Project Cycle Management (PCM) method. This first component of evaluation history, which is characterised as “performance measurementtype evaluation”, was accompanied by a second stream, initiated by the central government and aimed at improving administrative performance. However, these activities led to a “marginalisation of the policy evaluation system” and the tendency to equate policy evaluation with performance management (Japan, p. 197). In the meantime, this system has degenerated to such an extent that all that is left is that simple performance evaluation sheets are filled in by ordinary government officials. The reason given is: “This is because it is much easier for ordinary government officials to measure their performance rather than to evaluate outcomes” (Japan, p. 203f.). In Japan the “institutionalisation of evaluations has been led by the central government” (Japan, p. 210). While civil society’s demand for evaluation as a specific governance instrument seems to be limited (Japan, p. 210f.), many CSOs regularly use evaluations but are most likely to do so because of government requirements. CSOs are not involved in evaluation processes by default in the form of institutionalised procedures (Japan, p. 210f.) and also the “use of participatory approaches in the evaluation and monitoring system by institutions remains limited in Japan” (Japan, p. 211). In difference to all other countries yet mentioned in this part, Japan was able to establish a strong Evaluation Society (JES) including 500 members from academia, consulting firms and government. By publishing a peer-reviewed Journal and organising annual conferences JES “has become an essential communication platform” (Japan p. 213). This is the key driving force on the support side, while evaluation is still fragmented in the academic system of evaluation and some other, more discipline-oriented organisations are competing with JES. In cases where M&E is primarily used by planning institutions (India, Pakistan, South Korea, Bangladesh, Sri Lanka, Japan), it is mainly used for performance measurement, accountability and in some cases also for the improvement of public programmes. Monitoring and internal forms of evaluation often dominate. Much of the evaluation is organised in planning institutions and it is difficult for evaluation to develop an independent character. This situation also affects whether and how evaluation is practised as a participatory learning tool aimed at strengthening the dialogue between the state and civil society. This seems to have a certain effect on the supply side, because—with the exception of Japan—all other

16

SYNTHESIS

577

countries failed to establish appropriate institutions for evaluation in the system of professions. The incorporation of evaluation in the context of planning institutions is probably not supporting the institutionalisation of evaluation in the system of professions. One explanation for the success in Japan in contrast to this assumed mechanism may be the transnational scientific communication on New Public Management and Quality Management in the 1990s. Japan (and Japanese University) are probably much better included—not only via scientific networks but also via OECD (Japan is a member since 1964, and South Korea is the only other member joined in 1996). In the Americas and Europe, these discussions on new public management were strong drives for evaluation as an instrument. Evaluation in the Context of New Public Management and Programme Planning and Budgeting Systems The use of evaluation in the context of budgeting processes can be regarded as a second driving source of evaluation: In Australia, the tradition of evaluation dates back to the 1970s. Starting from the discussion of appropriateness, efficiency, and effectiveness of government activity and public service, a “shift towards New Public Management (NPM), which introduced programme planning and budgeting systems that integrated evaluation within the policy cycle” can be noted in the 1980s (Australia, pp. 67). Under the auspices of the Ministry of Finance, a “Financial Management Improvement Program” was developed, which was linked to the establishment of evaluation structures. Evaluation was integrated into all new policy proposals and measures were implemented by the Ministry of Finance to strengthen the evaluation capacity across departments. This means that the development of the Australian M&E system is strongly linked to a programme planning and budgeting system based on the Managing for Results concept (Australia, p. 67). In Australia, civil society uses and demands evaluation more frequently than in many other Asian countries. Although there is still a lot of potential for development in this case as well, overall aspects such as regular and institutionalised participation in evaluations as well as their regular and institutionalised demand are more advanced than in other countries in the region. Institutions of participatory governance are frequently being used to request evaluative evidence (Australia, p. 79f.) and there are evaluation regulations that require the involvement of civil society

578

R. STOCKMANN ET AL.

and organised ways of building on the strong tradition of participatory evaluation in Australia on a regular basis, as there is an awareness for its benefits (Australia, p. 80f.). As in other countries, many CSOs do not use evaluations regularly, and when they do, standardised evaluation types that are primarily used due to external accountability requirements often predominate. Only a few CSOs, such as the more “international focused” ones, have a diversified evaluation portfolio (Australia, p. 78). As in other countries of the region, this is also related to fragmented civil society knowledge about evaluation, which leads to low awareness of the organisational benefits of evaluation in many CSOs. The supply side is also well-developed and “evaluation practice is relatively mature and includes established higher education and training programmes, a range of professional development courses, a journal, a large marketplace for evaluation consultancies and a professional society with regional networks and special interest groups” (Australia p. 82). Since the early 1980s, training programmes (including a Master’s) are offered at several Australian Universities and also other important steps towards becoming a professional are made during the 1980s (first national evaluation conference in 1982, foundation of Australasian Evaluation Society in 1987, first edition of the peer-reviewed Evaluation Journal of Australasia in 1989, and the first version of the Code of Ethics in 1992). At least, there is some temporal coherence between this development and the New Public Management debate, leading to a very early establishment of evaluation as part of the academic system of professions and to a formation, stabilisation and institutionalisation of evaluation networks in the form of a professional and globally respected evaluation society. In the Philippines, evaluation has been part of the National Economic and Development Authority since 1987, which is responsible for macroeconomic forecasting, policy analysis, and research and has been supported by UNDP for several years (Philippines, p. 331f. and p. 337). Above all, the establishment of a results-based budgeting system is mentioned as a driving force. With this shift to results-based planning and budgeting in 2011, the need for evaluation steadily increased (Philippines, pp. 326 and 332f.). The Development of Budget and Management uses evaluation results for the country’s national budget formulation. Similar to Nepal and Sri Lanka, evaluation in civil society is used regularly in some CSOs in the Philippines—but usually in the context of donor-funded programmes. Civil society is occasionally involved in evaluations in the Philippines, but primarily in the data collection phase,

16

SYNTHESIS

579

which is why it is not yet possible to speak of a regular application of participatory evaluation approaches. Although there had been government initiatives for capacity building, they “are also heavily reliant on external evaluation companies or independent evaluators to provide such services” (Philippines p. 344). The evaluation community is fragmented and still very weak, maybe caused by this combination of government and externally dominated activities. There is a lack of institutionalisation in the academic system of professions and therefore “evaluations are often conducted by freelance practitioners (including consultancy teams) who have extensive practice and relevant experience in evaluation” (Philippines, p. 347) and not by professional academically trained evaluators. On the one hand, Taiwan’s M&E system sees itself as part of the ancient Chinese tradition of process evaluation and performance audit but emphasises “that our current notion of ‘evaluation,’ only commenced with the involvement of the United States in Post-World War II” (Taiwan, p. 392). Evaluation is part of the control system Control Yuan (CY) “the most comprehensive body in the context of evaluation at the national level”. CY is responsible for “auditing government agencies, censuring civil servants, and impeaching public officers” and is compared in scope and function with the Audit Court in France (Taiwan, p. 392). The control and monitoring aspect of evaluation is at the forefront in Taiwan. This is also expressed in the rather rudimentary involvement of civil society in these evaluations. This obviously has some impact on the supply side. One important specific of Taiwan (and China) is the higher degree of evaluation regulations. “These regulations determine the breadth and depth of the evaluation, its contents, and qualification requirements for evaluators” and “should not be confused with national laws and regulations wherein evaluation is mandated in principle for a specific sector or policy” (Taiwan p. 410). On the opposite, the civil society component of free and open evaluation networks is underdeveloped and as in many other countries, evaluation is fragmented and included in several sectors and academic disciplines. Civil society in Taiwan regularly uses evaluations, and it often does so in the context of political decision-making processes, unlike in most other Asian countries studied. In addition, it is often commissioned with evaluations of the government. Both contribute to the systemic control function of evaluation. Civil society participation in evaluations, on the

580

R. STOCKMANN ET AL.

other hand, does not play an emphasised role (Taiwan, p. 403). Despite numerous participatory governance mechanisms, evaluation has not yet established itself as a standard procedure even in this country with a strong civil society tradition. On the part of the civil society demand side, there are institutionalised possibilities for requesting evaluative evidence, and the open society of Taiwan makes regular use of these various possibilities—even though evaluation in the narrower sense is often not involved. Overall, it can be concluded that the control function is more significant than the dialogue function between the state and civil society. China can look back on an ancient tradition of control. The concept of evaluation was introduced with the “reform and open-up” policy of the 1970s and the associated aim of modernising government institutions and increasing its governmental competence. Currently, three types of activities can be differentiated: (1) The performance appraisal in government human resource management, (2) a Target Responsibility System to evaluate the performance of subordinates by the party and government offices, (3) fiscal auditing (China, pp. 125ff.). All three systems are primarily used for top-down control and inspection. In China, “there is a myriad of laws, ruling party guidelines, government decrees, and implementation methods on evaluation activities including performance appraisal, performance management, and performance audit, at the Central government and local government levels” (China p. 140). Like in Taiwan, the evaluation community is governed in a strict top-down manner and even less respect the international view of an open and equal exchange between independent experts. Existing organisations are part of government bodies and do not focus particularly on evaluation. Between these two, differences can be identified with regard to the role civil society plays in evaluation practices and institutions. Since in China evaluation is highly regulated and almost exclusively an internal government matter, an external control function by civil society is not implemented. The hardly pronounced and rarely expressed civil society demand for evaluation can also be seen in this context. Participatory approaches to evaluation are documented; however, they only play a limited role. In some countries (Australia, Philippines, Taiwan, China) evaluation has been advanced in the context of Results-Based Budgeting Systems or Programme Planning and Budgeting Systems. As with the M&E systems, which are controlled by the planning ministries, their goal is Managing

16

SYNTHESIS

581

for Results, but they are actually located in financial institutions and are also used for budgetary planning and control. They are often carried out internally and have a strong monitoring component. With the interesting exception of Australia, this mode seems to be not very helpful for establishing and developing a strong and lively evaluation community. In Australia, the strong inclusion into international discussions on public policy (both on the scientific and political side) in the early 1980s helped to build up appropriate, strong and stable evaluation institutions. Taiwan and especially China represent also to a certain degree a special case, in which the control and monitoring aspect not only includes goal performance, auditing, and budgeting, but also the performance of subordinates with the Target Responsibility System. These correspondents with a much stronger regulated evaluation community, offering services in a government-driven way for certain sectors and educated in traditional disciplines without much trials for building up an overarching evaluation profession. Evaluation in the Context of International Cooperation In Nepal, M&E has a long tradition dating back to the 1950s. At that time, it mainly served the goal of checking whether the plan had been fulfilled, whereby the focus was and still is on monitoring (Nepal, p. 266f.). The economic liberalisation policy of the 1990s established a new evaluation system “to achieve expected outcomes through regular, effective, and efficient M&E mechanisms” (Nepal, p. 238), which was further advanced under the Republican-Democratic governments (since 2007). This development was mainly driven by international development cooperation and international declarations (Paris Declaration on Aid Effectiveness, 2005; Accra Agenda for Action, 2008). Since a large part of the development budget comes from donor agencies, the instruments of M&E have been practised for a long time. This practice “contributed significantly to strengthen M&E systems in Nepal” (Nepal, p. 240) and inspired VOPEs and a parliamentary forum to develop proposed legislation, which, however, has been awaiting parliamentary approval for a long time. The Nepalese M&E system is still dominated by the donor perspective. Thus, it is reported that the recommendations are still “biased towards the donors’ agenda” and that in many cases, “evaluations are carried out to fulfil the donors’ interest, but they are hardly used for improvement, accountability and learning” (Nepal, p. 253).

582

R. STOCKMANN ET AL.

Donor requirements are one of the main reasons for evaluation practices in the Nepal civil society and also for the participation of the same in such evaluations (Nepal, p. 264f.). The strong donor focus also contributes to a relatively low awareness about the concept of evaluation in the broad variety of civil society actors, which finally leads to low civic demand of evaluation. On the supply side, the donor perspective is visible for quality standards, developed from VOPEs and used for evaluations by agencies (Nepal, p. 264f.). More indirectly, the donor perspective is relevant in non-academic trainings offered by several ministries and the National Planning Commission (Nepal, p. 261f.). The three VOPEs are also very active in transnational activities and connected to the Community of Evaluation South Asia and EvalPartners (Nepal, p. 264). There are no relevant alternative perspectives coming from universities or other academic institutions, the few training courses offered in the academic sector, for example, are also linked to development policy and programme management (Nepal, p. 260f.). In Bangladesh, too, donor practices are a driving force for evaluation. Evaluations continue to be carried out because they are part of donor responsibility (Bangladesh, p. 110f.). That’s also true for civil society, where local CSOs rarely independently conduct evaluations, but they likely do it in the context of “involvement by international organisations” (Bangladesh, p. 108). In Sri Lanka, where evaluation developed primarily as an instrument of the Ministry of Planning, several global events, such as the International Year of Evaluation 2015, have led to increased government attention for M&E. The government has also been inspired by donor activities to use evaluations to measure success in development projects. In the 1990s, the technical support provided by UNDP contributed “Results-based Monitoring and Evaluation System (RBME)”. This support contributed to government officials’ “[understanding and recognition] of the importance of focusing on results” (Sri Lanka, p. 363). Donor requirements can also be considered as the main driving factor when considering the use of evaluation in Sri Lanka’s civil society (Sri Lanka, p. 370). For example, at the rather local level, where CSOs more rarely work together with international donors and funding agencies, evaluation is much less well-known and applied (ibid., p. 371). Like in Nepal and Bangladesh, in accordance with low levels of knowledge on evaluation

16

SYNTHESIS

583

in the broad variety of CSOs and among citizens in all three countries, there is also no significant civic demand of evaluation, when demanded it is mostly due to external donor requirements of funded projects (ibid., p. 376). In some countries (Nepal, Bangladesh, Sri Lanka), international cooperation and donor evaluation practices have proved to be the main driving forces behind the development of evaluation structures. In some cases, they have been and continue to be supported by international organisations. In these cases, however, it has not yet been possible to set up independent national evaluation systems, so that the donor perspective still dominates. This is also true on the supply side, where evaluators merely practice on behalf of international donor agencies, and evaluations commissioned by national organisations are comparably low in number. There are many activities from international organisations, particularly UN agencies, for supporting the development of institutions for Monitoring and Evaluation in a couple of Asian countries (not only but especially in Bangladesh, Nepal, and Sri Lanka). Further information on this issue will be presented and discussed with respect of its impacts on the development of evaluation communities in Chapter 15. Looking at the role of civil society in this context, the international cooperation sector can be described as a field of action that is a driving factor in terms of participation as well as knowledge about evaluation, media discussions and the use of evaluations in civil society. Although in many countries civil society is relatively insignificant compared to the political system as a historical endogenous force in institutionalising evaluation, it can be said that initiatives and practices often originate from the field of international cooperation (e.g., Bangladesh, India, Pakistan, Japan, Philippines, Nepal, Sri Lanka). The historical origins of evaluation practices of civil society within the international cooperation sector can be traced back to the requirements of international donor organisations, which advocated and mandated the use of diverse project and programme evaluation methods, including participatory ones. However, contemporary evaluation practices in civil society are not solely shaped by external international factors. In some Asian countries, endogenous developments in civil society have started to shape evaluation practices and institutions within the international cooperation sector, albeit to a limited extent. These organisations that independently use evaluation are aware of the benefits of evaluation, usually have a stronger organisational learning culture, participate regularly in MEAL discourses and

584

R. STOCKMANN ET AL.

often work together with international and donor organisations (“international network contacts and embeddedness”). It should be noted that these developments are still in their early stages. To conclude, when compared to the impact of political frameworks—both at the international level through donor organisations and at the national level through state regulations—the influence of these endogenous developments on shaping evaluation practices and institutions in the international cooperation sector still remains relatively limited. Evaluation in the Context of the SDGs In recent years, the adoption of Sustainable Development Goals (SDGs) has spurred the need for M&E, although monitoring has played a greater role. Thus, for example, the India case study explicitly points out that the adoption of SDGs has developed a sense of urgency in promoting M&E. The obligation to document the achievement of objectives has significantly increased the need for statistical data and the collection of monitoring data for a number of parameters in India (India, p. 188ff.). The Pakistan report states: “Another major change in the perception of evaluation is driven by the adoption of SDGs by the government, which has underscored the significance of conducting evaluations” (Pakistan, pp. 289 and 308f.). The Bangladesh, Philippines, and Nepal case studies also point to the importance of the SDGs for the further development of the national M&E systems (Bangladesh, p. 104; Philippines, p. 337; Nepal, p. 249f.), In a number of countries (India, Pakistan, Bangladesh, Philippines, Nepal), the adoption of SDGs has improved political attention to M&E structures in recent years. Nevertheless, the Voluntary National Reviews (VNR) of these countries reveal some difficulties to take up this topic and to develop concepts for National Monitoring and Evaluation Systems (NMES) that are adequate for reporting progress towards the SDGs. Nepal, for example, linked the National Evaluation Policy with action plans and assigned a clear role to evaluation—and this is still a rare exception among the countries (Meyer et al., 2018). Hence, the institutionalisation of evaluation still seems to be too weak especially in the academic system and for developing solutions for such a complex goal system. It is likely that developments in the context of the SDGs will strengthen evaluation in different categories of social systems. First, the aforementioned VNRs, as well as SDG reporting in general, may

16

SYNTHESIS

585

strengthen the transparency dimension of M&E reports to the extent that evaluation plays a prominent role in them. In addition, it is plausible to assume that media coverage of evaluation in the context of the SDGs will increase (e.g., India, p. 173). Both can have a positive impact on the knowledge dimension and thus on the social perception of evaluation beyond specific M&E professional circles. The SDGs also encourage general uptake and adaptation by civil society, e.g., in the often-promoted form of implementation in the strategies of CSOs and corporations. This may also lead to greater civil society use of, and demand for, SDG evaluation. Finally, the multilevel governance perspective implemented in the SDGs contributes to a revaluation of the local level, which might also be expressed in the evaluation of the SDGs through the corresponding involvement of local civil society stakeholders (cf. Llanos et al., 2022, pp. 20–22). Evaluation in the Context of Organisational Development Evaluation has its origins in the political sphere and is therefore primarily a policy or governance instrument of that societal system. In addition, its use as an evidence-informed organisational development tool can be observed in some third-sector organisations. The overall picture is complex, with many empirical nuances and explanatory factors: Across the range of different civil society actors in a country, and in terms of regularity of use, civil societies make very little use of evaluation for political and organisational purposes, but when looking more closely at CSOs as a specific subset of civil society activities (cf. Chaplowe & Engo-Tjéga, 2007, p. 259), there are some examples in which evaluation is regularly applied and organisationally anchored. However, the evaluation practices of these organisations are not yet shaping the general “image of evaluation”, and thus cannot be seen as an independent driving force. Instead, many of the evaluation practices in use can be traced back to governmental or donor requirements, so in most cases, the political framework institutions and incentive structures shape and condition the development. To a limited extent independent organisational reception of evaluation as an own decision-making tool can be identified in a subset of CSOs. Like in other world regions, the development is in its initial stage. An explanation cannot be reduced to a single factor like organisational size. There are various explanatory factors that make a local reception more likely: First and foremost, the international orientation of the respective

586

R. STOCKMANN ET AL.

CSO, its embedding in global discourses on learning and accountability and the organisational awareness for the benefits of evaluations, but also organisational factors like the general learning culture contribute to it. International Organisations (IOs) are a main driving force in the evolution of evaluation, this is also true when looking at their organisational evaluation practices and routines (see Chapter 15). Here, evaluation is a well-established practice of organisational development. That’s why they serve as a role model or blueprint for other organisations. Based on this, evaluation as a tool for organisational development can become an important driving force at the local level in the future. CSOs and also other organisations from other societal systems like enterprises may increasingly become aware of the benefits of evaluation as well as the importance of organisational evaluation capacity and start independently receiving or specifically adapting the tool in their organisational routines. Finally, this has the potential to open-up further opportunities for joint learning and knowledge transfer between civil society, academia, politics, and economy. Overall, it can be stated that evaluation in the Asia Pacific region is mainly used for control, accountability, and performance management. Structurally, evaluation is primarily linked to planning ministries and government financing institutions in the context of budgeting processes. Accordingly, evaluation is used more for planning and implementation processes and measuring performance or in the context of budgeting processes. The donor organisations have promoted and supported evaluation development through international declarations, actively through projects for the development of M&E structures as well as through their practical application in the development projects and programmes financed by them. In some countries, the adoption of the SDGs has increased the understanding of the benefits of evaluations over the past decade. These two factors also contribute to a growing awareness of evaluation, at least sectorally and within certain CSOs: First, the multilevel governance approach of the SDGs specifically broadens the analytical perspective to also include local stakeholders such as CSOs, which also allows for learning effects in terms of corresponding multilevel participatory evaluation concepts. Second, external donor requirements contribute to a continuous evaluation practice within a part of civil society and, thus, in some cases to improvements in the organisational learning culture and

16

SYNTHESIS

587

especially an awareness for evaluation, which ultimately makes organisational evaluation capacity building and independent evaluation use more likely. However, considering the vast variety of cases, the development is still in its early stages. Assessed by the evaluation functions, the dominance of control and accountability is striking. To a much lesser extent, the learning function is also used. Evaluation as a legitimation instrument, in which it makes evaluation results transparent in order to discuss them in parliamentary or public debates to justify political action, plays a very insignificant role. Overall, while evaluation is largely a more or less institutionalised administrative practice of the public sector and the contribution of actors of the social systems to the institutionalisation of evaluation is relatively small compared to the relevance of the political system, the regular use of evaluation for organisational purposes and the beginning establishment of their own organisational approaches to evaluation in a specific subset of CSOs in some countries indicates a distinct development dynamic; however, this is often stimulated by political incentive structures (e.g., governmental evaluation regulations or those of donors) and often relates to a relatively small subset of CSOs in the countries. In rather local CSOs with less contact to international funding actors and global discourses on MEAL, the case studies are relatively unanimous in pointing out the limited and not regularly observable evaluation practices and the various constraints (e.g., low evaluation expertise or general low awareness for advantages of the instrument) in this regard. In some countries, there is evidence of regularly conducted government evaluations involving civil society, but this is largely not formalised in the region as a standard tool of participatory governance mechanisms, and compared to the control or performance management and accountability function, the dialogue purpose enabled by participatory evaluation is not particularly strong in Asian countries, with some exceptions such as Australia and South Korea. To realistically assess these findings, it is important to underscore that evaluation, as a governance instrument, has its historical origins in the political system. Only from there, the instrument increasingly diffused into civil society organisations as a requirement to analyse and evaluate the outcomes as well as the impact and sustainability of projects and programmes. There it is increasingly used and established as an organisational development instrument in a specific subset of civil society organisations; but on the whole, it is still on the way to anchor itself as an independent tool for the various functions of evaluation. The primary

588

R. STOCKMANN ET AL.

functions currently being pursued are reporting for external accountability and internal organisational results control for performance management. On the supply side, one has to state that this general focus on control and accountability seems to be a hindering factor for the inclusion of evaluation in the academic system of professions. In most countries, evaluation is still treated as a sub-issue in certain academic disciplines and cross-sectoral trainings and research are rare. Monitoring and Evaluation concepts are primarily developed and monopolised by government agencies and large think tanks, there is almost no academic discussion or exchange between practitioners in strong evaluation societies or networks. Interesting exceptions are countries like Australia and Japan—and to a certain extent also India—that have been better included in the global academic system. Particularly, Australia has taken up the discussions both in OECD (on the political level) and in academia on New Public Management in the 1980s and this is one of the reasons why the country developed institutions in the system of profession that are still existing today. However, there are some encouraging new developments in the last decade in some of the other countries, driven by international support and an increasing transnational exchange in the region. Organisations like APEA, EvalPartners, CoE South Asia established new platforms and enhanced opportunities for discussing and exchanging ideas within the growing evaluation community. Although this is merely limited to evaluation practice, one may expect some effects on the academic sector and new opportunities to anchor evaluation at universities both in teaching and research.

References Abbott, A. (1988). The system of professions. An essay on the division of expert labor. Chicago University Press. Anheier, H., & Ben-Ner, A. (1997). Shifting boundaries: Long-term changes in the size of the for-profit, nonprofit, cooperative and government sectors. Annals of Public and Cooperative Economics, 68(3), 335–353. Argote, L. (2012). Organizational learning: Creating, retaining and transferring knowledge. Springer Science & Business Media. Argyris, C., & Schön D. A. (1996). Organizational learning II. Theory, method, and practice. Addison-Wesley.

16

SYNTHESIS

589

Borrmann, A., & Stockmann, R. (2009). Evaluation in der deutschen Entwicklungszusammenarbeit. Band 1: Systemanalyse. Band 2: Fallstudien. Sozialwissenschaftliche Evaluationsforschung, Band 8. Münster: Waxmann. Boxenbaum, E., & Jonsson, S. (2017). Isomorphism, diffusion and decoupling: Concept evolution and theoretical challenges. In R. Greenwood, C. Oliver, T. B. Lawrence, & R. E. Meyer (Eds.), The Sage handbook of organizational institutionalism (2nd ed., pp. 79–104). Sage. Chaplowe, S. G., & Engo-Tjega, R. B. (2007). Civil society organizations and evaluation: Lessons from Africa. Evaluation, 13(2), 257–274. Cram, F., & Chouinard, J. A. (2023). Culturally responsive indigenous evaluation. An indigenous journey through culturally responsive approaches. In M. Alkin, & C. Christie (Eds.), Evaluation roots: Theory influencing practice (pp. 145–158). Guilford Press. Cullen, A., & Coryn, C. L. (2011). Forms and functions of participatory evaluation in international development: A review of the empirical and theoretical literature. Journal of MultiDisciplinary Evaluation, 7 (16), 32–47. Dahler-Larsen, P., & Boodhoo, A. (2019). Evaluation culture and good governance: Is there a link? Evaluation, 25(3), 277–293. Dewey, J. (1938). Logic. The theory of inquiry. Holt, Rinehart and Winston. Dewey, J. (1980). Democracy and education. In J. A. Boydston (Ed.), John Dewey: The middle works 1899–1924: Vol 15, 1923–1924 (pp. 180–189). Southern Illinois University Press. Dewey, J. (1988). Experience and education. In J. A. Boydston (Ed.), John Dewey: The later works 1925–1953: Vol 13, 1938–1939 (pp. 1–62). Southern Illinois University Press. Diamond, L. (2002). Elections without democracy: Thinking about hybrid regimes. Journal of Democracy, 13(2), 21–35. DiMaggio, P. J., & Anheier, H. K. (1990). The sociology of nonprofit organizations and sectors. Annual Review of Sociology, 16, 137–159. Dongre, Y. (2020). Agricultural cooperatives in Asia: State, market, governance, and sustainability. In M. Altman, A. Jensen, A. Kurimoto, R. Tulus, Y. Dongre, & S. Jang (Eds.), Waking the Asian Pacific co-operative potential (pp. 195–200). Academic Press. Easterby-Smith, M., Araujo, L., & Burgoyne, J. (Eds.). (1999). Organizational learning and the learning organization: Developments in theory and practice. Sage. Ebrahim, A. (2019). Measuring social change: Performance and accountability in a complex world. Stanford University Press. EIU. (2021). Democracy index 2021. The China challenge. Economist Intelligence Unit. The Economist Group. Elkjaer, B. (2021). The learning organization from a pragmatist perspective. The Learning Organization, 28(1), 59–70.

590

R. STOCKMANN ET AL.

Frolic, B. M. (1997). State-led civil society. In T. Brook, & M. Frolic (Eds.), Civil society in China (pp. 46–67). Routledge. Furubo, J.-E., Rist, R. C., & Sandahl, R. (Eds.). (2002). International Atlas of evaluation. Transaction Publishers. Ghosh, B. (2009). NGOs, civil society and social reconstruction in contemporary India. Journal of Developing Societies, 25(2), 229–252. Gillberg, C., & Vo, L. C. (2014). Contributions from pragmatist perspectives towards an understanding of knowledge and learning in organisations. Philosophy of Management, 13, 33–51. Grimmelikhuijsen, S., John, P., Meijer, A., & Worthy, B. (2018). Do freedom of information laws increase transparency of government? A replication of a field experiment. Journal of Behavioral Public Administration, 1, 1–10. Grundmann, R. (2001). Organisations, networks, and learning: A sociological view. In O. Jones, S. Conway, & F. Steward (Eds.), Social interaction and organisational change. Aston perspectives on innovation networks (Vol. 6, pp. 251–286). Series on Technology management. Imperial College Press. Holmqvist, M. (2003). A dynamic model of intra-and interorganizational learning. Organization Studies, 24(1), 95–123. Huntington, S. P. (1993). The third wave: Democratization in the late twentieth century (Vol. 4). University of Oklahoma Press. Jacob, S., Speer, S., & Furubo, J. E. (2015). The Institutionalization of evaluation matters: Updating the international atlas of evaluation 10 years later. Evaluation, 21(1), 6–31. Kang, J., Anderson, S. G., & Finnegan, D. (2012). The evaluation practices of US International NGOs. Development in Practice, 22(3), 317–333. Kaviraj, S., & Khilnani, S. (Eds.). (2001). Civil society: History and possibilities. Cambridge University Press. Kim, S. (2000). The politics of democratization in Korea: The role of civil society. University of Pittsburgh Press. Knight, L. (2002). Network learning: Exploring learning by interorganizational networks. Human Relations, 55(4), 427–454. Kocka, J. (2006). Civil society in historical perspective. In J. Keane (Ed.), Civil society. Berlin perspectives (Vol. 2, pp. 37–50). Berghahn Books. Ku-Hyun, J., & Inchoon, K. (2001). Republic of Korea. In T. Yamamoto, & K. Gould Ashizawa (Eds.), Governance and civil society in a global age (pp. 33– 65). Japan Center for International Exchange. Lauth, H. J., Pickel, G., & Pickel, S. (2014). Vergleich politischer Systeme. Eine Einführung. Verlag Ferdinand Schöningh. Llanos, A., Raven, R., Bexell, M., Botchwey, B., Bornemann, B., Censoro, J., Christen, M., Díaz, L., Hickmann, T., Jönsson, K., Scholz, I., Scobie, M., Sun, Y., Thompson, J., Thwaites, J., & Yunita, A. (2022). Implementation at multiple levels. In F. Biermann, T. Hickmann, & C. Sénit (Eds.), The political

16

SYNTHESIS

591

impact of the sustainable development goals: Transforming governance through global goals? (pp. 59–91). Cambridge University Press. Merkel, W. (2016). Eingebettete und defekte Demokratien. In O. W. Lembcke, C. Ritzi, & G. S. Schaal (Eds.), Zeitgenössische Demokratietheorie (pp. 455– 484). Springer VS. Mertens, D. M. (1998). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches. Sage. Meyer, W., & Zierke, N. (2022). Evaluationsdesigns. In R. Stockmann (Ed.), Handbuch zur Evaluation. Eine praktische Handlungsanleitung. (2nd ed., pp. 239–285). Waxmann. Meyer, W. (2016). The global state of evaluation as a profession. Some results (EES16–0161). Paper presented at the European Evaluation Society Conference in Maastricht. Meyer, W., et al. (2018, January). VNR reporting needs evaluation: A call for global guidance and national action (IIED-Briefing). https://www.iied.org/ 17446iied. The International Institute for Environmental and Development (IIED). Meyer, W., Stockmann, R., & Szentmarjay, L. (2022). The institutionalisation of evaluation. Theoretical background, analytical concept and methods. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 3–32). Palgrave Macmillan. Meyer, W., Stockmann, R., & Taube, L. (2020). The institutionalisation of evaluation. Theoretical background, analytical concept and methods. In R. Stockmann, W. Meyer, & L. Taube (Eds.), The institutionalisation of evaluation in Europe (pp. 3–34). Palgrave Macmillan. Mitchell, G. E. (2014). Why will we ever learn? Measurement and evaluation in international development NGOs. Public Performance & Management Review, 37 (4), 605–631. Moon, J. A. (1999). Reflection in learning and professional development: Theory and practice. Routledge. Mueller-Hirth, N. (2012). If you don’t count, you don’t count: Monitoring and evaluation in South African NGOs. Development and Change, 43(3), 649– 670. O’Leary, S. (2017). Grassroots accountability promises in rights-based approaches to development: The role of transformative monitoring and evaluation in NGOs. Accounting, Organizations and Society, 63, 21–41. Pohlmann, M. (2023). Soziologie der Organisation. Eine Einführung (3rd ed.). UTB/UVK. Putnam, R. D. (1993). Making democracy work: Civic traditions in modern Italy. Princeton University Press. Putnam, R. D. (2000). Bowling alone: The collapse and revival of American Community. Simon and Schuster.

592

R. STOCKMANN ET AL.

Rogers, P., & Davidson, E. J. (2013). Australian and New Zealand evaluation theorists. In M. Alkin (Ed.), Evaluation roots: A wider perspective of theorists’ views and influences (pp. 371–385). Sage. Sager, F., & Mavrot, C. (2021). Participatory vs. expert evaluation styles. In M. Howlett, & J. Tosun (Eds.), The Routledge handbook of policy styles (pp. 395– 407). Routledge. Salamon, L. M., & Anheier, H. K. (1998). Social origins of civil society: Explaining the nonprofit sector cross-nationally. Voluntas: International Journal of Voluntary and Nonprofit Organizations, 9(3), 213–248. Salamon, L. M., Sokolowski, S. W., & Haddock, M. A. (2017). Explaining civil society development: A social origins approach. JHU Press. Schwandt, T. (2019). Post-normal evaluation? Evaluation, 25(3), 317–329. Silliman, G. S., & Noble, L. G. (Eds.). (1998). Organizing for democracy: NGOs, civil society, and the Philippine state. University of Hawaii Press. SPI. (2022). Social progress imperative. Social progress index. Social Progress Imperative. www.socialprogress.org. Accessed on 15 March 2023. Stockmann, R., Meyer, W., & Taube, L. (2020). The institutionalisation of evaluation in Europe: A synthesis. In R. Stockmann, W. Meyer, & L. Taube (Eds.), The institutionalisation of evaluation in Europe (pp. 483–522). Palgrave Macmillan. Stockmann, R., & Meyer, W. (2022). The institutionalisation of evaluation in the Americas: A synthesis. In R. Stockmann, W. Meyer, & L. Szentmarjay (Eds.), The institutionalisation of evaluation in the Americas (pp. 451–507). Palgrave Macmillan. Stockmann, R. (Ed.). (2024). A practitioner handbook on evaluation. Edward Elgar. Tak-Wing, N. (1993). Civil society and political liberalization in Taiwan. Bulletin of Concerned Asian Scholars, 25(1), 3–16. Tien, H. M., & Chu, Y. H. (1996). Building democracy in Taiwan. The China Quarterly, 148, 1141–1170. UN. (2015). Transforming our world: The 2030 agenda for sustainable development. Resolution 70/1, adopted on 25 September 2015. United Nations. UN. (2022). World population prospects 2022. United Nations. Department of Economic and Social Affairs, Population Division. Online Edition: https:// population.un.org/wpp. Accessed on 7 November 2022. Vestman, O., & Conner, R. (2006). The relationship between evaluation and politics. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), Handbook of evaluation: Policies, programs, and practices (pp. 225–243). Sage. Wagner, A. (2012). ‘Third sector’ and/or ‘Civil society’: A critical discourse about scholarship relating to intermediate organisations. Voluntary Sector Review, 3(3), 299–328.

16

SYNTHESIS

593

Wahlén, C. B. (2014). Constructing conservation impact: Understanding monitoring and evaluation in conservation NGOs. Conservation and Society, 12(1), 77–88. Wnuk-Lipinski, ´ E. (2007). Civil society and democratization. In R. J. Dalton, & H.-D. Klingemann (Eds.), The Oxford handbook of political behavior. Oxford University Press. World Bank. (2021). GNI per capita, Atlas Method (current US$). https:// data.worldbank.org/indicator/NY.GNP.PCAP.CD. Accessed on 8 November 2022. Yeung, Y. M. (Ed.). (2002). New challenges for development and modernization: Hong Kong and the Asia Pacific Region in the new millennium. Chinese University Press.

Correction to: Evaluation in Australia Brad Astbury and Scott Bayley

Correction to: Chapter 2 in: R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_2 The original version of Chapter 2 was inadvertently published with errors, caused during production. These errors have now been corrected.

The updated version of this chapter can be found at https://doi.org/10.1007/978-3-031-36918-6_2 © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 R. Stockmann et al. (eds.), The Institutionalisation of Evaluation in Asia-Pacific, https://doi.org/10.1007/978-3-031-36918-6_17

C1