Shiva Sharma, MPP, Staff Writer, Brief Policy Perspectives

Policymaking is an inherently complex process because it strives to address problems that are grounded in changing social realities; this makes evidence collection difficult at best. Evidence-based policymaking is gaining prominence due to increased availability of data, new investments in rigorous research, and a growing focus on program results. Various methodological tools are at the disposal of policymakers and policy evaluators to assess the effects of programs, which subsequently help to shape new and existing programs with distinct policy designs. Thus, it is essential for policymakers to discern the goodness of evidence.

As students of public policy and public administration, it is imperative that we understand the strengths and limitations of different methodological traditions. Understanding these will allow us to evaluate the type and quality of evidence used in policymaking.

Types of Evidence

Evidence-based policymaking stresses the importance of rigorous evaluation of programs and their effectiveness. For instance, impacts of programs like a job training program or an educational program can be assessed by measures like increased wage or improved literacy. The evaluation process involves the use of different methodologies (such as randomized control trials, natural experiments, and mixed methods) to gauge these impacts.

A study can be limited by starting out the evaluation process with a prior methodology before framing the research questions. For instance, the evaluation of self-help groups in the study, “Women’s participation in self-help groups as a pathway to women’s empowerment: A case of Nepal” uses a regression analysis methodology to assess women’s empowerment as a function of working status, asset ownership, education status, and decision-making autonomy in the household. In other words, the regression analysis assesses women’s empowerment by using certain measurable, quantifiable variables. I want to direct attention towards the type of evidence that was used – empowerment as a function of these quantifiable measures. Although crucially important and necessary, these measures are not sufficient for analyzing the impacts of self-help groups on women’s empowerment. To explore that question, for example, we might also want to look at women’s identity-formations and changes in their self-perceptions through a case study or ethnography.

Good evidence consists of a body of different approaches to a particular question. As policymakers concerned with improving policy designs, it is important that we do not distinguish evidence by labeling it as “qualitative” or “quantitative.” Instead, we need to understand how different methodologies enable us to ask and answer different forms of questions. A diverse body of evidence postulates new theories that make causal links clearer. For instance, the field of comparative economic development used to be guided by the measurement of per capita  GNI, until the Nobel laureate Amartya Sen argued for a broader framework – the capability approach – to more fully assess poverty and development.

Martin Ravallion makes a similar argument with respect to randomistas – proponents of randomized control trials (RCT). He argues that evaluation should not be monopolized by any one method.  Conducting an RCT is an effective way to assess a program. However, RCTs cannot be the only evidence used to formulate policies because they do not always point towards a theory of change. For instance, in a teacher performance pay experiment in South India, the authors conclude that pay incentives improve teacher performance (as quantified by the increase in student test scores). However, this experiment should not be the basis for creating incentive payment policies in all Indian schools. The RCT evidence could be biased because teachers might behave very differently under a time-bound and monitored experiment: long-term human behavior can be different from behavior that is subject to a controlled environment. Interviews with teachers could help translate the evidence by sketching out a theory of change. Additional evidence helps make the results of RCTs more generalizable.

Quality of Evidence

The type of evidence being used is often assumed to have a particular quality. Alan Peshkin argues that research which is not theory-driven, hypothesis-testing, or generalization-producing may be dismissed as deficient. However, the quality of evidence should be determined on a case-by-case basis. For instance, if one is assessing the quality of a regression discontinuity model, then one should focus on that model’s assumptions, such as presence of a discontinuity, randomness of errors, or plausibility of omitted variable bias. On the other hand, if one is evaluating the quality of a case study research method, it should be assessed within the epistemology of the case study methodology, like the bounding of the case and triangulation. In other words, the quality of evidence should not be determined by using a hierarchical valuation of the methodology, but by the goodness and rigor of the methodological framework.

To summarize, evidence in policymaking creates knowledge reservoirs to inform policy design decisions. A diverse body of evidence consisting of different perspectives would not only deepen understanding of existing theories but help suggest new theories. Using a diverse body of knowledge, such as randomized control trials, case studies, ethnographies, phenomenology, and interview surveys brings about a comprehensive picture of reality which is deeply rooted in a complex and changing social, economic, and political context.