The Alliance for Useful Evidence1 is an open–access network that was launched in October 2011 with the aim of enhancing both the “demand for and supply of evidence for social policy and practice”. The Alliance comprises of 1,600 individuals from government agencies, universities, charities, business and local authorities in the UK and globally. What really though is the role of evidence in policy design? The key questions for policy designers include what constitutes evidence, who generates evidence, who uses it, how they use it, and how easy is it for users to understand evidence provided to them? Also, how does one ensure that users do not ‘cherry pick’ evidence to their advantage or to support policies or actions that were to be taken anyway (sometimes alluded to as ‘policy-based evidence’ when research findings are used to support premeditated policies). The Alliance organized a seminar in 2012 inviting experts to provide their insights on “What is ‘good’ evidence” (see video link here2)
Efforts towards enhancing the quality and size of the evidence base is based on the expectation that better evidence equates to ‘better policies’ and policy solutions. However in the real-world policymakers are limited by many factors including multi-stakeholder perspectives that challenge the design and adoption of policies solely based on evidence (Head, 2010)3. Evidence notwithstanding, has some limitations of its own. A recent editorial in Nature by Sutherland et al (2013)4 present “Twenty tips for interpreting scientific claims”, with the intention of aiding ‘non-scientists’ and policymakers to understand the “limitations of evidence” to be used in policymaking. These twenty tips include:
1. Variations in research can be caused by chance and a variety of natural factors including the factor or process that we are interested in studying. The challenge thus is to identify variation attributable to the factor/ process of interest alone.
2. Measurement errors are likely in all analysis, and need to be acknowledged.
3. Experiments are susceptible to biases from the subject and/or the investigator despite their best attempts in keeping neutrality.
4. Larger observation samples often provide more information and help subdue “natural variation and measurement error”
5. The possibility of correlation between two factors occurring due to chance or owing to outside or confounding factors should be considered.
6. Extreme patterns in data can also be sometimes attributable to chance rather than direct causation to explanatory variable.
7. Extrapolation or generalization of trends emerging from a set data to a range outside it may not necessarily be valid.
8. Chances of a ‘base-rate fallacy’, which suggests that “the ability of an imperfect test to identify a condi¬tion depends upon the likelihood of that condition occurring (the base rate)”, need to be considered.
9. Comparison of outcomes with a ‘control’ group is important.
10. Randomization of the sample avoids bias.
11. While attempts towards replication of results are important, ‘pseudo-replication’ i.e. replication in very similar settings should be discouraged in order to avoid false impression of large-scale applicability of the research methods/interventions.
12. Mul¬tiple, independent sources of evidence and replication are more substantial.
13. Statistical p values form an important parameter to assess the likelihood of the results to have been obtained purely by chance.
14. Not significant p values are different from ‘no effect’ being detected, which in turn may be a result of a small sample size.
15. Sometimes ‘effect size’ or the strength of a phenomenon might be more important than its statistical significance, especially in biological, physical and social sciences.
16. The relevance of the study to specific contexts limits its generalizations and wider applicability.
17. Feelings influence the perception of risk, and thus it can be over and under-estimated.
18. Inter-dependencies between various variables can also change the risk patterns.
19. Data can be ‘cherry picked’ i.e. chosen to reflect in a way that is to one’s advantage. Sutherland et al suggest that the question to be asked here is: ‘What am I not being told?’
20. “Extreme measurements may mislead” and should be considered with a close assessment of the various direct and indirect factors that may have influenced the results, before establishing a direct causation between two variables.
On the question of what constitutes good evidence, there is no easy answer. It depends on the question, purpose of enquiry, and the context in which the evidence is to be used. Nutley et al (2013)5 from the Alliance team argue that data becomes information when it “changes views” and further becomes evidence only when it generates action on its basis. All this however is rather subjective, owing to which developing standards of evidence is anything but an easy task. While developing such standards it is important to be cognizant of the various types of evidence. Brechin and Sidell (2003)6 for example suggest three broad categories of evidence- generated through empirical research, through theory or through experience. Evidence based on experiments can sometimes become a contentious issue as compared to that generated by observational research. In addition the quality of knowledge and the information evidence generates and the diversity of viewpoints the evidence would be subject to from the user’s perspective also needs to be considered while setting standards of evidence. Nutley et al suggest that evidence cannot be considered to be an end in itself as the evidence base is rather dynamic and evolving- a point that warrants attention in the midst of encouraging the setting of ‘rigid’ standards of evidence.
- http://www.alliance4usefulevidence.org/ [↩]
- http://www.alliance4usefulevidence.org/event/what-is-good-evidence-standards-kitemarks-and-forms-of-evidence-2/ [↩]
- Head, B., 2010. Reconsidering evidence-based policy: Key issues and challenges. Policy and Society 29, 77–94 [↩]
- Sutherland, W. J., and Spiegelhalter, D. and Burgman, M., 2013. Twenty tips for interpreting scientific claims. Nature 503, 335–337 [↩]
- Nutley, S., Powell, A. and Davies, H., 2013. What counts as good evidence? Provocation paper for the Alliance for useful evidence. United Kingdom [↩]
- Brechin, A. and Siddell, M. (2000) ‘Ways of knowing’, In Gomm, R. and Davies, C. (eds) ‘Using evidence in health care.’ Buckingham: Open University Press [↩]