Skip to main content

Our confidence in our recommendations

It is important to understand that all research and recommendations have some uncertainty associated with them. That is the nature of research.

For example, Which? magazine may recommend some model of camera, but if you buy a camera of that model, there may happen to be a problem with the particular one that you buy – which obviously the manufacturer should resolve. In other words, even an independent recommendation is not a rock-solid guarantee that the recommended item will be perfect.

With evidence-based charity recommendations, the situation is more complicated. This is because interpreting research is complicated. We’ll explain, but strap yourself in because this is complicated territory.

Suppose that there is a rigorous evaluation of a programme which distributes anti-malarial bednets, and it finds that they are highly effective.

Great.

It’s important to appreciate that that evaluation was done in a particular place, at a particular time, and it evaluated a programme delivered in a particular way by a particular organisation. That evaluation does not guarantee that distributing anti-malarial bednets will will get the same results in different places or at different times or if delivered in a different way and/or delivered by a different organisation. That is true even if the evaluation was done perfectly and its results are reliable in the time and place of the study.

For instance, suppose that the evaluation was done in Place A where malaria is rife, and the population knows about malaria and about bednets so the programme could just dish out bednets and people would know what to do. Suppose that the evaluation found that the bednet programme reduces cases of malaria by 20%. Now, suppose that we are thinking about running that programme in Place B, where, as it happens, people don’t know about bednets. In that case, the original approach of just dishing out bednets might not achieve much in Place B: it would just give people something that they don’t know how to use. That programme would achieve much less in Place B than in Place A. In the language of research, the results from Place A do not generalise to Place B.

Here’s a real example. A programme in Tamil Nadu in India (the Tamil Nadu Integrated Nutrition Project) sought to improve nutrition in pregnant women and young babies. It gave nutritional advice and healthy food to pregnant women and new mums. That worked. But when that approach was tried in Bangladesh, it didn’t work, partly because many households there work differently: often the husband’s mother is very involved, and it is she – rather than the wife / new mum – who determines who gets what food. So the results from Tamil Nadu did not generalise to Bangladesh.*

This doesn’t mean that the study is flawed

Note that the fact that a result does not generalise to another time or place is not a flaw of the study design. Rather it reflects differences in reality. The same issue arises even in physics: if you do a great experiment to measure the strength of gravity in Singapore, you will get a different answer to when you do that experiment in Stockholm. That isn’t a flaw in your experiment but rather because the strength of gravity varies from place to place (because the Earth isn’t spherical).

Figuring out whether a programme evaluated in one place and at one time would achieve the same result in a second (specified) place and time – whether the results would ‘generalise’ to that place and time – is complicated and there are whole books about how to do it (e.g., this is a good one).

How the Good Giving List deals with this uncertainty

We use independent, high-quality research about the effectiveness of programmes: each study was done at a particular time and a particular place. There is therefore unavoidably some uncertainty about the extent to which the findings of those studies apply to different places and/or different times. We try to reduce this uncertainty as much as possible. For instance:

  • In criminal justice, we use studies produced by the Ministry of Justice Data Lab in 2017 or later. With studies published before that, there seems to us too high risk of the results no longer being reliable. For instance, the role of prisons or the prison service may have changed.
  • A major reason that we use studies by the Education Endowment Foundation and the Ministry of Justice Data Lab is that they both evaluate that programme as delivered by that organisation: there is no transfer of the programme from one organisation to another. That eliminates a lot of uncertainty.

*Source: Evidence-based policy, Nancy Cartwright & Jeremy Hardie, OUP 2012, p82.

 

Close Menu