Justice Data Lab

To assess the effect of a programme, the JDL works out:

• the actual one-year proven re-offending rate for the people with whom an organisation works (ie, how many of them re-offended within 12 months of release). They are called the treatment group. JDL does this by looking at records on the Police National Computer;
• the one-year proven re-offending rate for a group of offenders similar to those in the treatment group, but who did not receive the intervention, known as the counterfactual. This is needed to make a fair assessment of impact. A simple comparison with national average figures for re-offending would not be valid because the composition of offenders in the treatment group may be very different to the average offender; and
• a statistical comparison between the two groups, which allows it to assess whether the organisation’s intervention has led to a change in re-offending rates.  

The counterfactual is a group of offenders similar to those in the treatment group. JDL creates a matched controlled group using a technique known as Propensity Score Matching (PSM). The goal is that the only major difference between the offenders in the treatment group and those in the control group should be that the former group received the intervention and the other did not.  This ‘isolates’ the effect of the intervention, which gives some confidence about how the intervention affected re-offending. Creating a matched control group allows JDL to control for other factors that might influence re-offending (if data is available on those factors), e.g., criminal history. 

There may still be other factors involved not covered by data held by the Ministry of Justice. Despite efforts to include all observed factors known to predict whether a person is selected onto an intervention and their risk of re-offending, there may be other important factors about which information is not recorded. It is therefore possible that unobserved factors could influence the results.

The final stage is that JDL compares the one-year proven re-offending rates of the treatment group to that of the control group using statistical significance testing.  This returns a statistical significance value (referred to as a ‘p’ value), which shows the level of confidence with which we can conclude that an intervention may have had a real impact on re-offending. A ‘p value’ of 0.05 (5%) means that we can be 95% confident that the difference was not down to chance.

By convention, a p-value of 0.05 or lower means that the results are considered statistically significant, and we can conclude that an intervention may have led to a change in re-offending. If the p-value is greater than 0.05, then there is insufficient evidence to confidently conclude that an intervention has affected re-offending. The lower the p-value, the more confidence we can have the intervention affected re-offending.

For each of its analyses, JDL says whether its analysis is statistically significant or not. This website only uses the analyses which are statistically significant. 

You can read about the JDL’s method in more detail here.