No method in the ‘troubled families’ madness

DCLG published a couple of documents yesterday relating to the alleged ‘savings’ accrued to ‘the taxpayer’ from the Troubled Families Programme. I’m going to try and do a couple of blogs on different aspects of these publications and the ‘news’ contained within them, and this first one is going to look at the methodology behind the reports.

It would be churlish not to acknowledge that the publication of a methodology report represents something of a symbolic step forward, considering the contempt that has often been shown to methodology in other aspects of the programme, not least the identification of the 120,000 families in the first case. But that’s kind of where the good news ends.

So, firstly, the report The Benefits of the Troubled Families Programme to the Taxpayer and accompanying press release, which highlighted the estimated £1.2 billion saving from the Troubled Families Programme (TFP), were based on data collected from 7 of the 150 plus local authorities involved in the programme. The methodology document states that ‘the results reported are not from the national programme as a whole’ (p4) but this is not made clear in the press release which accompanied the report. We are also told that, of these seven, ‘five local authorities used a sample, while two local authorities collected data on their full cohort of families’ (p4). We are also informed that:

‘The data collection methods used represents a significant step forward in evaluating costs and benefits at local authority level. Local authorities sought to use the best available data and to maximise the representativeness of their local samples. However, some samples may not be entirely representative’ (p4).

But we are never actually told what the sampling procedure was, which was a similar problem with the data accompanying the DCLG Understanding Troubled Families report.

Read on a little further and we are told that ‘care should be taken when directly comparing results between local authorities’ (p4) which is, again, a familiar concern with data collected under the TFP. The DCLG press release which featured comments from Eric Pickles, Danny Alexander and Eric Pickles didn’t appear to exhibit much in the way of ‘care’ when it triumphantly proclaimed ‘the new figures showed that the programme had already saved taxpayers an estimated £1.2 billion’. Neither the report nor the methodology document actually mention the figure of £1.2 billion saved, but that didn’t stop the DCLG press ignoring the advice of its own report by extrapolating estimated costs from seven local authorities and applying the same flawed logic to over 150.

These are not official statistics that we are dealing with and local authorities have used their own ‘local administrative systems’ to collect data. Where this has not been possible ‘local authorities have used data provided by key workers’ (p4). So this ‘significant step forward’ includes using data that local authorities already had and asking a few workers to complete some forms – which could be used to help justify or jeopardize their precarious jobs – when they have no other way of finding information.  The report then goes on to acknowledge that ‘data collection limitations have prevented local authorities recording information on every measure’ (p4).One local authority managed to collect information for only 14 of 39 indicators and most achieved around between 22 and 29 out of 39.

As Jonathan Portes pointed out on Twitter, the fact that no impact assessment has been carried out shows that the governments assertions about ‘savings’ are ‘largely meaningless’. The methodology also notes that whilst the cost savings calculator does use ‘the best available ‘deadweight’ estimates’ (p5) to understand what might have happened without the TFP intervention, the report itself deals only with ‘gross’ figures and therefore assumes that all of the ‘savings’ across all of the families can be attributed to the TFP intervention. Which is quite ludicrous really. But then the whole thing is…

Best wishes,

Steve

As I said at the top of this post, I’ll hopefully do more blogs on this in the next couple of days on the language used within the main report and the lack of reference to previous ‘research’ and analysis on the ‘cost’ of ‘troubled families’.