‘What is an official statistic?’ asked Paul Spicker when the Troubled Families Programme was launched, as he queried the number of 120,000 ‘troubled families’ and the alleged cost of services to these families. Spicker highlighted that these figures were not ‘official statistics’ and were therefore outside of the remit of the UK Statistics Authority, bu that their use by government ‘opens the door to the possibility that there will be two sorts of official statistics – the formal sort, which meet professional standards, and the others which don’t’ (p51).
The recent publication in July of the DCLG report Understanding Troubled Families and an accompanying ‘interim report’ of the evaluation of the programme provide the latest instalment of the ‘troubled families’ numbers game. Despite, Louise Casey writing in the foreword to the report that ‘These are statistics to be concerned about’ (p4) and telling the Sunday Times that ‘this is the first time we have been able to evidence the extent of the problems’ (emphases added), the ‘Family Monitoring Data’ which supports the report falls into the ‘unofficial’ statistics group, probably doesn’t deserve to be called ‘evidence’ and and doesn’t really meet any professional standards at all.
The data was collected by local authorities. We do know that although ‘local authorities were asked to randomly select the sample of families for monitoring purposes, it is not possible to be certain that families were chosen randomly in all cases’ (p9). As Declan Gaffney pointed out on Twitter, it should have been possible to provide a randomisation procedure to all local authorities to ensure some kind of consistency, but only some suggestions on how to provide a random sample were provided – see p4 of this.
It would appear from the interim evaluation report that local authority key workers provided the data via a standardised template. However, the interim report also states that ‘some data submitted will be local authorities ‘best fit’ to data requested’, that there are ‘many different data collection systems in place, some of which were not capable of collecting sufficient data in time for this return’ in local authorities and that some local authorities are involved with ‘retrospective data gathering’ (p5, emphasis added). So far, not so good. The use of key workers to provide the data is also interesting, and not without complication. The indicators ‘combine both recognised standard measures and those which rely on practitioner intelligence’ (p4). In other words, a lot of the information submitted may not have been provided by families themselves or from information held about them by public bodies; some of the information has been provided based on workers perceptions of these families. Some of the indicators which rely on ‘practitioner intelligence’ include:
- ‘Parenting difficulties’
- Family members suffering from Domestic Violence or Domestic Abuse
- Adults dependent on non-prescription drugs
- Adults suffering mental health problems
- Children suffering mental health problems
- Adults dependent on alcohol
- Adults with long-standing illness/disability
- Children with long-standing illness/disability
(The last five of these bullet points are provided as an ‘alternative’ to a clinical diagnosis)
There are numerous reasons why the information provided in response to these indicators may not be particularly robust. No explanation is given as to what constitutes ‘parenting difficulties’, even in the guidance document sent to local authorities, so the information provided for this indicator is entirely subjective. There are good reasons why individuals or families may want to hide evidence of Domestic Violence or Abuse (a point which is recognised in the report) and quite how key workers with presumably little medical training can be expected to provide an assessment of mental health, illness/disability or a drug or alcohol dependency is not made clear. Again, there are good reasons why individuals or families may seek to hide this information from key workers, but there is also potential for some key workers to overstate the prevalence of these problems in these ‘troubled families’.
When David Cameron launched the programme he said he ‘wanted to be clear’ what he meant by the term ‘troubled families’. He went on to describe some of the issues they face: ‘Drug addiction. Alcohol abuse. Crime. A culture of disruption and irresponsibility that cascades through generations’ (emphasis added). Louise Casey has written in the Listening to Troubled Families report that ‘Violence appears in many cases to be endemic – not just domestic violence between parents but violence between siblings, between parent and child, outside the house and inside the house. Violence, verbal and physical abuse was described in an almost matter-of-fact way. (p2 emphasis added)
So it is entirely possible that key workers are looking for issues which might not be there, or expecting to find them when they are working with ‘troubled families’ – their senses are heightened, if you like, and this is not a good way of collecting information. Interestingly, even when subjective submission are taken into account, the majority of families are not characterised by drug addiction, alcohol abuse, crime or Domestic Violence at all.
There are many other examples of why the data shouldn’t be trusted, and certainly isn’t worthy of presentation and discussion in official government documents. It would appear that at least 10 families were involved in the programme when they didn’t meet the criteria for being labelled as ‘troubled’ as they experienced 2 or less issues (the criteria for inclusion in the TF programme requires at least three criteria to be met). There are low completion rates (with a high number of ‘blank’ returns) for some indicators, with, for example, only two of the 16 possible health indicators receiving over 50% completion rates. One local authority provided entry data that was entered entirely in error which resulted in this information being excluded from the analysis. In a section of the interim evaluation report which explores the characteristics of the families, we find out that in 10% of cases, no information was provided regarding the number of children under the age of 5 in each family. We also find out that nearly 30% of responses regarding the families’ ethnicity were blank and a further 10% of responses to this indicator were in the ‘not known’ box. Finally, no information is provided on the local filter criteria used by local authorities in targeting ‘troubled families’ in their local areas, so the data could simply be a reflection of the characteristics of families that local authorities have decided to work with via the TF programme.
None of these issues highlighted above suggest a particularly sound method and nor do they provide much in the way of quality assurance. In short, the data provided probably shouldn’t have been published by the government in the way that it has been, and it certainly shouldn’t have been used as justification for the expansion of the programme. In the press release accompanying the report, Louise Casey said ‘this data also shows how big the challenge is and why we need to take this approach to a wider group of families with a wider set of problems as soon as we can’. As Ruth Levitas has pointed out previously in relation to the the monitoring and progress information provide by local authorities in the TF programme ‘Neither the quality of these statistics, nor the political claims made for them, bear scrutiny’.
The ‘political claims’ or representation of the data is the subject of Part 2 of this series, to be published Tuesday 2nd September 2014.