(Mis)Understanding Troubled Families (Part 1)

‘What is an official statistic?’ asked Paul Spicker when the Troubled Families Programme was launched, as he queried the number of 120,000 ‘troubled families’ and the alleged cost of services to these families. Spicker highlighted that these figures were not ‘official statistics’ and were therefore outside of the remit of the UK Statistics Authority, bu that their use by government ‘opens the door to the possibility that there will be two sorts of official statistics – the formal sort, which meet professional standards, and the others which don’t’ (p51).

The recent publication in July of the DCLG report Understanding Troubled Families and an accompanying ‘interim report’ of the evaluation of the programme provide the latest instalment of the ‘troubled families’ numbers game. Despite, Louise Casey writing in the foreword to the report that ‘These are statistics to be concerned about’ (p4) and telling the Sunday Times that ‘this is the first time we have been able to evidence the extent of the problems’ (emphases added), the ‘Family Monitoring Data’ which supports the report falls into the ‘unofficial’ statistics group, probably doesn’t deserve to be called ‘evidence’ and and doesn’t really meet any professional standards at all.

The data was collected by local authorities. We do know that although ‘local authorities were asked to randomly select the sample of families for monitoring purposes, it is not possible to be certain that families were chosen randomly in all cases’ (p9). As Declan Gaffney pointed out on Twitter, it should have been possible to provide a randomisation procedure to all local authorities to ensure some kind of consistency, but only some suggestions on how to provide a random sample were provided – see p4 of this.

It would appear from the interim evaluation report that local authority key workers provided the data via a standardised template.  However, the interim report also states that ‘some data submitted will be local authorities ‘best fit’ to data requested’, that there are ‘many different data collection systems in place, some of which were not capable of collecting sufficient data in time for this return in local authorities and that some local authorities are involved with ‘retrospective data gathering’ (p5, emphasis added). So far, not so good. The use of key workers to provide the data is also interesting, and not without complication. The indicators ‘combine both recognised standard measures and those which rely on practitioner intelligence’ (p4). In other words, a lot of the information submitted may not have been provided by families themselves or from information held about them by public bodies; some of the information has been provided based on workers perceptions of these families. Some of the indicators which rely on ‘practitioner intelligence’ include:

  • ‘Parenting difficulties’
  • Family members suffering from Domestic Violence or Domestic Abuse
  • Adults dependent on non-prescription drugs
  • Adults suffering mental health problems
  • Children suffering mental health problems
  • Adults dependent on alcohol
  • Adults with long-standing illness/disability
  • Children with long-standing illness/disability

(The last five of these bullet points are provided as an ‘alternative’ to a clinical diagnosis)

There are numerous reasons why the information provided in response to these indicators may not be particularly robust. No explanation is given as to what constitutes ‘parenting difficulties’, even in the guidance document sent to local authorities, so the information provided for this indicator is entirely subjective. There are good reasons why individuals or families may want to hide evidence of Domestic Violence or Abuse (a point which is recognised in the report) and quite how key workers with presumably little medical training can be expected to provide an assessment of mental health, illness/disability or a drug or alcohol dependency is not made clear. Again, there are good reasons why individuals or families may seek to hide this information from key workers, but there is also potential for some key workers to overstate the prevalence of these problems in these ‘troubled families’.

When David Cameron launched the programme he said he ‘wanted to be clear’ what he meant by the term ‘troubled families’. He went on to describe some of the issues they face: ‘Drug addiction. Alcohol abuse. Crime. A culture of disruption and irresponsibility that cascades through generations’ (emphasis added). Louise Casey has written in the Listening to Troubled Families report that ‘Violence appears in many cases to be endemic – not just domestic violence between parents but violence between siblings, between parent and child, outside the house and inside the house. Violence, verbal and physical abuse was described in an almost matter-of-fact way. (p2 emphasis added)

So it is entirely possible that key workers are looking for issues which might not be there, or expecting to find them when they are working with ‘troubled families’ – their senses are heightened, if you like, and this is not a good way of collecting information. Interestingly, even when subjective submission are taken into account, the majority of families are not characterised by drug addiction, alcohol abuse, crime or Domestic Violence at all.

There are many other examples of why the data shouldn’t be trusted, and certainly isn’t worthy of presentation and discussion in official government documents. It would appear that at least 10 families were involved in the programme when they didn’t meet the criteria for being labelled as ‘troubled’ as they experienced 2 or less issues (the criteria for inclusion in the TF programme requires at least three criteria to be met). There are low completion rates (with a high number of ‘blank’ returns) for some indicators, with, for example, only two of the 16 possible health indicators receiving over 50% completion rates. One local authority provided entry data that was entered entirely in error which resulted in this information being excluded from the analysis. In a section of the interim evaluation report which explores the characteristics of the families, we find out that in 10% of cases, no information was provided regarding the number of children under the age of 5 in each family. We also find out that nearly 30% of responses regarding the families’ ethnicity were blank and a further 10% of responses to this indicator were in the ‘not known’ box. Finally, no information is provided on the local filter criteria used by local authorities in targeting ‘troubled families’ in their local areas, so the data could simply be a reflection of the characteristics of families that local authorities have decided to work with via the TF programme.

None of these issues highlighted above suggest a particularly sound method and nor do they provide much in the way of quality assurance. In short, the data provided probably shouldn’t have been published by the government in the way that it has been, and it certainly shouldn’t have been used as justification for the expansion of the programme. In the press release accompanying the report, Louise Casey said ‘this data also shows how big the challenge is and why we need to take this approach to a wider group of families with a wider set of problems as soon as we can’. As Ruth Levitas has pointed out previously in relation to the the monitoring and progress information provide by local authorities in the TF programme  ‘Neither the quality of these statistics, nor the political claims made for them, bear scrutiny’.

The ‘political claims’ or representation of the data is the subject of Part 2 of this series, to be published Tuesday 2nd September 2014.

Advertisements

Troubled Families: a ‘super’ ‘social problem’?

Last week, Simon Hughes, the Justice Minister suggested that families who repeatedly appeared in the family courts would, in the future, be helped by the government’s Troubled Families Programme. This is the latest in a long line of ‘presenting problems’ linked with ‘troubled families’ and is entirely consistent with Macnicol’s view that

‘it is necessary for proponents of the underclass concept to lump together a wide variety of diverse human conditions (in order to make the problem appear significant), yet attribute to them a single cause (so that it appears a problem amenable to solution)’ (Macnicol, 1987: 315).

So far, we have seen ‘troubled families’ associated with a remarkably long list of ‘troubles’. To add to the three national criteria of Crime/ASB, educational truancy or exclusion and worklessness, local authorities are invited to add local filter criteria which can be whatever they choose, as long as it represents a high cost to the public purse. A number of local authorities have identified issues such as child protection and/or domestic violence whilst some have also included slightly different – and disparate – criteria such as: living in a deprived neighbourhood; having a parent with a long-term limiting illness or disability; not taking up the offer of free childcare for two-year-olds or; having ‘low parental capacity’.

Louise Casey, in her Listening to Troubled Families report helpfully listed the problems that were ‘revealed’ by her interviews with 16 families:

  • Intergenerational transmission
  • Large numbers of children
  • Shifting family make-up
  • Dysfunctional relationships
  • The anti-social family and friends network
  • Abuse
  • Institutional care
  • Teenage mothers
  • Violence
  • Early signs of poor behaviour
  • School
  • Anti-social behaviour
  • Mental health – depression
  • Drugs and alcohol

As if these three national criteria, unlimited local criteria and 12 familial issues aren’t enough, other government publications and individuals have sought to extend and ‘diversify’ the potential impact of the Troubled Families Unit. In the aftermath of the riots of 2011, the government published a report on Ending gang and youth violence stating that the work of the Troubled Families Unit would be ‘crucial’ in reducing involvement in violent crime and disorder. No mention of gang membership in any national criteria or Casey’s report.

James Brokenshire, the Security Minister has stated that he is ‘keen to ensure that the Government’s work to support troubled families is aligned to our work to support vulnerable individuals at risk of being drawn into terrorist activity’. This concern has been operationalized by at least one local authority who have included a local criteria of ‘Family member is believed to have been influenced by violent extremism.’ Again, no mention of extremist activity in national criteria or Casey’s report.

This ‘lumping together’ (or ‘pathological concentration’ as Garland called it) of a number of different ‘social problems’ under the banner of ‘troubled families’ is akin to the development of some kind of ‘super social problem’. The label of ‘troubled family’ therefore becomes an official hook on which to hang whichever social problem is in the news at the time. But the diversity of criteria demonstrate that it is disingenuous to think of a homogenous group of ‘troubled families’ and even more absurd to think that a single policy response can ‘turn round’ the lives of all of the families being discussed. Does a family with a parent with a long term limiting illness require the same sort of ‘family intervention’ as a family with a member involved in extremist or gang-related activity? Where is the evidence base for ‘family intervention’ reducing extremism? By hanging everything on the ‘troubled families’ hook, ministers, like Hughes, feel able to justify cuts to services in other areas. Who needs a comprehensive range of public services when a single ‘family intervention’ model can ‘deliver’ across a number of different family and social policy areas….

References

Garland, D. (2002). The culture of control: Crime and social order in contemporary society. Oxford: Oxford University Press

Macnicol, J. (1987) In pursuit of the underclass, Journal of Social Policy, 16 (3), 293-318

Many thanks to Debbie Key, David McKendrick and @FearlessJones for information provided for this post.

‘High risk’ families and ‘decision-based evidence making’

On 24th June 2013, the government announced that the Troubled Families Programme was going to see a ‘massive expansion’ that would lead to an extra ‘400,000 high risk families’ receiving ‘intensive help’. There was a lack of further detail in the announcement, and so I sent a Freedom of Information request to DCLG to see if I could find out a bit more about the 400,000 newly identified families. The main bulk of the text from my request is below

… could you please provide me with the following information regarding the expansion of the Troubled Families Programme to include ‘high risk families’

  1. Information relating to how the figure of 400,000 high risk families was calculated –  including details of the methodology and any data sources used.
  2. Will the 400,000 high risk families be broken down into individual local authority areas as the Troubled Families have been and, if this has already happened, could you please provide the details of these numbers and how they were calculated.
  3. What the criteria for being identified as a ‘high risk family’ is. i.e. what is the definition of a high risk family?
  4. How many families the £200 million is expected be spent on. i.e. how much money per family does the government expect to be allocating to local authorities in 2015-2016.
  5. Clarify if the £200 million is ‘for’ 2015-2016 or is ‘available from’ 2015-2016. If it is the latter, could you please provide details of the term that the funding covers. (there is some ambiguity on the gov website regarding this)
  6. Details of the Payment By Results framework for high risk families. What outcomes will need to be achieved for councils to access the PBR programme and when will the government contribution be allocated?
  7. Information on the ‘new incentives for services such as the police, health and social work to work more closely together in order to reduce costs’ that are mentioned on the government website published on 24 June 2013 relating to the extra funding.

I received a response from DCLG just over a week ago and, unfortunately, it didn’t provide me with much extra detail. The response included clarification on point 5 – the £200 million funding is for the financial year 2015-2016, with further allocations presumably subject to future spending reviews

Unfortunately, that was the only real clarification I received. The response from DCLG states

In relation to the remaining questions, I can confirm that the Department does hold some of the information falling within the terms of your request.  However, it is being withheld because it is exempt from disclosure under section 35(1)(a) and (b) of the Freedom of Information Act.  This relates to the formulation or development of government policy and ministerial communications. 

The response went on to state that

the development of the detail of the expanded programme is still in its early stages and further work is required to ensure that the vision set out in the Government’s announcement is translated into a rigorous policy framework and delivery model.

Whilst I can appreciate that this may be the case for points 2, 4, 6 & 7, it seems a bit unsatisfactory when applied to points 1 and 3. The figure of 400,000 is used 6 times in the 2 ‘news story’ documents on the DCLG website promoting the expansion (here and here) and was widely reported in several newspapers at the same time. I would propose that his number appears to have already been formulated and, therefore, is no longer under development. If the number has been agreed (which, I would suggest, it obviously has) there must have been some methodology supporting it (point 1) and that methodology must have included criteria or a definition (point 3). You surely cannot count something unless you are clear about what it is you are trying to count.  Again, there were some references to the ‘troubles’ that families might have been experiencing and which might identify them as ‘high risk’  in both of the DCLG ‘news story’ documents. Danny Alexander stated that

Extending this intensive help to 400,000 more families will enable us to tackle problems such as truancy, anti-social behavior and crime

whilst Louise Casey said

It is great news that the momentum we have built up on the Troubled Families programme can continue by extending the approach to a wider group of families who, for example, are struggling with health problems or parenting, where their children are not in school or are at risk of being taken into care.

Of course, as the response from DCLG states, the government may well have this information, but have chosen not to release it because it “the public interest is best served by protecting the need for the necessary degree of internal discussion, during which suggestions can be made and considered” and “there is a powerful public interest in ensuring that there is space in which ministers and officials are able to discuss policy making and implementation”

All of this reminded me very much of a phrase used by Tom Slater (which he attributed to Rob Penfold), that of ‘decision-based evidence making’ (Boden & Epstein have also called it policy-based evidence making).

A decision has been taken to ‘massively expand’ the TFP and it has been announced in a very high profile way. However, at present, there is no evidence in the public domain that allows the public the opportunity to scrutinize this policy, which isn’t very transparent or accountable. One could conclude, if one were slightly cynical about the TFP, that civil servants were currently trying to produce evidence that supported the policy that had already been decided and announced.

But then, we know that statistics and criteria isn’t the strong point of the TFP and, as Eric Pickles himself has said in relation to this programme, ‘I’m in a hurry.’  Why let facts get in the way of a good story?

Reference

Slater T (2008) “A literal necessity to be replaced”: A rejoinder to the gentrification debate. International Journal of Urban and Regional Research 32(1) 212-223

I am planning to ask for an internal review of the decision, and will post again when I hear the outcome of this review.