Whatever happened to evidence-based policy making?

Prime Minister Gillard in parliament yesterday:

Because I believe in tackling the big challenges in the national interest… I see Closing the Gap as a way of understanding the problems.

It is evidence-based, accountable and transparent. It tells us what needs to be done first and fastest and builds a methodical approach. It allows us to build consensus in support of specific progress, instead of debating abstract ideas. To do what we can, with what we have, where we are.

Because I believe Australians judge Governments on delivery … I see Closing the Gap as a way of working on the solutions.

It is a way of making specific, measurable progress. It is practical and cumulative. It gives us new information which means we can invest where investment will make the greatest difference. Information which means we can be sure that the Government is meeting its responsibilities.

Worthy sentiments. If only they stacked up. Take the New Income Management program. The federal government is adding another serious question to its social and financial policy competence by informing the public that it is proceeding with the promised evaluation of NIM. The initial form of income management (IM) was a Howard initiative which was  justified as part of an “emergency” measure presumed to protect Aboriginal children.

The ALP government extended the IM program as a core part of the NT Intervention, thereby quarantining 50% of the benefit and pension incomes of all residents in 73 designated Aboriginal communities. However, when the UN was highly critical of the program’s suspension of Racial Discrimination Act as it breached Australia’s obligations under it, the government changed tack.

To be able to withdraw the suspension, Macklin decided to de-racialise the program by passing legislation in June 2010 that allowed the Australian government to expand the program. Now they can compulsorily quarantine the incomes of anyone on benefits in certain time or age based categories (but not more respectable pensions) in any declared area.

This change was pursued despite much solid evidence and presentations to a Senate Community Affairs Committee that the program had not proved its value in its previous three year existence. The government’s case was that this expansion was part of an evidence based broader welfare strategy to address social exclusion.

Despite government claimed “evidence” being refuted by a wide range of academic and welfare groups and the government’s proposal for extension being supported by only three of nearly a hundred submissions, the government majority (and the Abbott Opposition in Parliament) backed the legislation. The Majority Report did suggest that the lack of evidence needed to be addressed before any further extension of IM to other areas and groups. Ergo the evaluation of the proposed NT extension that has finally emerged.

The participation categories that remain eligible totalled 11,564 in June 2010, and another 9510 were on pension type payments not covered by compulsory payments under the new Act.

Unfortunately for the disadvantaged population in the rest of Australia this expansive and expensive evaluation is unlikely to protect their current payments systems from changes, as evidence from the three-year evaluation is unlikely to seriously affect or inform this government’s policy actions.

While the media release may imply otherwise, the intentions of the government are clear in its terms of reference on p3 of the executive summary. The terms of reference for developing the evaluation framework are that the evaluation:

• be completed by December 2014
• provide information on the implementation of the NIM in the Northern Territory by the end of 2011 in order to inform decisions about an expansion of the model beyond the Northern Territory
• result in data being collected that can be used to evaluate short, medium and, where possible, longer-term impacts/outcomes of new income management,
• include a set of ethics guidelines and an ethical clearance strategy relevant to this evaluation project.

The above terms of reference clearly suggest that the government will decide early in 2012 to extend the scope of the involuntary program to beneficiaries outside the NT. The timeline means this will happen despite little or no evidence by then of whether there are actual benefits from program for recipients.

This change fits with the PM’s already stated intention of clamping down on welfare and pressuring people into the paid work and a shift from welfare rights to massively conditional welfare payments.

The change will happen despite serious questions on whether it worked during the Intervention and a reluctance to wait and see whether evidence from this evaluation will find indications of effectiveness and cost benefits, in particular whether the substantial extra admin costs of around $100M per annum could be better spent on other services for this group.

The evaluation document does illustrate both the difficulties of finding evidence from any other examples of such programs and of evaluating this particular program. These problems arise from the program’s diverse origins and add-ons by government over the last three years. Given that the initial form of income management was justified as part of an “emergency” measure presumed to protect Aboriginal children, it was not opposed by the then Rudd Opposition and most of the welfare sector, just some Aboriginal groups.

All of which means no evidence of its value in this area has ever been offered, in fact a recent report to the NT government on showed child protection had deteriorated over the last three years, and failed to mention the intervention.

This is now nearly nine months after the government declaring certain benefit recipients in the whole NT as subject to the New Income Management, as well as transferring most  existing recipients to the ‘new’ scheme. This version includes a few new bribes for those who voluntarily sign onto income management and a complex, quite difficult exit system for those who may consider they do not need to be managed.

This includes compulsory budgeting workshops and financial literacy tests. There is also evidence that Centrelink is trying really hard to retain those no longer covered as voluntary clients as outlined by Paddy Gibson last year.

There had been no sign of the proposed evaluation till last week when the Minister and FaHCSIA media release emerged and stated:

The Australian Government is today releasing the framework for the independent evaluation of the new model of non-discriminatory income management. The Government has now rolled out the new model of non-discriminatory income management across the Northern Territory, including child protection income management. A voluntary income management and child protection income management pilot operates in Western Australia.

Future roll-out of the new model of income management beyond the Northern Territory will be informed by evidence gained from the independent evaluation…

The evaluation will include analysis of existing data on income management, as well as surveys of child protection staff, financial literacy service providers, and retailers. There will also be interviews and focus groups with people on income management.

The rest of the first stages of this evaluation (2011) mainly cover views on the processes of implementing the program by everyone but the recipients. While the release covers an extensive evaluation to report finally in 2014, as indicated above, the government will not wait for the results.

There are other questions on the value of the evaluation document. One is whether much of the data from the NT would apply in the very different locations and populations in the rest of the states and territories. However a major difficulty is the structure of the various bits of the program which create complications in any  evaluation of this program. To sum up the various difficulties as derived from the document:

• There is no baseline data because the program has already been implemented but in different places, times and ways.
• There are multitude of “populations” within those on NIM that have different entry points, needs and possible exits:
  1. These include the majority of compulsory recipients whose criteria for entry have no basis in their income management deficits but they just qualify because they are on certain benefits.
  2. Another group “volunteers” to sign up and there is current pressure and bribes in WA and elsewhere to add to these.
  3. Some are included because they are deemed “vulnerable” by Centrelink social workers because of certain very broad criteria and compulsorily placed on IM.
  4. Some are referred as parents who are reported to child protection authorities because they are neglecting their children or appropriate care.
  5. There are unconfirmed reports that some refugee families are being targeted because they are sending money to families overseas.
  6. There are also a limited number of Cape York families and maybe a few others who are disciplined by being put on reduced payments because of failure to send children to school
  7. The WA programs and many others are embedded with other support services which are patchy and not available elsewhere
  8. There are massive language and cultural issues in NT and some other isolated communities which make interviewing and informed consent difficult and raise major ethical issues.
• The difficulties of defining common ‘problems’ in such diverse groups, particularly as many issues do not relate clearly to their financial management
• This means it is very difficult to set outputs and outcomes to measure because the common problems may not exist (see below)
• The problems many would manifest in the compulsory category of benefit groups may not relate to personal difficulties in managing financially, but reflect the inadequacy of benefits that may not meet basic costs in urban settings.
• There are indications that criteria for exemptions set impossibly high standards which will discourage people from applying
• As most of these groups are also involved in other programs, often as part of the above categorisations, can research tell what changes are causal?
• As some have been on already for three years plus and some only a few weeks, how and when can relative changes be measured?
• What culturally appropriate measures are there to assess possible negative effects of shame and anger at being targeted in this way?

These show how the particularities of the NT and origins of the program in the Intervention make it unlikely that any findings would be able to be applied widely.

The evaluation document does recognise many of these problems, but makes some comments about “triangulating” data to make up for the gaps. However, the authors fail to critically question whether there is value in undertaking such a complex project with such possible flaws. They do not raise questions on the misuse of the evidence for the policy expansion decision nor question whether this process is primarily a post hoc searching for validating evidence.

It is not clear in the evaluation document whether there may be other benefits in pursuing research on this type of welfare change. The question is why has Macklin funded this complex and costly evaluation?

The answers only make sense if we connect this project to other aspects of the current government archaic models of delivering current social policy. The driving principles appear to be beliefs that “social inclusion” can be achieved by fitting the excluded into the interstices of current social systems. The basic assumption is that the excluded are flawed not the society.

Ergo, those on welfare for a long time must have serious personal issues, which inhibit their entry to paid work. The government then assumes the right, or maybe the moral duty, to coerce their compliance to models of better behaviour which will fix the problems.

In this case, the government (Macklin?) assumes regularising recipients’ money practices will create some order in their lives, and therefore result in better parenting, higher workforce participation and other desirable ends. These beliefs are spelt out in the paper with examples of ‘conditional welfare payments. These types of changes fail to answer the question of causation and ignore entirely the caveat expressed by the evaluation schema authors:

Many who have written about conditionality, whether in its favour or not, conclude that conditionality can be philosophically and morally justified provided that considerable care is taken to avoid burdening those people who are already unjustly disadvantaged (Deacon 2004). P51.

Most of those on benefits, such as Aboriginal people are already unjustly disadvantaged by systems and prejudice.

The evaluation paper academics have included a literature review with presumably, a wide search for examples of similar schemes as exemplars for the current process. Those reported on do not offer much in direct evidence that there are similar schemes that have worked. The listed examples do not offer any serious evidence that conditional welfare works well as a national policy model in developed countries.

Many of  the thirteen quoted examples are relatively small programs, often not continuing and set up in very different circumstances.

There are three USA programs, two of which are Food Stamps and TANF, which are the only national ones in developed countries. One a private NY program that was short term, and there is another short term private Canadian program. The other nine programs cited are all in developing countries: Mexico, Malawi, and Brazil. India, Bangladesh, Columbia, Honduras and Guatemala, dealing with very different problems and cultures, let alone economic systems.

These are odd countries for Australia to search for examples. The designs and problems faced, e.g. birthrates and malnutrition, are not necessarily related to Australia. So, apart from some questionable benefits from the named US programs, there are no developed comparable welfare states that have adopted such massive welfare system changes.

Given these constraints and problems, I wonder why reputable academic institutions such as SPRC, AIFS and the ANU would engage in this project and risk legitimating bad decision making and further examples of evidence being ignored?  Could it just be the temptation of the money for a research starved institution? Are there too many self-funded research units at universities being government servants? I did originally write tarts but many s-x workers may have better ethics.