Recently I was asked by an NGO in the social sector to review some work which had been done for them by Access Economics. They requested my involvement because they had first some concerns about the quality of the work and second problems in trying to understand the Access Economics report. I presume the reason they asked me is because I am a professor of health economics and the report dealt with health economics issues.

Having examined the draft report from Access Economics, I responded to the NGO that I thought there were various problems in it, some methodological, some that just leaving me unable to fathom what Access Economics had done.

In my report I challenged Access Economics on the fact that in their report they referred very often not to learned works in peer reviewed economics journals to support their arguments, but to previous Access Economics reports. They responded by saying that these were peer reviewed. They stated: “All these reports are in the public domain and all have been peer reviewed in their preparation using steering/reference group processes with clinicians and other technical experts in their respective fields.”

Access Economics’ concept of “peer reviewed” is rather different from mine especially in the way that they rejected almost all my criticisms of their work! Who is doing this “peer reviewing”? How independent of Access Economics is their “peer review” process? It may be but let’s hear.

Some of the methodological problems were quite technical and difficult to convey to a Crikey audience in a short piece like this. Others however arose with respect to quite simple aspects of economics. There were for example misunderstandings of what cost benefit analysis is, misunderstandings that would be inexcusable in an undergraduate essay. Access Economics also misused standard economic terminology such as productivity and confused that with production.

They also claimed that in considering the valuation of human life their method “has been showcased by DEEWR (and OBPR) as best practice“. Well their method is on that site but it is THEY (Access Economics) who call the process “best practic””.

In the lead in to the report on that Australian Safety and Compensation Council site, the ASCC state: “To the extent that the material in this document includes views or recommendations of third parties, such views or recommendations do not necessarily reflect the views of Office of the ASCC or indicate its commitment to a particular course of action.”

I was gentle with my comments in my report to the NGO on the work of Access Economics. For example I wrote:

The current draft is written in such a way that the specialist economist consultant [i.e. me] has had some difficulties in understanding it. That is also true of the non specialist readers at [the organisation involved]. In some instances the consultant was unable to follow the logic or the meaning of what the authors intended.

Access Economics’ response? “We disagree that the report is difficult to understand.” They did however offer to”“circulate a revised draft for a further set of comments/opinion” but added “this would require a contract variation,” i.e. they wanted more money to sort it out!

In the light of all of this I advised the NGO not to publish or in any way use the Access Economics report. Whether they take that advice is of course up to them.

It is clear however, given Jane Halton’s comments on Access Economics’ work on alcopops, that my worry is not unique nor does it relate only to the report that I looked at. Jane Halton criticised their data. That sort of problem can be spotted reasonably easily. More difficult to pick up are methodological issues. These worry me more. How are organisations in the social sector who want to get some ‘hard stuff’ on which to base their policy to judge if the ’hard stuff’ is really sound?

So I had a look at other reports on the Access Economics website. Another report states: “A cost benefit analysis evaluates the monetary flows from investing in an intervention followed by benefits that are incurred as a result.” That to me is at best an odd definition of cost benefit analysis. But I do not have the time or inclination to check more Access Economics reports.

Several of the reports that Access Economics have done are about estimating the total costs of various illnesses. These are requested by various organisations seemingly to try to persuade government to invest in their disease: “Government! Please look just how much this disease is costing the Australian economy. Please, given these big numbers, invest in reducing our disease!”

The real issue here is not how big a cost or burden this or that disease is but what can be done about the disease. If the government has, say, $10 million more to spend, in which disease should it invest? It is not about what the biggest problems are but where the best buys are. These studies by Access Economics — even if they were perfectly conducted — would not help to address the question of best buys.