Skip to main content.

Review of methodology for consequence assessment

Introduction

In 2005, the then Department of Agriculture, Fisheries and Forestry (now the Department of Agriculture) of the Australian Government was in the process of establishing a Centre of Excellence for Risk Analysis and determining its early work priorities. One of the objectives of the Centre was to research risk analysis methodologies as a way to build on and strengthen the integrity of Australia’s import risk assessment (IRA) process.

The Centre will build on Australian expertise in risk analysis techniques by undertaking and managing a cutting-edge research program on the appropriate methodologies for analysing risk. In particular, the Centre’s research is expected to enhance Australia’s capacity to assess import risks and the science applied to IRA processes. In this regard, the Centre will be a particularly valuable resource for Biosecurity Australia, the Australian Government agency that has responsibility for conducting import risk analysis.

As an input to the Centre's work once it was established, the Government sought tenders for a research project to compile and review consequence assessment methodologies and prepare a public report.

This project will examine and evaluate models used by other agencies to assess impacts (direct and indirect, including economic, environmental and social) arising from pest and disease incursions or similar hazards. The methodologies used by other national organisations to determine consequences (eg, the Department Environment and Heritage, Productivity Commission, Office Gene Technology Regulator, Australian Pesticides and Veterinary Medicines Authority, Food Standards Australia New Zealand) and similar international organisations will be examined.

Broadleaf was the successful tenderer. The work was undertaken by Dr Dale Cooper and Dr Sam Beckett. The full report is available from the Download link above.

Background

The objectives of this project were to catalogue the range of approaches to consequence assessment currently used by Government, to draw on the experience of others to further develop and improve Biosecurity Australia’s approach to consequence assessment and to identify areas where further work in this field might be undertaken by the Centre of Excellence.

It has been suggested that an improved approach to consequence assessment that takes into account the best available information would lead to improved decision making, greater public understanding of, and engagement with, the biosecurity policy-making process, and more consistent and repeatable import risk analysis outcomes. Other program areas within the Australian Government Department of Agriculture, Fisheries and Forestry (DAFF), and other departments and agencies, would also benefit from an understanding of the breadth of methodologies used in Australian Government.

We approached the project in two steps.

First, we carried out a survey of three key Australian Government departments to identify the divisions and agencies that carry out risk analysis, and therefore consequence assessment, as a part of their core business. The Government departments included were DAFF, the Department of Environment and Heritage (DEH) and the Department of Health and Aging (DHA). Two non-departmental agencies – the Australian Pesticides and Veterinary Medicines Authority (APVMA) and the Productivity Commission – and the Department of Industry, Tourism and Resources (DITR) Geoscience Australia, were also included in the survey. In the latter stages of the project, and at the recommendation of Biosecurity Australia, the Plant Health Australia (PHA) Regional Economic Impact Model was added to the survey. The project’s Steering Committee judged that although most other Commonwealth departments and agencies carry out risk analysis, and, as an element of this, consequence assessment, their methods and approaches are likely to be directed at issues too disparate from pest and disease analysis to be of significant value to the study.

Next, we held interviews with representatives from the key divisions and agencies (including Biosecurity Australia), and collated a cross-section of relevant published and internet materials. These formed the knowledge base for a systematic review of the context within which each division or agency carries out risk analysis, the frameworks used, and, most specifically, the methods employed to assess consequences. Each review led to concluding comments which were used as the basis for discussion and analysis. The analysis in turn led to a set of project recommendations.

After preliminary analysis of the three departments, their portfolio agencies and the various non-departmental agencies, the following were included in the review:

  • Australian Bureau of Agricultural and Resource Economics (ABARE)
  • Approvals and Wildlife Division, DEH
  • Australian Greenhouse Office
  • Biosecurity Australia
  • Biotechnology, DEH
  • Marine Division, DEH
  • Food Standards Australia New Zealand (FSANZ)
  • Therapeutic Goods Administration (TGA)
  • Office of the Gene Technology Regulator (OGTR)
  • Geoscience Australia
  • APVMA
  • Productivity Commission.

Of these, Biotechnology and APVMA were not found to be using methods or approaches likely to be of value to Biosecurity Australia. The analysis therefore focused on Biosecurity Australia and the remaining nine divisions and agencies.

In brief, the analysis found that three key difficulties could be encountered when applying Biosecurity Australia’s current method for consequence assessment.

  • The assessment of impact at systematic sub-national levels: this approach becomes difficult for impacts that are not naturally associated with levels of government. The approach is also difficult when assessing impacts accrued from multifocal outbreaks.
  • The period over which impacts occur: it is currently difficult to estimate the impacts of pests and diseases that cannot be eradicated quickly, or are likely to become endemic. It is also difficult to estimate impacts that continue to be accrued after eradication of the pest or disease, or its containment in a controlled zone.
  • The generic qualitative descriptors for the significance of national impact: because these descriptors apply to each of the direct and indirect forms of impact that Biosecurity Australia considers (termed ‘criteria’), they have no absolute meaning and tend to be used as a de facto ranking system.

The simple solution to the first difficulty is to accept that national impact is the goal, and that estimating impact at sub-national levels, whilst important for evaluating some Government costs, should not be carried out for all of the direct and indirect impact criteria.

The second difficulty does not have an immediate solution, and would be faced under any qualitative or quantitative model for consequence assessment. That said, the difficulty could be ameliorated by standardising the way in which longer-term or permanent impacts are handled, thus ensuring consistency within and between analyses.

The third difficulty could be addressed by assessing the significance of each direct and indirect impact against a different scale, or benchmark. We stress here that the development of suitable ‘scales’ for assessing direct and indirect impacts is a separate objective to the development of methods for assessing impacts per se. Scales should be transparent, and should equate to a measure or quantity that analysts and readers can readily relate to. Once developed, such scales can be adopted as the benchmarks against which the significance of direct and indirect impacts will be assessed. Individual pest or disease assessments, which can utilise a range of analytic or descriptive tools, methods and approaches, can then be compared with each of the scales and rated accordingly. Under the current system the single qualitative scale is not adequately defined, and this crucial step of the consequence assessment is made difficult and relatively non-transparent.

The Discussion and Analysis (Section 10, page 70) correlates these difficulties with the apparent strengths of the methods for consequence assessment employed by each of the agencies reviewed. The objective of this was to identify areas in which Biosecurity Australia or the Centre of Excellence could best focus ongoing collaborative research and development into this important aspect of risk analysis. In this part of the document, the agencies were arranged into three groups; based on their core expertise or the focus of their work. The first group of agencies provided strength in economic analysis; the second in the assessment of environmental impacts, of climate change and of vulnerability; and the third in qualitative risk analysis under legislated guidelines.

The process led to the development of nine key recommendations (below). For detail about these recommendations, the reader is referred to the Discussion and Analysis and to relevant parts of the body of the review.

Recommendations

The report made nine recommendations, in the areas of

  • Economic analysis
  • Environmental impacts, climate change and vulnerability
  • Qualitative risk analysis.

Full details are contained in the public report, available through the Download link above.

Centre of Excellence

The Australia Centre of Excellence for Risk Analysis (ACERA) was established in the School of Botany at the University of Melbourne on March 1st, 2006. ACERA ceased operations in 30 June 2013, and on 1 July 2013 it was replaced by the Centre of Excellence for Biosecurity Risk Analysis (CEBRA) in the University of Melbourne's School of Botany.