Skip to main content.

Optimising a public policy decision

Summary

Public policy decision-making often involves choosing between policy options that are each subject to uncertainty. Individual uncertainties might be relatively well understood, but it can be difficult to assess the combined effect of multiple uncertainties affecting both costs and benefits. This case study describes the use of Monte Carlo simulation to model the aggregate effect of multiple uncertainties associated with policy options for dealing with a potential animal health pandemic. The purpose was to assist policy makers to optimise their decisions in the context of tight political deadlines.

Background

Many managers and risk practitioners will be familiar with the use of quantitative risk analysis, using Monte Carlo simulation, to establish a probability distribution of the overall cost or schedule of a project. Such analyses can be used, for example, to determine an appropriate contingency amount to be included in the business case for a project, to compare several investment options with one another, and to assess the likelihood of a net positive economic impact in terms of net present value or internal rate of return.

A similar approach can be applied to assist in public policy choices between options for any kind of future investment where, almost always, forecast costs and benefits cannot be determined precisely. A typical scenario might involve a choice between a ‘safe’ but more expensive option and other options that may appear to be attractive initially, but where there is a chance of a high cost, long duration, or low benefits. These options have a greater chance of resulting in an unsatisfactory outcome than the safer option.

Figure 1 illustrates this for two options where cost is the main criterion:

  • Option 1 appears to be cheaper than option 2 for most single measures of cost, such as the mean (or expected) cost, or the most likely cost (at the peak of the distribution), but it has a much wider range
  • The right-hand side of the distributions show that option 1 entails a possible exposure to higher costs than option 2 would ever incur
  • Hence option 1 is superficially cheaper, but option 2 is safer.

Information like this allows decision makers to understand the balance between, on the one hand, the chance of a low expected cost with the possibility of a high cost over-run (like option 1) and, on the other, the much more certain prospect of an intermediate cost (like option 2). They might be concerned not only with the cost itself, but also that the outcome of option 2 can be predicted with far more certainty than the outcome of option 1. Certainty is very important in much public sector decision making.

Figure 1: Comparing the costs of two options

Modelling an animal health pandemic

The potential pandemic

This case concerns a potential pandemic disease of herd animals and how it might be managed across an agricultural region. The specific disease of interest can achieve pandemic spread by contact transmission between animals within a herd, and by uncontrolled movement of animals between herds in the region. Symptoms may not appear until some time after infection, which may delay diagnosis and response.

Infection leads to a severe reduction in the value of infected animals and their products, to the point where they make little or no economic contribution to farm income.

It is not possible to cure infected animals. On farms, isolation and culling of infected animals is required, accompanied by farm cleaning and decontamination. Within the region, government-controlled movement restrictions and containment are required, with extensive surveillance, testing and tracing.

Broadleaf was tasked with modelling the uncertainty in the cost, over a 10-year period, of four different options for managing a potential animal health pandemic. The aim was to support the development of public policy responses to be deployed were an infection to be detected within the region.

Options for control of the pandemic

Detailed modelling had been carried out to determine the costs over 10 years of four options for managing the pandemic, summarised in Table 1. The costs included direct on-farm costs, indirect off-farm costs for the providers of agricultural products and services, operational costs for government if a disease incursion were detected in the region, and wider macroeconomic costs.

Table 1: Control options

Option

Implications

A: Rapid eradication (aggressive culling of infected animals over a short period)

Cost of compensating farmers for animals that are destroyed

A large increase in animal testing and surveillance capability incurring costs over a relatively short period

Minimal long-term economic impact on the industry and the economy as a whole so long as it is successful

B: Phased eradication (culling of infected animals over a longer period)

Cost of compensating farmers for animals that are destroyed

A significant increase in animal testing and surveillance capability and costs spread over a longer period

Minimal long-term economic impact on the industry and the economy as a whole so long as it is successful

C: Long-term management of the disease

Government investment in testing, tracking and tracing capability (the approach taken by many other countries)

A large economic impact on the industry and the economy as a whole, mainly through lower production volumes

D: Wind-down of Government efforts to contain the disease

Large long-term damage to the industry and the economy

Approach and outcomes

The detailed models for each option used a large number of parameters and up to 100 kinds of cost, assembled from a wide variety of sources and organisations. As the time available for the analysis was short, we developed an initial quantitative model for each option with around 50 uncertainties, focussing on those items to which the cost outcomes were most sensitive.

The ranges applied to these uncertainties were derived initially in consultation with a small number of experts. They were then reviewed with other experts and individuals from the wider group of specialists available, and the ranges were revised where necessary.

Many of the uncertainties to which the outputs were most sensitive were the same across most of the options. In particular there were common uncertainties in lost production for each infected animal, commodity prices of agricultural products, the percentage of animals infected, the market value of culled animals, and the number of animals that would need to be culled from infected properties. This simplified the modelling task, as well as enabling a like-for-like comparison between the options.

Once the uncertainty ranges were finalised using this approach, the outputs from the simulation were used to determine the total response cost and industry impacts over 10 years for each option (Figure 2). This form of presentation of the results highlights the likelihood of meeting or exceeding a target, which is a key factor in public sector decision making.

Figure 2: Modelled option costs

To support the public policy decisions should a pandemic like this occur, the 90th percentiles of the simulation output distributions were used as the basis for comparing the options. The policymakers involved felt that 90-percentiles represented an appropriately conservative and cautious basis for a decision that had major implications to a regionally important industry. In particular:

  • Options A and B had far more certain outcomes than options C and D, indicated by their steeper slopes and narrower ranges in Figure 2, so budget planning could be undertaken with more confidence were either of these selected
  • Although it was possible that the costs associated with options C and D could be much lower than those of options A and B, there was a strong chance they might be far more expensive, and in the extreme there could be very unpleasant surprises associated with the long right-hand tails of the cost distributions.

By showing the full range of potential outcomes, the quantitative modelling provided the decision-makers with an understanding of the uncertainty they would face if they chose one of these four options. This gave them the confidence they needed to make a sound and justifiable policy decision. They had a high degree of confidence that the total cost for the chosen option was unlikely to be exceeded.

Lessons

Complex policy decision-making often involves choices between options that have costs and benefits that are subject to considerable and differing levels of uncertainty. The approach adopted here showed how quantitative risk modelling, using Monte Carlo simulation, can assist decision makers to compare options on a risk-adjusted basis according to their risk appetite, in this case characterised by the 90th percentile forecast cost. Benefits can be included easily in the analysis as well if required, thus modelling costs, benefits and the extent to which one might exceed the other.

Despite the complexity of the deterministic models, the assignment was able to be completed within a few days by preparing initial quantitative models quickly and determining from the sensitivity analysis which of the many uncertainties warranted close attention by the expert groups involved. Political deadlines were met, and the chosen policy option has been supported subsequently by all the key stakeholders in the industry.