A large multinational resources business consolidated its business systems into a single global system. The scale of the system and the sense that the business was putting all its eggs in one basket raised concern in the Australian part of the business about disaster recovery.
A proposal was received from one of the companies supplying IT infrastructure and services to operate a hot standby but the cost was high and it was unclear if it was worthwhile.
Broadleaf facilitated a risk assessment of the new system’s possible modes of failure and the business team provided inputs that allowed the frequency, severity and duration of failures to be modelled. Each major part of the system (general ledger, invoicing, purchasing, HR and so on) was then assessed as it was implemented to assess how it would be affected by each failure mode and how the business would respond. This too was modelled.
The models of failure modes and of responses to failures represented uncertain events and uncertain quantities such as the cost of damaged equipment or response times and numbers of personnel required to deal with the system going down. The models were evaluated by Monte Carlo simulation to assess the annual average cost and peak exposure.
The development of the models with the business teams led to disaster responses being thought through, often for the first time, and pragmatic recovery processes being defined. Many of those concerned were surprised to discover how well they would be able to respond. The loss of a central system will always be disruptive and disturbing. However, temporary measures to maintain proper records, temporary staff to handle tasks that are usually automated and recovery processes to reinstate the main system’s records when it was restored proved simpler and less costly than anticipated.
The analysis concluded that the fixed annual cost of a hot standby system was about ten times as much as the forecast annual value of bearing the risk and that even the peak exposure was bearable in the context of the turnover of the business.
The business was able to avoid a significant ongoing expense and recovery procedures were drafted through the facilitated analysis. These draft procedures were later developed into formal procedures to be implemented in the event of the system being lost.
The business was left in a more resilient state with personnel who now understood the risks associated with system failures and how their function would need to respond if one happened.