Skip to main content.

Evaluating the effectiveness of risk management

This guide describes a systematic way of finding how effective is an organisation’s current approach to managing risk. It considers the intentions of the organisation, how they are expressed and communicated and also what happens in practice. This leads to a realistic improvement program for the organisation’s framework for managing risk and each application of the risk management process. The guide stresses how management must be involved in all stages to ensure success.

Introduction

All organizations of all kinds face internal and external factors and influences that make it uncertain whether, when and the extent to which they will achieve or exceed their objectives. These objectives are its highest expression of intent and purpose, and typically reflect an organisation’s explicit and implicit goals, values, and imperatives or relevant enabling legislation.

The international risk management standard, ISO 31000:2009, defines risk as the effect of uncertainty on objectives. The effective management of risk is therefore essential if organisations are to achieve their objectives and satisfy the needs of their stakeholders.

It has been long recognised that good governance and effective management are best achieved through the development and deployment within an organisation of one coherent and consistent framework, methodology and vocabulary for management of risk, to be used for all types of activity. This ensures that:

  • There is a consistent and defensible basis for decision making at all levels, particularly where effort or capital is expended
  • Change activities are more likely to succeed
  • The organisation can pre-empt and capitalise on external changes such as those involving demographics, customers’ needs and government policy
  • All employees are encouraged to focus on and give priority to actions that aid and enhance the execution of strategic and project plans and the organisation’s objectives
  • The organisation is prepared for and protected from major incidents and losses
  • Tactical moves, to identify and seize opportunities are stimulated and enhanced
  • Accountability for risks and, most importantly, for controls and the monitoring and assurance of controls is clear and not doubtful.

In time this will also lead to a significant change in culture as the organisation as its employees engage on activities directly related to ensuring the achievement of goals and objectives and the successful completion of projects.

What is a framework and how does it lead to effective risk management?

An organisation’s ability to manage risk effectively depends on its intentions and its capacity to achieve those intentions. This intent and capacity is referred to as its risk management framework and is part of its system of governance and management.

The quality of the framework is important because effective risk management requires:

  • Clear expectations from ‘the top’
  • Appropriate capability (skills, resources, support)
  • Sound relationships with stakeholders
  • Integration of necessary risk management practices into the day-to-day activities and accountabilities of the management team
  • A commitment to continually learn and improve.

The risk management framework should not attempt to replace the natural capability of people to manage risk; rather it should enhance good practices so that the process is reliable, comprehensive and consistent. For this to occur and for the required capability to be achieved, the organisation requires:

  1. A set of suitable ‘tools’
  2. A coherent approach to training and communicating to people so that they can use those tools in a competent and consistent manner
  3. An approach that signals and reinforces the correct behaviour and way of thinking.

The typical elements of a framework and an illustration of how this supports the integration of the risk management process is shown in the figure below.

The framework for risk management

General approach to effectiveness evaluation

After many years of practical experience in evaluating and enhancing frameworks for risk management in organisations, Broadleaf believes that success depends as much in the manner in which any changes to a framework are developed and implemented as it does in the detail of the tools and written materials generated. This is why we would strongly recommend to our clients that we help it through a management of change process, where key internal stakeholders are carefully involved and engaged in evaluating the existing approach and in planning how, where and when enhancements will be made.

The core of this management of change process involves internal stakeholder representatives participating in a facilitated gap analysis and evaluation that then leads to a clear and practical enhancement and implementation plan. This is depicted in the “Y Model” shown in the figure and described below.

To enable those stakeholder representatives to participate effectively, they need to be well briefed on current risk management thinking and shown examples drawn from other organisations of elements of a risk management framework.

This approach has the added benefit that the participants of this process then become the organisation’s “Champions” who are motivated to lead the implementation process in their own departments and functions. They also act to convince their superiors of the merits of the approach and motivate acceptance and use.

Y Model

To be successful and efficient, the management of change approach requires:

  1. An accepted and accurate representation of the current arrangements for managing different forms of risk – the present situation
  2. The fundamental concepts of risk and risk management and the desired goals in terms of the risk management framework and process to be clearly understood by those sponsoring the change – the wanted situation
  3. A clear and accepted appreciation of the elements of the existing framework that need to be enhanced or improved and the nature of those changes and any additional elements that need created – what needs to change
  4. The exploration of options, constraints, enablers and critical paths leading to an appropriate plan of actions with timings
  5. A clear commitment to the plan and its implementation through the allocation of suitable resources by senior management and by their continued oversight of progress.

These steps can be tackled separately and the results fed back to senior management. However, after many years and numerous attempts we have found that most efficient approach, and the one that gains the greatest degree of ownership and endorsement, is to involve representatives of senior internal stakeholders in all these steps over a short space of time. This approach is described in detail below.

Phase 1 - Preparation

Evaluation studies typically start with an initial meeting where the detailed arrangements, including the schedule of activities and delivery dates, the documents to review reviewed and the interview candidates are agreed.

Prior to the meeting we issue a checklist of background documentation we would like to review and will often open up a secure Internet portal to which documents can be uploaded. This list can include:

  • Relevant policy statements, framework descriptions, internal standards and procedures, with a particular focus on decision support and controls assurance
  • Internal standards, procedures or guidelines that deal with particular applications of risk management. For example in the area of safety, procurement, security, operations, maintenance, BCM, compliance and project management
  • The current strategic plan and objectives
  • Examples of risk management plans and control assurance plans
  • Extracts from the risk management information system including risk registers and risk treatment plans
  • Methodology for and outputs from any quantitative risk analysis studies (range analyses) for schedule, capital and value evaluation and contingency estimation
  • Copies of recent reports to any risk management steering committees or review groups and the oversight committee that show risk management performance
  • Copies of any existing training and briefing materials that deal with risk management.

We then normally undertake a preliminary review of the materials and, from this, develop an aide memoire of sample questions that we might ask those we interview. This document is sent to those who are to be interviewed to allow them to prepare.

Phase 2 - Elicitation and verification

In our experience it is vital to observe and review how risk management takes place in practice. This is particularly true if there might be any discontinuity of practice across the organisation or inconsistent processes and systems. It is also important to test management’s perceptions of the current approach to risk management to see if it is currently viewed as effective and is likely to satisfy their future needs.

We therefore undertake this observation through a series of structured interviews with senior managers from which we will draw conclusions on:

  • The suitability of the current framework and tools to manage risk associated with an organisation of a comparable size and complexity, its risk profile and the risk criteria that should reflect its attitude (appetite)
  • The drivers of that attitude, based on what are recognised as the ‘key success factors’ and growth objectives for the organisation
  • The perceived usefulness of the current risk management process and its degree of integration into key decision-making processes;
  • The strengths and limitations of the other approaches to risk management specific to particular kinds of risks that co-exist in the organisation
  • Whether the tools and methods currently being used are capable of providing the organisation with a current, correct and comprehensive understanding of its risks and inform it whether the risks are within its risk criteria
  • The level of understanding of senior managers about aspects of the risk management culture
  • An outline of the perceived risk profile of the organisation and whether this varies from the risks reported to senior management and oversight committees.

Each interview usually takes about one hour and a member of the organisation’s risk function normally accompanies us to help transfer knowledge.

While the predominant purpose of the interviews is to obtain information from the participants to support our review, they also provide an opportunity to explain the purpose of the study.

At the conclusion of the series of interviews we normally provide immediate feedback to the organisation’s risk staff on:

  • Our findings
  • Our conclusions on the level of maturity, the strengths and weaknesses
  • Our initial thoughts on where the organisation could enhance the management of risk and the steps that should be taken.

This meeting also allows any misunderstandings or misperceptions to be rectified.

Phase 3 - Gap analysis and evaluation

Using the information we have gathered we conduct a detailed gap analysis and evaluation of effectiveness using the guidelines and principles in ISO 31000 and what we understand is world’s best practice as a basis for comparison. Often this is conducted as a facilitated workshop involving the management team.

The gap analysis looks at how the organisation expresses its intentions for managing risk and the elements of the capacity it claims it provides. In practice this involve us looking all the elements of the risk management framework and process shown above to determine if they are present and are suitable for the organisation and its environment.

We normally prepare a full gap analysis and evaluation report that includes our findings in terms of:

  • The framework and how it facilitates the integration of risk management into decision making, including risk management plans and the strategy for their implementation
  • How risk management is applied in strategy development and during the concept and development phases of projects, for decision-making and change management and as part of design review
  • Control assurance and reporting
  • The reliability of each element of the risk management process
  • How risk management is used to deal with changes and to provide contingency arrangements that respond to disruptions, including how learning and feedback take place after events, incidents and decisions
  • How the overall risk profile of the company is obtained and evaluated through aggregation and roll-up and how risks are treated at a corporate level
  • The form and content of governance reporting
  • How risk treatments are closed out and monitoring and review of risks, controls and risk treatments occurs
  • The organisation’s culture as it pertains to the management of risks in terms of both intent and practice
  • The adequacy and effectiveness of the systems and resources available to support the management of risk, including human resources.

Phase 4 - Gaining ownership and detailed planning

We believe that it is important that senior managers appreciate and can comment on our findings and conclusions and that this leads to support for any enhancement plan. It is important that this takes place before our report is made available to the oversight committee so that it can indicate management’s response.

We therefore normally present our findings and recommendations at a short meeting with senior managers. A typical draft agenda will be:

  • Fundamentals of risk and best practice risk management
  • Overall findings and assessment of the benchmarking review
  • Suggested improvements and enhancement strategies
  • Draft enhancement plan.

The planning component of this session follows the ‘Y model’ (see above) to elicit feedback and ownership of the current situation, the wanted situation and what needs to change. The management team is encouraged to discuss and compare options and then to finalise the enhancement plan actions and agree timelines. These agreements are recorded and included in our final report.

Phase 5 - Report to the oversight committee

Our clients often ask us to present our findings to their oversight committee. This provides them with the confidence that the evaluation was conducted in an independent manner and to enable the members to challenge and question any outcomes.

Normally our report is accompanied by the management-agreed enhancement plan, to indicate the organisation’s commitment to improvement.

In most cases the oversight committee is provided with progress reports against this enhancement plan at subsequent meetings.