Summary

This case outlines qualitative and quantitative analyses of the uncertainty associated with a high-profile project to develop a scientific instrument. The case demonstrates the value of qualitative and quantitative analyses when they are conducted at an early phase of a project.

Background and purpose

The project

An international agency operated several large facilities for scientific research. It was updating one of its major devices to allow a new operating mode that would produce output radiation in a different spectral range and signal strength. A different form of detector would be needed to make use of the new output, with supporting hardware and associated control software, all of which would need to be integrated with the existing device and facilities.

Our client was a scientific technical organisation with a strong reputation for its expertise in developing large-scale, high-precision scientific instruments. The international agency asked it to design, build, test and install the new detector.

Purpose of the risk assessment

Our client’s project director wanted to systematically identify and improve those areas of the project that represented the greatest risk to the achievement of the organisation’s objectives. Historically, when our client had undertaken projects like this, requiring a high level of scientific and technical innovation, they had often exceeded both budget and schedule targets. Constrained financial circumstances at the time of this project meant that large over-runs were to be avoided.

In particular, the project director wanted to:

• Identify and understand the major uncertainties to which the project was exposed, and their relative priorities
• Assess quantitatively the uncertainty in the cost and schedule.

The organisation’s objectives were wide-ranging. The device to be updated had a high international profile, and the organisation saw the project as an opportunity to showcase its scientific and technical expertise and its ability to apply that expertise in novel and important ways. As a result, project-specific objectives like cost and schedule performance were intertwined with the strategic aims of the organisation, and they had to be interpreted from this broad perspective.

The overarching objectives were to achieve:

• Technical excellence, both real and as perceived by the scientific community, with performance, reliability, availability, maintainability, safety and quality characteristics that met customer and user requirements
• An enhanced reputation and image in the industry and across the scientific community, with demonstrated capability to deliver high-quality instruments on time, within budget and to high scientific standards
• Delivery to the agreed schedule and contractual conditions
• Financial value for the organisation
• Minimal disruption to the organisation’s performance in its core business
• Improved scientific, technical and engineering skills, knowledge and experience within the organisation.

Project phases

At the time of our involvement the project was in the concept design phase (Figure 1). The risk profile that was developed in this exercise reflected a concept design view of the work, although it included a consideration of the entire project life through all its phases. Further risk assessments were planned for subsequent phases to maximise the chance of success at each stage. These were set out in the project’s risk management plan.

Initial qualitative risk assessment

Risk assessment process

The initial qualitative risk assessment was aligned with the process in ISO 31000. The objectives noted above formed the basis for the criteria against which the consequences of risks were measured.

A set of 25 key elements was developed for structuring the risk assessment and setting the workshop agenda. They covered:

• Matters associated with the project phases (Figure 1), the activities to be conducted in each, and the resources and project management systems that would be needed, looking forward to the entire life of the project while recognising that it was only at the concept design stage
• Technical and scientific aspects of each major component of the new detector and its supporting hardware and software environment
• Interfaces with their client's existing device and facility infrastructure.

Risk assessment outcomes

The assessment identified 106 risks, of which 40 were high priority. These were primarily associated with risks having highly uncertain outcomes due to the innovative scientific and technical development work that had to be undertaken. In most cases technical solutions were known in principle, but some of them had not been constructed before and the time and cost involved in building practical, working instruments could not be predicted easily. Other high-priority risks were associated with damage to components that had long lead times for repair or replacement.

The high-priority risks became the focus of intense development effort and planning. Resources were allocated to address them, risk owners were assigned and the project plan was revised accordingly.

A further 11 risks had high potential exposures, meaning that the consequences for the project would be high were the existing controls to fail. Reviewing the controls for these risks, and where appropriate enhancing them, was an additional focus of management attention.

Quantitative risk analysis

Overview

The draft Concept Design Study had produced an initial estimate of cost and schedule for the project, but the risks identified in the qualitative assessment revealed considerable uncertainty in the cost and schedule estimates for individual work packages. These uncertainties needed to be taken into account in agreeing cost and delivery commitments with the customer organisation.

Quantitative models were developed that incorporated the most significant uncertainties and their effects on the cost and duration of the work. They were built using the @RISK add-in for Excel and MS Project, which allowed parameters to be represented by probability density functions and models to be evaluated using Monte Carlo simulation. The outputs represented the uncertainty in the predicted cost and duration performance of the project as a whole.

The distributions of values in the outputs were realistic forecasts of the uncertainty in the project’s cost and duration. They provided a sound basis for establishing targets and commitments for the completion of the work, which could be used in negotiations to demonstrate the rationale for the proposed budget and delivery date. The general form of a model output is shown in Figure 2. Information of this kind is a valuable means of holding the line when a project is under pressure to commit to unrealistically tight targets.

Schedule uncertainty

A simple schedule model was developed first using @RISK for Microsoft Project. The uncertainty in the duration for each package of work was evaluated through each project phase. Correlations were included between the durations of related technical work packages where there were common drivers of uncertainty.

As is often the case when task durations are subject to considerable uncertainty, the schedule model indicated a wide range of possible project durations, with an expected duration longer than the initial schedule. The expected value of the project duration is almost always greater than the initial estimate because:

• Most tasks have more chances of over-running than opportunities to finish early
• Where two or more tasks have to be completed before a successor task can proceed, the successor suffers the longest delay of all the predecessors, an effect known as nodal bias that tends to amplify the effects of uncertainty in component tasks.

The schedule model output for the customer acceptance milestone, which marked effective project completion, is summarised in Table 1. (The dates have been adjusted and the distribution has been normalised to a one-year duration for the purposes of this case study.) The schedule model also generated a distribution of the time by which Factory Acceptance Tests would be completed, an important commercial milestone for a large stage payment.

The main drivers of uncertainty in the schedule (Figure 3) were:

• The preliminary design of the system control and interface software
• The design and manufacture of a specific optical component
• The final design of a complex detector
• Commissioning
• Instrument assembly and integration.

Cost uncertainty

A cost model was constructed, with time-dependent costs based on duration forecasts from the schedule model. In most cases, the uncertainty in each work package cost estimate took account of the associated risks identified in the qualitative analysis, but there were a few discrete risks that were incorporated explicitly in the model as separate sources of uncertainty.

The cost model output is summarised in Table 2 and Figure 4.

The main drivers of uncertainty in the cost were:

• Instrument container C-1 manufacturing
• Project management
• Detector manufacturing
• Instrument container C-3 manufacturing
• Structure manufacturing.

Lessons

Addressing risk early in the project

The analysis described here took place in the concept design phase of the project. This delivered several benefits:

• The qualitative risk assessment helped to identify major uncertainties that became the focus for management attention and resource allocation, leading to adjustments to the project plan and likely improvements in project outcomes
• The quantitative analysis took account of uncertainty, including the risks identified in the qualitative assessment and the consequent improvement activities, leading to more realistic estimates of completion milestones and improved estimates of the cost to complete the project
• Quantitative sensitivity analyses, like the one in Figure 3, also assisted in identifying focus areas for reducing uncertainty.

Starting risk analysis activities early in the life of a project allows major sources of uncertainty to be identified and addressed, with revisions through the phases as work is completed and the pattern of uncertainties changes. This promotes continuous improvement in the project and enhances the chances of success. It also promotes healthy communication within a team that helps to expose misunderstandings, clarify interdependencies and resolve divergent expectations or assumptions.

The quantitative analysis enabled the project to ensure that all stakeholders understood the rationale for the cost and schedule commitments that were made and to avoid being pressed into unrealistically tight targets. Demonstrating a sound grasp of the risks the project faced, and the existence of sound plans to prevent these risks from jeopardising the project’s targets, were important factors in obtaining funding and approval to proceed.

Risk and project phases

Leading organisations usually require periodic updates of the project team’s views of risks and the associated schedule and cost estimates, often with independent verification. These are mandatory components of the decision support packages that are inputs to stage-gate (tollgate) approval decisions for major projects. The release of funding for successive stages of the work is often dependent on submission of a comprehensive and rigorous risk assessment and risk management plans.

The iterative nature of the qualitative and quantitative analyses proposed for this project and illustrated in Figure 1 represents good practice. Updating the analysis at each stage, while retaining essentially the same structure and layout for the analysis and the results, ensures decision makers are able to understand the risk profile of the work. It also supports effective communication between the project and those controlling the funding.