Project management systems - Sensitivity analysis
Sensitivity analysis
Introduction
This section provides a brief overview of sensitivity analysis.
See also comments on Factorial Experiment Design.
It is a procedure to determine the sensitivity of the outcomes of a model based upon changes to its parameters.
This would not usually include changes in its environment.
If a small change in a parameter (input factor) results in relatively large changes in the model outcome, the outcome is said to be sensitive to that parameter.
If this is the case, either the input factor will need accurate control or the process will need redesign to reduce the sensitivity.
That is, sensitivity analysis is used to assess how robust the results are to uncertain decisions or assumptions in the model.
A process can be investigated using a mathematical model.
The model can comprise a series of equations, input factors, parameters, and variables which aim to characterise the process.
Computer-modelled simulations are used widely in the investigation of complex physical systems.
These models typically contain parameters, and the numerical results can be highly sensitive to small changes in the parameter values.
The model will be only as good as the input data which will be subject to uncertainty.
For example:
- Measurement errors
- Missing information
- Poor and limited understanding of the underlying mechanisms involved
- Human failings
These uncertainties limit our confidence in the output of the model.
It may not be possible for complete prediction in the model as it may be subject to natural random events (stochastic events).
Hence, it is good practice when modelling to provide an evaluation of the confidence in the model.
This may arise by assessing the uncertainties in the modelling process and its outcome.
Uses
GeneralSensitivity Analysis can be used to determine:
- How the model resembles the process under study
- It can identify those factors that mostly contribute to the variability in output
- Any interactions between the input factors
- The quality of model definition
- It is possible to identify which factors give rise to the maximum model variation
- It is useful when carrying out calibration studies in that it can identify those factors providing optimum output and those providing unstable output
It is also useful for:
- It can help identify critical assumptions and the more important criteria
- It can help to compare alternative model structures
- It can lead to improved data collection in the future
- By gaining knowledge of the uncertainty in production parameters manufacturing tolerances can be optimised
- Resource can be better managed
It can be used in any area where models are developed.
These may include financial use, risk analysis and others.
Other uses may include medical products research testing.
For example, criteria may include:
- Blind versus open studies
- By dose of intervention
- By severity of condition at start of a trial
- By size of trial
- By geographical location of study
- By quality of study
Business thrives on decision making.
In this context a model may provide better understanding of the key factors involved leading to a more informed decision making process.
Conversely, if we can identify those factors that have no influence over the model output which may lead to savings.
There are some aspects to be aware of in collecting the required data.
Simple models assume that the input factors are independent and have no influence over each other.
This is not usually the case in the real world.
Sales volumes and selling prices are usually connected for example.
Any model will be designed by human beings who will base their assumptions on their own experiences.
These may be perfectly valid but equally may no longer apply either now or in the future.
Any assumptions should be reviewed carefully.
In a similar fashion, the identification of any optimistic and pessimistic values is subjective and is likely to vary from person to person.
This can be compared to the use of ‘minimum’ and ‘maximum’ values when estimating for the PERT analysis.
In all of these cases, the values should be considered very carefully and reviewed before using in any models.
Method of analysis
The detail of any methodology will not be covered here as it would require some understanding of the underlying statistics.
Only the simplest and most common method is referred to here.
The most used method is sampling-based.
In this method input factors are considered and a known distribution of these is designed.
The model is then repeatedly executed by choosing combinations of these input factors, within the known distribution.
The outputs for each execution of the model are then measured and recorded.
The basic process has several steps:
- Identify the target functions and specify one of interest
- Assign a distribution function to the any selected input factors
- From these design a matrix of inputs with that distribution(s)
- Execute the model and compute the distribution of the target function output
- From the output assess the influence or relative importance for each input factor on the target function
Under PRINCE2® Sensitivity analysis is raised as part of the Business Case.
For PRINCE2 2005 the Business Case is the corner stone of any project under PRINCE2.
It must be viable before a project will be approved by the Project Board.
It does not just exist at the start of a project but features all the way through.
If at any point the Business Case fails the project must be stopped.
[see Business Case - part 1]
Under PRINCE2 2009 [see ‘The Complete Project Management plus PRINCE2’] sensitivity analysis is also described in the Business Case.
Sensitivity analysis can be used to determine whether the Business Case is heavily dependent on a particular benefit.
If it is, this may affect project planning, monitoring and control activities, and risk management, as steps would need to be taken to protect that specific benefit.
Sensitivity analysis involves tweaking the input factors to model the point at which the output factors no longer justify the investment.
For example, the project is worthwhile if it can be done in four months, but ceases to be worthwhile if it were to take six months.
[see Business case - The PRINCE2 approach - The contents]
PRINCE2® is a Registered Trade Mark of the Office of Government Commerce in the United Kingdom and other countries.