Catalog

	Table of Contents
p. 63 of 74 BACK | NEXT I. Introduction II. Background III. The Economic Rationale of A Project IV. Macroeconomic and Sectoral Context V. An Integrated Approach To Economic Analysis VI. Identification and Quantification of Costs and Benefits VII. Valuation of Economic Costs and Benefits VIII. Large Projects, Linkages, and National Affordability IX. Least-Cost and Cost-Effective Analysis X. Investment Criteria: Economic Viability XI. Discount Rate XII. Uncertainty: Sensitivity and Risk Analysis XIII. Sustainability of Project Effects XIV. Distribution of Project Effects XV. Projects and Policies XVI. Appendices Appendix 1: Key Questions For The Economic Analysis of Projects Appendix 2: Project Economic Rationale: Market and Nonmarket Failures Appendix 3: The Project Framework Appendix 4: Identification and Measurement of Consumer Surplus Appendix 5: Treatment of Working Capital Appendix 6: Depletion Premium Appendix 7: The Use of Constant Prices In The Economic Analysis of Projects Appendix 8: General Methodology For Building Up Project Statements Appendix 9: Economic Evaluation of Project Output and Input Appendix 10: Economic Price of Traded Goods and Services Appendix 11: Valuation of Nontraded Outputs and Inputs Appendix 12: Shadow Wage Rate and The Shadow Water Rate Factor Appendix 13: The Economic Price of Land Appendix 14: Treatment of Resettlement Components of Projects Appendix 15: Calculating Economic Prices At The Domestic Market Price Or World Market Price Levels Appendix 16: Estimating The Shadow Exchange Rate Factor and The Standard (Or Average) Conversion Factor Appendix 17: Example of An Economic Rate of Return: An Irrigation Rehabilitation Project Appendix 18: Effect On Net Foreign Exchange and Budget Flows: An Example Appendix 19: Least-Cost Analysis and Choosing Between Alternatives Appendix 20: Estimating The Economic Opportunity Cost of Capital Appendix 21: The Treatment of Uncertainty In The Economic Analysis of Projects: Sensitivity and Risk Analysis Appendix 22: User Charges, Cost Recovery, and Demand Management: An Example For Piped Water Appendix 23: Financial Returns To Project Participants: An Illustration Appendix 24: Economic Evaluation of Environmental Impacts Appendix 25: Distribution of Project Effects Appendix 26: Impact On Poverty Reduction Appendix 27: Difference Between Economic and Financial Prices Appendix 28: Use of Economic Prices In Measuring Effective Protection Appendix 29: Exchange Rate Issues In Project Analysis XVII. Others

Guidelines for the Economic Analysis of Projects : XVI. Appendices

---

Appendix 21 : The Treatment of Uncertainty in the Economic Analysis of Projects : Sensitivity and Risk Analysis

I. Introduction

1. The economic internal rate of return (EIRR) of a project is calculated using the most likely forecast values of economic benefits and costs. However, the stream of benefits and costs is influenced by a wide variety of factors that may vary from the base case. Sensitivity analysis shows the extent to which the project EIRR or net present value (NPV) changes for different values of the major variables. Quantitative risk analysis considers the probability that different values will occur, and summarizes the associated risk attached to the project. These techniques can be used to assess the implications of uncertainty for the choice between project alternatives or for project viability.

2. Both sensitivity and risk analysis focus on alternative assumptions that have an unfavorable effect on the project result. Where the project outcome depends upon one or two major variables that are uncertain, mitigating actions should be included in the project design. Where a high level of risk is associated with a project that promises substantial returns, then the decision of whether to accept the project or not in its present design will depend on the decisionmakers attitude to risk.

3. Sensitivity analysis should be applied to all projects and subprojects with quantified benefits and costs. It should be applied also to project financial analysis and to the environmental components of project analysis where these have been quantified. The purpose in all cases is to identify actions that can mitigate the effects of uncertainty, or to redesign the institutional structure of the project to ensure sustainability. It should also be applied to projects, such as in education, health, and family planning , where benefits may not have been fully quantified. In such cases, sensitivity analysis can be oriented around a summary project measure, such as the unit economic cost of providing a new service.

II. Sensitivity Analysis

4. Sensitivity analysis is undertaken to help identify the key variables that can influence the project cost and benefit streams. It involves recalculating the project results for different values of major variables where they are varied one at a time. Combinations of changes in values can also be investigated. Sensitivity analysis involves four steps:

• selecting those variables to which the project decision may be sensitive;
• determining the extent to which the value of such variables may differ from the base case;
• calculating the effect of different values on the project results by recalculating the project NPV and EIRR; and
• interpreting the results and designing mitigating actions.

5. Project statements are made up from underlying project data and assumptions. For example, vehicle operating cost savings are made up from traffic projections for different proportions of vehicle type, their division into without project and generated traffic, data on road quality and maintenance operations, and data on the vehicles and their operating costs. Sensitivity analysis of the project benefits for a road improvement project should be based on changes in such underlying variables rather than the aggregate benefit measure. Focusing on underlying rather than aggregate variables facilitates the design of actions to mitigate against uncertainty.

6. Some of the variables entering into the project cost and benefit streams will be predictable and small in value compared with total costs and benefits. It is not necessary to investigate the sensitivity of the project to such variables. Other variables may be larger and less predictable. Postevaluation studies and previous project experience may indicate both the type of variable that is uncertain and the likely extent of divergence from the base case value. There are some types of variable in every project that are likely to affect the project result and may be key variables for the project.

7. The quantities of inputs required to produce the expected quantity of outputs will be given in the corresponding technical feasibility study. However this is often subject to considerable uncertainty. Inadequate supplies or maintenance can change the ratio between inputs and outputs and reduce project outputs. In addition, the quantity of output produced for a given set of input supplies will depend upon the incentives created for producers. Changes in management, improved skills, and financial returns to the producer will all influence the output produced from the available inputs. Consideration should be given to both the technical and institutional characteristics of the project as a guide to sensitivity analysis.

8. Quantities of outputs and inputs can also be affected by changes in technical or market conditions. Quantities should be broken down into their underlying componentsfor example, agricultural outputs into areas and yields, or vehicle cost savings by type of vehicle, or construction costs into unit costs and quantitiesand the sensitivity of the project to each of the components considered. Output quantities will also depend upon demand forecasts and market analyses. The underlying assumptions of these forecasts and analyses should be subject to sensitivity analysis.

9. Changes in the major values in the project statementsthe main outputs, inputs, and investment costsmay occur because of changes in prices for any of these items. Changes can occur in the market prices or shadow prices used in calculating costs and benefits directly or used in the estimation of opportunity costs. Commodity prices for major outputs and inputs can fluctuate considerably from year to year. The influence of the average annual forecast prices on the project worth should be tested by varying the forecasts, which should take into account the effect of possible changes in the quality of outputs over time on prices. The prices of labor and nontraded goods can also be subject to change although these might not have the same degree of impact on the project worth.

10. The timing and coordination of project activities may differ from the basecase. The timing of investment costs that occur early in the project life can affect the measure of project worth considerably. Alternative timings incorporating pessimistic assumptions about construction delays should be assessed. Different investment components need to be coordinated, for example, dam completion and resettlement in irrigation projects. The possible costs of delay in one investment component on the others should be investigated through alternative timing assumptions.

11. Project results can be seriously affected by the extent to which the investment assets are utilized. Lower utilization rates than in the basecase will be reflected in lower output levels and lower operational costs, but without any decline in investment costs. Utilization is commonly expressed as a percentage of feasible capacity use. The effects of a reduction in the rate of utilization should be investigated through adjustments to both benefit and cost streams, where possible distinguishing between fixed and variable costs.

12. Economic analyses of projects involve the estimation of opportunity costs for the outputs and inputs. In most calculations economic costs and benefits are calculated by using the ratio of the shadow price of a project item, or the resources that go into it, to its market price. The effect of the estimated ratios on the project worth should be investigated through sensitivity analysis. Except for the most labor intensive projects, it is rare that a project result would be significantly affected by a variation of the shadow wage rate for surplus labor; and for most projects, variation in the shadow wage rate for scarce labor is also unlikely to be significant. More significant will be the value assumed for the shadow exchange rate (SER) and therefore the shadow exchange rate factor (SERF), or the standard conversion factor (SCF), whichever numeraire is being used in the economic analysis. Alternative estimates of the SERF will affect both benefits and costs in the sensitivity analysis. Most simple estimates of the SERF (SCF) take account only of the tax and subsidy system and not of other factors separating financial and economic prices, such as monopoly rents; it is pertinent to include in the sensitivity analysis a higher value for the SERF (lower value for the SCF).

III. Procedure

13. The following procedure should be followed when assessing the consequences of changes from base case values of major variables.

• Variables to which the project is likely to be sensitive, such as those referred to above, and for which there is some uncertainty, should be listed. Alternative values should be assumed, based on previous project data where available. The change in the value of the variable should be calculated and expressed as a percentage of the original value. The extent of change should be stated for those variables such as timing of activities where a percentage change is not meaningful.
• The project NPV and EIRR should be recalculated for stated changes in variables one at a time. Unless a different country estimate is available, the NPV should be calculated using an economic discount rate of 12 percent.
• A sensitivity indicator (SI) summarizing the effect of change in a variable on the project NPV should be calculated. The SI is calculated as the ratio of the percentage change in the NPV to the percentage change in a variable (see Addendum). A high value for this indicator indicates project sensitivity to the variable. For variables where percentage changes are not meaningful, the percentage change in the NPV should be stated along with the stated change in the variable.
• A switching value (SV) should also be calculated. Where the base case shows a positive NPV, the SV shows the percentage increase in a cost item (decline in a benefit item) required for the NPV to become zero (which is the same as the EIRR reducing to the cut-off level of 12 percent). The SV is itself a percentage, the percentage change in a variable for the project decision to change (see Addendum). It can be compared with the variation shown in postevaluation studies or in price forecasts. For many variables, the SV will be high, implying a very substantial change in the variable before the project decision is affected. For a few variables, the SV will be relatively low showing there may be a significant risk for the project outcome.
• In deriving the economic costs and benefits of a project, a SERF (or SCF) will have been used along with other general conversion factors. Sensitivity analysis should include changes in the SERF (SCF) and other general conversion factors to see to what extent the project results are sensitive to the conversion factors used in the analysis.
• The change in the NPV should be calculated for combinations of variables, for example, a lower level of demand and a delay in investment completion, or an increase in cost together with a lower output price. The rationale for any combination of variables should be stated, bearing in mind that changes in more than one variable may have a common cause.
• The results of the sensitivity analysis should be presented in a table showing the base case results, the change in each variable considered, the sensitivity indicator, the switching value, and the changes in project worth for cases where these indicators cannot be calculated, or for combinations of variables. The table should include the consequences of alternative values relating to all technical, economic, environmental, and distributional aspects of the project.

14. The results of the foregoing sensitivity analysis should be reviewed considering the following questions:

• Which are the variables with high SIs?
• Have the calculations used the likely changes in these variables?
• Do the likely changes come close to, or exceed, the switching values that will change the project decision?
• How likely is it that the combinations of the variables investigated will occur?

These questions will help identify the truly key variables for the project, those that have a substantial effect on the project results, where plausible changes come close to or exceed their switching values. For the key variables identified in this way, a statement should be made of the likelihood of the variation tested actually occurring, the switching values for the key variables that should provide a basis for project monitoring, and the measures that could be taken to mitigate or reduce the likelihood of such variations from the basecase.

15. Where projects are seen to be sensitive to specific variables, steps should be taken to reduce the extent of uncertainty surrounding those variables. This may require actions at the project, sector, or national level, for example:

At the project level

• the agreement of long-term supply contracts at specified quality and prices to reduce uncertainty over operating costs;
• the formulation of training activities to ensure technical ratios are achieved and maintained;
• the development of information or publicity programs to increase access and use of new goods or services;
• the incorporation of external effects into project costs through regulation or taxation to ensure they are taken into account; and
• where there is considerable uncertainty in a large project or program, the implementation of a pilot project or phase to test technical assumptions and to observe users reactions.

At the sector level

• tariff and price adjustments to ensure appropriate incentives for producers and the financial liquidity of implementing agencies;
• technical assistance programs to develop project and operational management skills; and
• loan covenants to prompt necessary institutional reforms.

At the national level

• changes in tax and credit policy to influence incentives and simplify procedures;
• implementation of legislative reform and regulation to provide a more certain framework for productive activities; and
• changes in exchange rate and fiscal management to provide greater stability in prices and costs.

IV. Sensitivity Analysis: An Example

16. The irrigation rehabilitation project example of Appendix 17 is used here to illustrate the application of sensitivity analysis. The project involves a predicted increase in cropped area for irrigated rice, in cropping intensity, and in yield, as a result of rehabilitation, with a compensating decline in vegetable cropped area. The base case result, EIRR of 19.0 percent and economic NPV of Rs1,440 million at 12 percent discount rate, is also based on a long-term relative economic price decline for rice and a long-term relative economic price increase for fertilizer. The main variables to which the base case may be sensitive, together with the possible changes in those variables, are selected as follows.

17. On the basis of previous rehabilitation projects, there is uncertainty over the farmer response to improved irrigation. Postevaluation studies indicate the possibility of lower values for cropped rice area, cropping intensity and yield by 9, 10 and 6 percent, respectively. There is also uncertainty over the levels of cropping intensity and yield of both vegetables and rice, without the project. Increases in these variables of 10 percent have been included in the sensitivity tests.

18. The forecast price of rice and fertilizer should be key variables in the project analysis, as the project will increase both the quantity of rice output and the quantity of fertilizer input. In the sensitivity analysis, the forecast price of rice, which declines over the first ten years of the project anyway, is predicted to follow the same pattern but to be at the level of the lower range of the 70 percent distribution given together with the basic World Bank price forecasts. This is equivalent to a price 39 percent lower than in the base case. On a similar basis, the fertilizer price is tested at a price 42 percent higher than in the base case, at the higher range of the 70 percent distribution.

19. Other variables are also included in the sensitivity analysis. There have been delays in the implementation of previous projects. A two-year delay is considered here. The effect of a 10 percent higher investment cost is tested. The project benefits depend upon continued maintenance activities. Rather than a higher level of maintenance costs, the last five operating years of the project are excluded to allow for the possibility of inadequate maintenance activity. The two principal shadow price factors, the SERF and the SWRF, are subjected to lower and higher values, respectively, by 10 percent. Finally, some combinations of variables are also tested.

20. The results of these sensitivity tests on underlying and specific benefit and cost factors are given in Table 1. By observing the SVs in each case, very large changes are required in some variables for the project decision to change. This includes investment costs, the economic price of fertilizer, the cropped area for rice with the project transferred from vegetable production, and the SWRF. For some other variables, cropping intensity and yield for rice without the project, the SERF, and the reduced operating life because of inadequate maintenance, not so large but still unlikely differences from the base case would have to occur for the project decision to change.

IRR = Internal Rate or Return
NPV = Net present value

21. There are four variables to which the project is most sensitive and to which most attention should be paid. These include the economic price of rice, the cropping intensity, and the yield for rice with the project. The forecast values for these variables need only be less favorable by 20 and 14 percent for the project decision to change. The project result is also sensitive to delays in implementation. The first variable is outside the control of the producers and the country. The other three are part of the project design and implementation process, which the executing agency can affect with more or less success. The combination of higher costs and lower yields, which has also been tested, shows considerable sensitivity, together with the further combination also involving a lower economic price of rice.

22. The following recommendations are made in the light of these results of the sensitivity analysis:

• The monitoring of benefits during and after implementation should particularly include the cropping intensity and yields for rice together with its economic price.
• There is considerable risk because the project returns are so dependent on rice production and there is a great degree of uncertainty about the future economic price of rice. Under institutional development, funds should be provided for research activities at an experimental level into alternative crops for diversification purposes, including higher quality vegetable crops.
• The domestic price for rice and the rice marketing system must be reviewed to ensure there is sufficient financial incentive for farmers to switch from vegetable to rice production in early project years, otherwise the economic benefits of the project will be delayed.

V. Quantitative Risk Analysis

23. Quantitative risk analysis provides a means of estimating the probability that the project NPV will fall below zero, or that the project EIRR will fall below the opportunity cost of capital. This irrigation rehabilitation project is subject to uncertainty particularly with respect to cropping intensity and yields for rice together with its economic price. Risk analysis considers combinations of values for these major variables and the probability that they may occur.

24. In this case, a quantitative risk analysis can be recommended because of the substantial combined risk associated with the main with project crop. The following information is required for each of these variables to conduct the risk analysis:

• the results of the sensitivity tests;
• a range of values above and below the base case value;
• an upper and lower bound and a value in between; and
• a probability of occurring for each of these values.

In this case, a forecast distribution for the price of rice is available from the commodity price projections of the World Bank and can be used to derive this information. Changes in rice yields with irrigation have also been investigated through a number of postevaluation studies for similar projects, and these studies can be used to define the distribution of values for rice yield. Less information is available about cropping intensities, and assumptions will have to be made for this particular variable.

25. Quantitative risk analysis involves randomly selecting values for these three variables from the probability distributions that have been determined; combining these values with all other base case values to give an NPV result; and repeating such a calculation a large number of times to provide a large number of NPV estimates. These NPV estimates can be summarized in a distribution. The key feature of this distribution is the proportion of NPV values that fall below zero, and hence the probability that the project result might turn out to be unacceptable. There is no fixed criterion for using such a result. High risk probabilities may be associated with projects that have a high expected NPV. The probability of achieving a less than acceptable result is provided as part of the information on which a project decision is based.

VI. Addendum

1. Calculation of Sensitivity Indicator

SI = (NPV_b - NPV1 ÷ V_b - V1) ÷ NPV_b V_b

where V_b is the value of the variable in the base case
NPV_b is the value of the NPV in the base case
V1 is the value of the variable in the sensitivity test
NPV1 is the value of the NPV with the sensitivity test.

2. Calculation of a Switching Value

SV = 100 * (NPV_b / (NPV_b - NPV1 )) * ((V_b - V1 ) / V_b ) %

where the variables are defined as before.

3. Example of SI and SV Calculation

The following results are obtained when the price of a project output is varied downward.

Base Case Sensitivity Results

NPV_b 900 NPV1 720
V_b 10 V1 8.5

Calculations:

SI = ((900 720) / 900) ÷ ((10 8.5) / 10)
    = 0.2 ÷ 0.15
    = 1.333

The percentage change in the NPV is greater than for the price.

SV = 100 * (900 / (900 720)) * ((10 8.5) / 10) %
     = 100 * 5 * 0.15 %
     = 75%

The price of the output would need to be 75 percent lower for the NPV to fall to zero.

Back Appendix 20: Estimating The Economic Opportunity Cost of Capital	Next Appendix 22: User Charges, Cost Recovery, and Demand Management: An Example For Piped Water