Evaluation Framework

2.1 Concept and Definition

For the purpose of the study on Infrastructure and Regional Cooperation, a “cross-border” infrastructure project is defined to be either an infrastructure project with activities spanning two or more countries, or a national infrastructure project that has significant cross-border impact. Examples of national projects with significant cross-border impacts include transport infrastructure (land, sea, and air) projects that create international traffic, power projects involving sales of electricity to neighbor countries, and cross-border telecommunications networks.¹

In the context of ADB, its Regional Cooperation and Integration Strategy (ADB 2006a) does not explicitly define “cross-border infrastructure,” but it articulates its distinguishing feature as cross-border externalities that would not be caused by a collection of national projects. The strategy also requires that such infrastructure create “additional” net benefits that constitute some form of positive spillover effects accruing to all participating countries (i.e., regional public goods). That is, the net benefits of cross-border projects would be larger than those that would be obtained by national projects alone.² The current practice of project benefit-cost analysis is typically limited to quantifying direct impact in the nature of partial equilibrium. The analysis of cross-border projects needs to cover broader and indirect impacts in the nature of dynamic general equilibrium to the extent reasonable.³

The larger are these cross-border positive externalities, the stronger is the case for regional public goods, and therefore, the stronger is the economic rationale for regional cooperation. Another rationale for regional cooperation derives from coordination failure among national governments. National governments may not be willing to contribute their share of costs of providing regional public goods. This may be because they think that the partner countries might not match their contribution. Such a failure to contribute would not necessarily be strategic (because neighbor countries are aware that they will interact again in the future), but could be due to a combination of resource constraints and institutional weakness. The larger is the extent of such coordination failure, the stronger is the economic rationale for public intervention (see Appendix 1 [ PDF 23.3KB | 2 pages ] for a discussion on regional public goods)⁴. The rule of the game in regional cooperation is that all participating countries benefit (a win-win result for the club).

2.2 Project Cycle and Evaluation

In general, project evaluation requires that a consistent framework is applied at all stages of the project cycle. The process starts with analytical work that identifies potential investment projects. Then a country-level or sector-level program selects the projects to be financed. Then each project is formulated by financiers including external donors and executing agencies of the client country. After approval by all stakeholders, the project agreement is signed and its implementation commenced. A project performance report is prepared by the project officers as part of the monitoring of the project's implementation. After the project's planned activities are completed, the project officers produce a project completion report. This provides an opportunity for self-evaluation of the process of implementation, identifies success factors and problems encountered, checks all the assumptions made at the appraisal stage, recalculates financial and economic rates of return, and reports on compliance with social and environment safeguards. It also identifies lessons and follow-up actions, where appropriate, to ensure that project benefit streams continue. Further monitoring and progress reporting is carried out by executing agencies of the client country. After a few years, the project is post-evaluated. This involves rating project performance in relation to contribution to development impact using a set of criteria such as relevance, effectiveness, efficiency, and likely sustainability of the project outputs and outcomes.

2.3 Project Design and Monitoring Framework

At the project design stage, some logically consistent framework is required that can be carried through to the later stages of project cycle. Such a framework provides causal links between inputs or activities and outputs, and between outputs and outcomes (short- and medium-term outcomes and long-term impact). The framework also provides a basis for project monitoring and evaluation at the implementation and operation stages. For example, benefits to be monitored associated with cross-border project benefits can be related to the following categories of gains:

1. Market gains such as domestic market extension and economies of scale (e.g., power), connectivity and access to markets (e.g., transport);
2. Efficiency gains (cost saving) such as reduction in time and vehicle operating costs, reduction in physical barriers to trade (e.g., better roads, railways); reduction in non-physical barriers to trade (e.g., reduced transaction costs due to trade facilitation, harmonized customs, and border formalities); and
3. Welfare gains such as contribution to trade creation, benefits to all participating countries and subregional community, and increased regional and global integration.

2.4 Ex Ante Evaluation

Projects are pre-evaluated looking at project rationale and strategic links, alternatives, costs and benefits, and risks and uncertainties. For example, ADB's guidelines for project appraisal (ADB 1997) provide project appraisal concepts and step-by-step methods. In addition, there are sector-level handbooks for project economic analysis and also a methodology note and handbook for economic analysis of subregional projects.

While economic analysis of national projects is concerned primarily with whether the overall net economic benefits are positive and generally helping the poor, the economic analysis of cross-border projects goes a step further to analyze how their benefits and costs are distributed across participating countries. As multiple governments must agree on and coordinate the institutional and financing arrangements of cross-border projects to ensure all stakeholders can anticipate net gains from the project in question, analysis of benefit-cost distribution helps to alleviate coordination failure and ensure project sustainability. In particular, it is important to identify possible net losers and build in necessary adjustments for socioeconomic and environmental externalities. Most obviously, for example, such adjustments can include compensatory arrangements for relocated residents and for participating governments to ensure operation and maintenance and to address unintended negative impacts.

In principle, project analysis, whether national or cross-border, should care about externalities and distributional aspects in order to better inform decision makers and stakeholders. ADB's guidelines (ADB 1997) dedicate one appendix to distribution analysis. Adhikari and Weiss (1999a, 2004) extended this framework to cross-border aspects with the stakeholder breakdown between participating countries. Therefore, a basic methodology of distribution analysis for cross-border projects has been put forward (see Box 1 [ PDF 16.2KB | 1 page ]).

As discussed above, a successful cross-border project should offer benefits for all participating countries. To determine this, analysts and evaluators need to look at project outcomes and impacts, i.e., short- and long-term benefits, and their distribution at local, national, cross-border, subregional and regional (extra-subregional) levels. Complementary investments can create additional demand and increase project benefits. For example, economic corridors complement transport corridors and also mitigate costs (e.g., to environment and public health). Benefits arise from cost savings, time savings, efficiency gains because of the larger scale of operations, and welfare gains because the subregional society as a whole benefits. However, estimation needs to be carried out carefully to avoid double counting of such benefits as vehicle operating cost (VOC) savings and time savings vis-à-vis growth in transport sector activities, better quality roads vis-à-vis road safety (time savings of passengers and cargo, savings of lives and injuries), and additional economic activities vis-à-vis better access to market (not included in the cost savings).

Equally important is to identify risks to sustainability of outcomes and impacts. Important considerations for this risk identification include operation and maintenance by transit countries or locations and revenue generation or special funding for that purpose. Common project risks and sustainability factors relate to financial returns and revenue generation of the project, sharing of costs among participating countries, technical upkeep (operation and maintenance), foreign exchange risk (e.g., in cases of power purchase), and complementary investments.

2.5 Post-Completion Evaluation

Upon its completion, the project's performance is evaluated based on relevance, effectiveness, efficiency, and sustainability. Relevance refers to the adequacy of the design and the consistency of the project's impact and outcome with the government's and donor's development strategies at the time of approval. Effectiveness refers to the extent to which the project outcome as designed and approved has been achieved. Efficiency refers to how economically resources have been converted to results, typically expressed as the economic internal rate of return or cost-effectiveness indicators. Sustainability refers to the likelihood that human, institutional, financial, and other resources are sufficient to maintain the project outcome over its economic life.

2.6 Impact Evaluation

Impact evaluation involves systematic identification of a given development activity's effects—positive or negative, intended or not—on individual households, institutions, and the environment. It helps us to better understand the extent to which activities reach the poor and the magnitude of their effects on people's welfare. Tools for impact evaluations can be large-scale sample surveys in which project populations and control groups are compared before and after the intervention and possibly at several points during it. Evaluations can also be done through small-scale rapid assessment and participatory appraisals where estimates of impact are obtained from combining group interviews, key informants, case studies, and available secondary data.

There are two basic approaches to impact evaluation: non-experimental and random experimental. The non-experimental approach uses statistical techniques to construct the counterfactual outcome. These techniques, though frequently used, are often subject to biased results that may lead to incorrect development impacts. Random experimental methods are common in the pharmaceutical and other industries but are new to the economic development field. In the medical field, random assignment to treatment and control groups (also called a “randomized control trial”) is implemented depending on the nature of the medical intervention, who the beneficiaries are, or what the benefits are. Thus, as in the medical field, the unit of randomization in the economic development field could be individuals, groups (e.g., schools, primary health care centers), or geographical areas (e.g., villages, cities). The random selection of the treatment (under the project) and control (outside the project) groups ensures that in general, these groups are identical at the outset. Any difference in the socioeconomic or environmental outcomes between the two groups after the project is taken to be attributable to the project.

Random experimental assignment is considered to be the most reliable approach to measuring net impact of development assistance. As Duflo, Glennerster and Kremer (2008) have pointed out, only properly implemented randomization can solve selection bias completely. For example, the PROGRESA program (now called Oportunidades) in Mexico is one of the best known examples of a randomized evaluation in developing countries. The program was launched in 1998 with a deliberate built-in evaluation design in which half of the households in target communities were selected to receive the program, and baseline and subsequent data were collected for both treatment and control communities. Actual evaluation of the program was contracted to International Food Policy Research Institute and the data was shared via their website. The evaluation had a demonstration effect and was replicated in many other Latin American countries.

The nature of randomization requires that the treatment groups (areas) as opposed to control groups (areas) be fairly disaggregated, such as at the level of individuals, villages, or at most districts. Randomized experiments are therefore more suitable for “targeted” projects such as education and health programs than infrastructure projects because the “treatment” (or influence) of the latter is by nature diffused geographically. Furthermore, while the main purpose of targeted interventions is to deliver direct benefits to the intended beneficiaries, much of the benefits of cross-border infrastructure projects are indirect and diffused with varying influence over beneficiary groups, making it difficult to design randomization even at the project design stage. Nonetheless, a carefully carried out impact evaluation can be useful in many important ways. Box 2 [ PDF 19.3KB | 1 page ] summarizes the scope of usefulness, as well as the strengths and drawbacks, of rigorous impact evaluation.

Given the special characteristics of cross-border infrastructure projects, their impact evaluation must strike an appropriate balance between theoretical rigor and practicality of implementation. Most likely, the best option is to apply a quasi difference-in-difference approach by comparing “before” (baseline) and “after” (post completion) situations between the areas (people) under project influence (e.g., along the designated economic corridor) and those located relatively farther away from project influence.⁵

Economy-wide general equilibrium modeling such as the use of computable general equilibrium (CGE) models has the advantage of capturing indirect effects through forward and backward linkages within and across economies given the rigorous economic structure built in the models. However, it has a high data requirement that may not be normally available at disaggregated levels in developing countries. Also, CGE analysis tends to be simulation or scenario exercises based on certain assumptions that dictate changes in exogenous or policy variables. This type of analysis may be more suitable for macro- and sector-level analysis, which is left to other discussion papers in this research project.

2.7 Evaluation Criteria

With the above framework in mind, we can develop a set of practical evaluation criteria specific to cross-border infrastructure projects applicable at various stages of the project cycle. These stages are (i) preparation stage (project rationale and inputs); (ii) implementation stage (process of deployment of inputs to produce outputs); (iii) implementation completion stage (physical outputs and initial outcomes); and (iv) post-completion stage or operation (ultimate desired outcomes and impacts). A particular focus should be on the identification of cross-border externalities—both positive and potentially negative. Table 1 [ PDF 23.9KB | 2 pages ] presents the kinds of criteria applicable in evaluating cross-border infrastructure projects.

Download this Paper [ PDF 386.7KB| 44 pages ].

Comment(s)

There are [0] comment(s) for this entry. Post a comment.