- 6 - EXECUTIVE SUMMARY This evaluation assesses the effectiveness of the Financial Sector Assessment Program (FSAP) from the perspective of the IMF. A parallel evaluation by the World Bank’s OED assesses the World Bank’s role. The FSAP was established in 1999 to provide advice to strengthen the financial systems of member countries by facilitating early detection of financial sector vulnerabilities and helping to identify financial sector development needs. Although a voluntary program, it has become the principal platform for financial sector diagnosis at the IMF. It is a joint IMF-World Bank exercise (except in industrial countries), but with different outputs for different purposes, including a confidential report to the authorities and separate summary reports to the Boards of the IMF (the Financial System Stability Assessment or FSSA) and the World Bank (Financial Sector Assessment or FSA), dealing with issues that are in their respective areas of responsibility. Our overall assessment is that the FSAP represents a distinct improvement in the IMF’s ability to conduct financial sector surveillance and in understanding the key linkages between financial sector vulnerabilities and macroeconomic stability. It has significantly deepened the IMF’s understanding of the financial sector in specific countries, helped articulate policy recommendations, prompted better discussions with authorities, and helped support policy and institutional changes. The FSAP also permits an integrated approach to assessing financial sector vulnerabilities and development needs that could not be achieved by an ad hoc series of assessments. The evaluation also suggests that the joint IMF-World Bank nature of the exercise has been generally beneficial. Thus, putting in place this major new initiative within a relatively short period represents a substantial achievement. Despite these achievements, the initiative is at a critical crossroads and there is a danger that some of the gains could be eroded without significant modifications. The evaluation indicates two related sets of problems. First, financial stability assessments have not yet been fully “mainstreamed” as a regular part of IMF surveillance. Second, looking beyond the stage of initial FSAPs, there are serious doubts that current incentives for participation and associated priority-setting procedures will be sufficient to ensure coverage of countries where a strengthening of financial sector surveillance is most needed. The evaluation also points to the need for changes in the way the IMF organizes its own activities in order to make the best use of scarce technical expertise as well as to a range of measures that would further improve the quality and effectiveness of FSAPs. The evaluation has used a variety of evidence including cross-country analysis of all FSAPs; surveys and interviews of stakeholders; in-depth reviews of 25 FSAPs (including discussions with most authorities) as well as of all post-pilot Updates and post-2003 assessments; and interviews with a range of market participants. For comparison purposes, desk reviews were also undertaken of financial sector surveillance in a small group of systemically important countries that had not undertaken an FSAP. The evaluation examines evidence on the various links in the chain of influences that go from FSAP inputs through immediate outputs to intermediate and final outcomes. The main - 7 - evaluation findings address the following critical areas: (i) the nature of priority-setting under the FSAP; (ii) the efficiency of FSAP processes and quality of the main diagnostic tools; (iii) the overall quality of FSAP content, including the communication of findings and recommendations; (iv) whether the joint IMF-World Bank nature of the FSAP has been effective; (v) how well the IMF has used FSAP results in its surveillance, technical assistance and program activities; and (vi) evidence on the overall impact of the FSAP on the domestic policy dialogue, changes in policies and institutions, and market participants. The evaluation concludes with seven recommendations. Priority-setting under the FSAP Choices on priorities under the FSAP—which countries to assess and what issues to examine within each country—are critical to the program’s overall effectiveness. Several aspects of the FSAP make priority setting especially challenging—including the voluntary nature of the exercise and the joint IMF-World Bank approach with consequent multiple objectives. Although country selection has largely followed the guidelines set by the two Boards, a significant proportion (some 20–25 percent) of countries that are “systemically important” and/or have vulnerable financial systems—two key criteria endorsed by the IMF and World Bank Boards—have not been assessed, because the countries concerned have not volunteered to participate in the initiative. Moreover, a significant proportion of FSAPs that have been undertaken for countries that fit these criteria are becoming dated. Again, this largely reflects a reluctance of some countries to volunteer for FSAP Updates, so that the actual participation is not in line with the broader objectives of the initiative. This reluctance by some countries to participate has eased what would otherwise have been potentially sharp tradeoffs between different priority criteria. Going forward, if incentives to participate can be strengthened, the priority criteria may need to be modified to clarify how these tradeoffs should be managed. In response to the resource intensive nature of the initiative, the 2003 review of the FSAP called for more selectivity by reducing the depth of analysis of certain issues and the number of standards to be assessed in detail for each country, while remaining comprehensive in coverage. The evaluation suggests that these streamlining efforts have not adversely affected the quality of the overall vulnerability assessment in most cases. But there is inadequate discussion of the expected scope of the FSAP, including with the authorities, at the terms of reference stage. Moreover, there are limits to how far selectivity can be taken without losing the broad overview of intersectoral linkages that is one of the key advantages of the FSAP approach. Different IMF and World Bank budget procedures (including treatment of AML/CFT) complicate estimates of overall FSAP costs, but IEO estimates suggest that the total direct average costs (i.e., excluding overhead) of a post-2003 FSAP initial assessment were about $668,000 ($438,000 for the IMF alone). Average costs fell by about 6 percent (10 percent for the IMF) since 2003, reflecting the effects of streamlining and the fact that some of the most - 8 - complex financial systems were assessed prior to 2003. Since 2003, there has been surprisingly little difference between the average costs of FSAPs for advanced, emerging market, and low-income countries. Quality of FSAP processes and diagnostic tools Country authorities generally rated the technical quality of the FSAP teams highly, particularly the expertise of specialists. However, insufficient time for the FSAP team to prepare and familiarize themselves with country-specific circumstances was a widespread complaint. In addition, the burden of the FSAP on the authorities, inevitably very high, could be eased by better planning, including greater prior consultation with authorities at an early stage, more lead time on information requests, and greater personnel continuity. In many countries the FSAP has contributed significantly to assessing financial sector vulnerabilities—by helping to change the culture towards one that emphasizes system-wide risk assessments and to upgrade methodologies. Within this overall positive experience, however, there are significant differences across countries, and several shortcomings need to be addressed: i) Reporting of results from stress-testing in many FSAPs takes a “black box” approach, with too little discussion of the limitations implied by data and methodological constraints and choices on which shocks to analyze. This often results in overly simplistic messages about the strength of the financial sector. Greater “health warnings” about the interpretation of results are needed. ii) There is a considerable gap between the “good practice” approaches to modeling shocks and those used in many other cases. Some assessments have avoided analyzing the consequences of politically sensitive shocks (e.g., public debt defaults). iii) FSIs have generally not yet been used in a meaningful manner in most assessments, reflecting problems with data and interpretation of appropriate benchmarks for signaling vulnerability. iv) The quality of the data on the financial system is often not emphasized sufficiently. In some countries, more caution is needed before using available statistical data at face value, either for stress-testing or other analysis. v) Integration of the various standards and codes assessments into an overall FSAP assessment has added value, but the degree of integration varied from case to case. Moreover, there appears to be excessive focus on the “number” of principles for which a country was fully or largely compliant, which could give a misleading signal on the potential downside consequences of remaining gaps. vi) While the assessments of standards generally distinguish between de jure standards and de facto implementation, the crucial significance of institutional weaknesses for actual implementation is often not emphasized sufficiently. - 9 - FSAP content i) Overall quality The overall quality of the FSAP assessments is high, although problems were encountered in a minority of cases. A high quality overall assessment is one that combines effectively the results from the various evaluation tools to present the main risks and vulnerabilities to the financial sector with an indication of criticality and consequence. Such a “comprehensive” approach combines a variety of assessment instruments, coverage of the overall financial sector, and an analysis of the interaction between key macroeconomic risks and the financial sector in a manner that the sum is greater than the individual analytical components. In a minority of cases, the overall assessment does not give a clear indication of the macroeconomic/systemic importance of vulnerabilities and potential consequences if key problems are not addressed. These comprised cases where there was inadequate analysis of the criticality or urgency of vulnerabilities, the potential spillover effects to other segments of the financial system or corporate sector, and the macroeconomic impact and potential policy implications. One area that received too little attention in many FSAPs was the analysis and integration of financial cross-border issues. FSAP stability assessments have generally been limited to the segments and risks of the financial system that have domestic implications, even when some external/macro risks were considered for the stress testing analysis. FSAPs in countries with extensive cross-border financial sector participation have generally made little inroad into the broader global and regional dimensions of those cases, with limited contribution to identifying and highlighting potential spillover channels and effects. The effectiveness with which FSAPs addressed both stability and development issues in an integrated manner varied substantially and appears to have depended in part on the nature of the development issue. While overall judgments by IEO assessors on the balance between stability and development issues were generally quite favorable, FSAPs were more successful in handling some types of development issues than others. When the issue was one of reforming existing segments of the financial system to promote growth, there tended to be a close association between the development and stability aspects and FSAPs often handled these issues well. However, when it was a question of promoting the development of largely non-existent financial sectors, or encouraging the provision of financial services to underserved or excluded groups, there was generally little integration between the two aspects. Indeed, whether the FSAP is the best vehicle to address such types of development challenges remains an issue. ii) Articulation of findings and recommendations The main findings of the FSAP were generally presented in a reasonably candid manner in both the FSAP aide memoire and the more widely circulated FSSA, although couched in cautious language. But there were significant shortcomings in the prioritization of - 10 - recommendations in many cases. The reviews of the latest vintages of initial FSAP assessments and of FSAP Updates suggests that these shortcomings have continued. While there was no major loss in candor across earlier stages of the FSAP process, candor was sometimes lost at the critically important stage of integration with Article IV surveillance reports. There was a “loss in translation” in a number of cases between the messages of the FSSA and those incorporated in the staff reports for Article IV surveillance. When this happened the Board discussion tended to focus on the issues discussed in the Article IV report, crowding out problems that were flagged only in the FSSA, even though the latter report was also available to the Board. Factors that influenced how well key FSSA messages were integrated into the Article IV report comprised the degree of country ownership and the degree of integration between work of the FSAP team and area department teams. The Joint IMF-World Bank nature of the FSAP The principal rationales for making the FSAP a joint IMF-World Bank initiative were that, in light of the overlapping mandates of the two institutions on financial sector issues and the scarce technical expertise on such matters, considerable potential synergies could be attained by addressing stability and development aspects in a comprehensive manner and that combining the respective expertise of the two institutions would produce a more integrated analysis and set of recommendations. The evaluation suggest that, while there were some coordination problems, organizing joint teams that include both IMF and World Bank staff members has contributed significantly to the depth of analytical expertise and credibility of the findings in many, but not all, cases. Also, there is no evidence that the joint approach has led to a “watering down” of messages in order to achieve consensus between the institutions. Discussion of the relative weight to be given to stability and development issues was generally inadequate in earlier cases but there have been some improvements over time. However, the tools for analysis of financial sector development issues remain less well developed, a point noted in the parallel OED evaluation. How well has the IMF used the FSAP output? The overriding message emerging from the evaluation is that the FSAP exercise has deepened the IMF’s understanding of the financial sector and strengthened the quality of the surveillance dialogue on financial sector issues, but the IMF is not yet using the results as effectively as it might. The incorporation of FSAP results into Article IV surveillance has broadened the scope of monitoring of financial sector issues. Coverage of financial sector issues and vulnerabilities in Article IV consultations generally improved from the treatment before the FSAP, but financial stability issues have not yet been fully mainstreamed into Article IV assessments. - 11 - In many cases, discussion at the Executive Board of financial sector issues has been weak. In a few extreme cases, the Article IV surveillance reports and subsequent Board discussion failed to pick up on key messages in the FSSA (e.g., Dominican Republic). Contributing factors include: the use of cautious language in most FSSAs; the traditional focus and expertise (of both area departments and the Board) on macroeconomic policies; the prevalence of area department views when there were disagreements with the FSAP team; the failure of the internal review process to ensure an effective integration of FSAP issues in such cases; and the secondary role that the FSAP team leader played vis-à-vis area department and PDR staff at Board surveillance discussions. In terms of follow up, the financial sector content of surveillance in years following the FSAP has tended to diminish, but generally remained better than before the FSAP. The availability of adequate technical expertise within surveillance teams has been the major constraint on the effectiveness of follow-up activities when complex issues are involved. In many such cases, tracking the implementation of FSAP recommendations has taken a “checklist” approach of enumerating measures rather than appraising whether underlying vulnerabilities have been addressed. Focused assessments, with expert assistance from MFD (or ICM), have done a more thorough analysis of implementation of recommendations in particular areas. Only FSAP Updates appear to have had the capacity to undertake an in-depth tracking of implementation in specific areas; in the case of comprehensive reassessments, they also were able to take a broader view of how vulnerabilities had been addressed and of remaining challenges. The FSAP and associated ROSCs have become increasingly important drivers of IMF technical assistance (TA) in the financial sector, with a substantial proportion of TA going to emerging market countries. Within individual countries, post-FSAP TA provided by the IMF was in most cases broadly in line with the main areas of FSAP recommendations. However, many FSAPs have significant shortcomings as a platform for organizing follow-up TA, reflecting insufficient prioritization of recommendations and sense of sequencing as well as limited judgments on implementation capacity. Moreover, while the countries themselves should obviously take ownership of any follow-up plans of action, it would be helpful to have a clearer institutional framework for linking FSAP recommendations to plans for TA delivery that coordinate the activities of all important donors. A number of actions have been taken recently to provide a better interface between the FSAP and TA follow-up work, but it is too early to judge the results. The extent of conditionality on financial sector issues in IMF-supported programs has increased markedly since the late-1990s, but evidence suggests that this reflects underlying developments in the financial sector rather than the existence of an FSAP per se. A comparison of program conditions with the main FSAP recommendations suggests a mixed picture with regard to alignment. - 12 - Evidence of FSAP impact Attributing specific final outcomes within complex systems to particular activities such as the FSAP is extremely difficult. In this context, the evaluation sought to identify the proximate contribution of the FSAP (in terms of influence of the policy debate or use by country authorities) and assess what has actually happened in terms of changes in key policies and institutions. i) Impact on the policy debate The greatest impact has been on within-government dialogue and in supporting the authorities’ position in discussions with the legislature. In contrast, the use of the FSAP in general public debate has been very limited. In many cases, the main value-added of the FSAP process was through the interaction of the FSAP team with high-level policy makers, not through the final report. The impact on the policy debate was not confined to developing countries; among advanced economies, the FSAP has been instrumental in raising a number of “taboo” subjects or in influencing an ongoing political debate within the administration or legislature. The largest impact was in those countries where the government already had a high commitment to financial sector reforms. ii) Impact on policies and institutions The evaluation has identified a wide range of cases in which significant changes did take place subsequent to the FSAP and in which there is some evidence that the FSAP was at least a contributory factor, although direct attribution is not possible. The most commonly identified value-added of the FSAP was as an independent, expert “second opinion” on the financial system and reform plans. In a number of cases, this contribution increased the credibility of reform initiatives (including in the legislature). Critically, there has been a change in the “culture” in many countries vis-à-vis approaches to financial sector risk assessments. While there have been a number of other major influences from relevant institutions (e.g., BIS, Financial Stability Forum, etc.), the FSAP initiative does appear to have played an important contributory role in this change. But there were also a number of “missed opportunities” where the FSAP did not, for various reasons, lead to timely changes to forestall problems. The most dramatic example was in the Dominican Republic where a banking crisis broke out less than a year after the FSAP. There has been a high level of satisfaction among various standards-setting bodies with the feedback received from the IMF (and World Bank) on the standards through formal and informal channels. Greater efforts by the IMF to distill common cross-country messages from the various FSAP exercises would be welcomed. - 13 - iii) Impact on markets While many authorities identified the “signaling role” to markets as one of their motivations for participating in the FSAP exercise, the impact of FSSAs on the views of financial market participants appears modest. Credit rating agencies appear to use FSSAs somewhat more that other market participants. Within this generally limited impact, the effects appear greatest in countries where overall transparency is the least; failure to participate or to publish a FSSA is regarded as perhaps the most significant signal. Recommendations The evaluation’s seven recommendations are focused on three key themes: (i) reconsidering incentives for participation, clarifying priorities, and strengthening the links with surveillance; (ii) steps to maintain and strengthen further the quality of the FSAP and organizational changes within the IMF; and (iii) the working of the joint IMF-World Bank approach. Consistent with the IEO’s mandate, the recommendations are couched in terms of actions to be taken by the IMF, although, given the joint nature of the initiative, a number could require decisions by both the IMF and World Bank Boards. The recommendations are elaborated further in Section VII. Recommendation 1. The IMF Board and management should refine the criteria for setting priorities on IMF resource inputs into financial sector surveillance, including the FSAP. Based on these priorities, IMF staff should indicate, as part of its medium-term planning, what components are needed for strengthening financial sector surveillance in each country, drawing upon a range of possible modalities. These strategies would form the basis for more explicit accountability on results. Recommendation 2. To strengthen incentives and drawing upon these country-specific plans, IMF management should clearly signal to the Board those countries that it sees as the highest priorities for FSAPs and Updates, irrespective of whether these countries have volunteered. These lists should be the basis for periodic discussions by the Board of country-specific priorities. Recommendation 3. Strengthen the links between the FSAP and surveillance by mainstreaming FSAPs and follow-up work into the IMF’s regular surveillance activities. Recommendation 4. Implement steps to improve further the quality of the FSAP and strengthen its impact. Recommendation 5. Introduce changes in the organization of IMF mission activities to utilize scarce financial sector technical expertise (especially in MFD and ICM) more effectively in the surveillance process. Recommendation 6. Maintain the current joint approach, but clarify further the distinctive contributions the Fund and Bank can make, with the IMF taking the lead - 14 - where significant domestic or global stability issues are present, and the Bank taking the lead where financial sector development issues are more paramount. Such clarity should include a clear delineation of primary responsibilities for setting priorities (and contributing resources). Recommendation 7. The IMF, in conjunction with the World Bank and other technical assistance providers, should seek to establish a clearer framework for coordinating follow-up capacity-building technical assistance activities, based on the country’s own action plans.