1.2 - K. Measuring the success of e-government

Three background papers were presented during the session, the purpose of which was to facilitate discussions among participants. The first ESCAP paper examined the various ways of measuring e-government performance mirroring those used for traditional public programmes. The second ESCAP paper reviewed the status, harmonization proposals and national good practices of measuring e-Government success from the point of view of national statistical systems. The final paper by an Economic Affairs Officer from the United Nations Department of Economic and Social Affairs presented the United Nations Global e-Government Survey 2003.

Food for thought on measuring e-government performance

An Economic Affairs Officer of ESCAP said measurement issues were closely tied to marketing e-government success. He explored the commonalities and possible differences between e-government and traditional public programmes, in terms of their performance measurement aspects.

In particular, a case was made for the use of financial and related metrics for quantitatively assessing e-government programmes and projects. While those metrics were routinely used by most Governments and the private sector as objective tools to prioritize traditional programmes and projects, e.g., for infrastructure projects, they were rarely used by governments in Asia and the Pacific for performance measurement of e-government.

He stated that through the measurement of e-government, an indispensable management tool was provided, especially for resource allocation decisions and to communicate results. In addition it would allow Governments to identify constraints and barriers and also have a prerequisite for constituentcentric e-government.

E-government could be measured based on a hierarchical or logical framework or by process or attribute. Issues such as interoperability (national, international) and privacy also needed to be considered when measuring egovernment. Some methods discussed were:

• Financial metrics ("value")
• Risk measurement of the Institute for Development Policy and Management's (IDPM) eGov4Dev study
• National Office of the Information Economy (NOIE)⁷ Australia benefit/risk
• Internal Change Management

Data to measure e-government could be collected through traditional methods such as:

• Random telephone surveys
• Cost-benefit analysis
• Basic data gathering
• Web-based pop-up or clickable 'opt-in' surveys
• e-government specific web tracking

The representative of ESCAP made the following recommendations on measuring e-government performance:

1. Constituent-centric, knowledge management: implement a user and constituent-centric measurement system, in terms of objective, workflows and the underlying ICT architecture, making full use of modern knowledge management and expertise location management processes and technologies.
2. Financial metrics: include financial metrics such as costbenefit, Return on Investment (ROI), for e-government performance measurement, for improved decision-making and communication of programme and project priorities.
3. Privacy: allow sufficient time and resources for a public debate on privacy and security policies for all egovernment measurement activities.
4. Mass-customization: track the degree to which personalized services are provided to the individual citizen.
5. Compulsory performance standards: measurable performance standards need to be compulsory for all e-government programmes to ensure accountability.
6. Reach, inclusion: include performance standards that measure the impact of e-government initiatives on government operations as well as on citizens, including measures of user coverage and inclusion.
7. Risk: integrate risk assessments into the performance measurement and planning process.
8. Interoperability: include and monitor measures of the interoperability (G2G) of processes and tools.
9. Technology-neutral, open-source tools: metrics and processes to be preferably technology-neutral.
10. Ensure maximum flexibility, adaptability and full coverage of alternative communication channels.
11. Use open-source software and tools, particularly for knowledge management.

Experiences and trends in official and less official statistics

The presentation by a Statistician of the Statistics Division of ESCAP reviewed significant experiences in measuring the “success” of e-Government. He said the development of e-government should be seen in the context of the information society as a whole (Figure 13 [PDF 112KB | 1 page]). Measuring the success of egovernment was a topic that had not yet received adequate attention by policymakers, researchers, practitioners, statisticians and other stakeholders. The term "e-government" itself had been defined in a variety of ways, leading to some confusion about its real meaning and the underlying concept.

He defined e-government as enhanced government interaction at all levels using ICT. Interactions included all traditional interactions of government, plus those interactions that had become possible by virtue of technology advances.

Good examples in measuring e-government included Japan, Canada, Hong Kong, China and Australia. In Hong Kong, China, in order to measure the demand side, the e-government coordinating office commissioned an opinion survey to obtain users feedback on the design of government web sites and the provision of e-services. In Australia, the Australian Bureau of Statistics designed surveys to measure adoption of ICT in society as a whole, but some of the questions in surveys were directed to measuring use of government online services, enabling them to measure the demand side for government. It also conducted a survey for measuring specific use of government web sites. The examples all had common elements in defining terms for measuring the success of e-government. A common factor was that public administrations fashioned implementation of e-government through plans and strategies they established for themselves, by setting principles and then more measurable targets for G2G and G2C/G2B interactions (Figure 14 [PDF 152KB | 1 page]). However, there appeared to be different approaches in determining success, and thus, measuring those categories of interactions.

Success of G2G interactions could be measured in terms of costs/ benefits analysis. Such analysis required an initial benchmarking of all costs of doing business in the traditional way, in addition to estimating costs related to transforming the way business was done. The convenience of implementing new technologies was then measured through a comparison of traditional business costs and additional costs incurred by conducting business in the new way. The level of satisfaction reported could also measure the success of G2G interactions: internal customer satisfaction was a key concept that might help in measuring the level of success with regard to G2G interactions (Figure 15 [PDF 131KB | 1 page]). Fully integrated transactions measured the supply side of e-government (provision of electronic public services): plans and strategies were identified and lists compiled of all services to be provided to the public in electronic format and availability and what degree of sophistication and interactivity was reported. A number of data capture techniques such as opinion surveys or ICTuse surveys also provided useful insights into the success of e-government G2C/G2B interactions from the demand side. Further, some context or input indicators, such as the number of skilled staff in the public administration, might help in providing a quantitative description of the institutional context where e-government developed. Generally, the periodicity of data collection was annual. It was recommended that countries that had not yet started programmes for measuring e-government should do so.

The United Nations Global e-Government Survey 2003

The final presentation by an Economic Affairs Officer, of the United Nations Department of Economic and Social Affairs, explained why the United Nations Global e-Government Survey 2003 was conducted and reported on its results. She said ICT and e-government measurement was required to:

• Track national progress
• Identify disparities in access to ICT
• Move towards an inclusive information society
• International comparison

The survey was primarily issues-based and provided a measure of egovernment initiatives within a long-term development setting. The conceptual framework of the Survey stemmed from the Millennium Development Goals in which there was a specific recommendation that ICT benefits be available to all. Therefore, the Survey followed the Goals, in particular those that referred to information technology.

The survey focused on the issue of how willing and ready were Governments to employ the vast opportunities offered by e-government to improve the access and quality of basic economic and social services to the people and involve them in public policy-making via e-participation. Within that framework, the Survey contributed to the development efforts of the member States by focusing on the question: is e-government contributing to the socioeconomic uplift of the people? The Survey provided a benchmark to gauge member States’ comparative state of e-government readiness. The objectives of the Survey were to provide an appraisal of the use of e-government as a tool in the delivery of social services to the consumer and to provide a comparative assessment of the willingness and ability of governments to involve citizens in e-participation.

The Global e-Government Survey 2003 presented a comparative ranking of the countries of the world according to two primary indicators, (a) the state of e-readiness and (b) the extent of e-participation worldwide. The 2003 Survey showed that Governments had made rapid progress worldwide in embracing ICT technologies for e-government in the past years. In 2001, the Survey listed 143 member States as using the Internet in some capacity. By 2003, 91 per cent had a web site presence. According to the Survey’s e-Government Readiness Index, North America (0.867) and Europe (0.558) led, followed by South and Central America (0.442), South and Eastern Asia (0.437), Western Asia (0.410), the Caribbean (0.401) and Oceania (0.351). South Central Asia (0.292) and Africa (0.241) had the lowest average e-government readiness reflecting a low telecommunication index and a relatively low human capital index. The United States (0.927) was the world leader in e-global readiness. Among the developing countries, Singapore (0.746) led, followed by the Republic of Korea (0.744), Estonia (0.697) and Chile (0.671).

The Survey concluded that there were wide disparities between the "ehaves" and "e-have-nots". There was no standard formula for effective egovernment. However, the basic message of the Survey was that despite challenges, e-government as a tool, if applied correctly, held the promise of delivering, where many other innovative approaches had not in the past. However, it required a revisitation of global and national frameworks that presently guided the political, economic, social and technological strategies underpinning e-government programmes worldwide.

During the discussion that followed, the importance of allowing countries to develop their own measures of e-readiness was emphasized. Lessons and experiences of developed countries were not comparable to those of developing countries and developing countries should not be expected to achieve the same level of optimal achievement on e-government. What was most important was willingness and ability.

The analysis of national level web sites used as an indicator of ereadiness was also questioned as it was not considered to fully represent egovernment activities. There were a number of reasons, including that businesses did not interact with national level web sites. Further, national level web sites only represented a very small percentage of web traffic, about 30 per cent, and many interactions were between businesses and intermediaries rather than between government and businesses. She said the Survey methodology did reach deeper than the national level, but agreed that the Survey had not focused on business delivery of services, and needed to also provide information on sectoral level interactions. She explained that the Survey had not focused exclusively on business interactions as other existing surveys did so. The Survey was designed to focus on a basic needs approach: were citizen getting education, health, finance, employment and social welfare that were included in the Millennium Development Goals. These were considered to be the basic needs for information from Governments.

It was also suggested that measuring e-government performance should be broadened from e-readiness to also include e-maturity, that is, how to measure the outcome of e-government performance. Further, it was suggested that the Survey should be divided between developing and developed countries, with different methodology and criteria developed to make a fairer comparison as that might help to encourage countries in developing their egovernment projects more effectively. Development of a regional egovernment performance measurement system suitable for developing countries was suggested. That was supported by the meeting, with agreement that it would be useful to bifurcate some aspects of the measurements.

The common message from the participants was that there was a need to look at all the different aspects including impact. Difference in government structure was very much linked to the question of how countries with insufficient resources could set up e-government initiatives. Looking at the impact level could help to answer that question by providing guidance on how to set up partnerships, how to guard against too high expectations, and even establishing possible arrangements to share resources both at the national and regional levels.