Evaluation of The Second MFTOT Course

A. The Evaluation Framework

The framework used in evaluating the MFTOT centers around the key debates on the effectiveness and efficiency of distance learning in the case of developing countries.

The first evaluation question is concerned with whether the MFTOT improved access without sacrificing quality. In this context, three factors are considered:

• Quality of course materials. Course materials are to be evaluated on not only the content, they whether they are well- adapted into self-study materials for distance learning;
• Student support system. Student support and monitoring system is crucial to address the issue of isolation in self-learning; as such, we evaluate the effectiveness of mechanisms for student support provided to foster learnerteacher and learner-learner interaction; and
• Accreditation system. A well-recognized accreditation system that can raise the credibility and worth of certification obtained from distance learning activities.

The second question is concerned with whether the MFTOT improved access without sacrificing equity. This will mean evaluating whether the course design and materials were sensitive to the needs of the underprivileged group, the differences in initial endowments as well as cultural and linguistic differences.

The third question looks at whether the MFTOT utilized an appropriate mix of technologies. This requires a look at the extent to which the MFTOT built on the existing ICT infrastructure and utilized a mix of media and technology that were complementary as well as cost-effective.

Finally, we asked if the MFTOT addressed long-term institutional building in developing countries. Since donor-supported activities are time-bound, and since training institutions in developing countries generally have limited capacity to initiate distance learning courses, it is important that donors capitalize on the strengths of existing training institutions while keeping in mind long-term institutional building objectives.

The framework is recapped in Table 3 [ PDF 84.1KB | 1 pages ].

B. Evaluation Process and Profile of Evaluation Respondents

Evaluation forms were handed out to participants during the final videoconference session of the course. Along with background information on participants, the forms included eleven specific evaluation criteria covering different aspects of the course. Each criterion required a numerical rating from a scale of 1 to 5 , with a score of 5 considered excellent, 4 very good and 3 fair. Open questions on what participants liked most and least about the course, as well as suggested improvements were included. The complete evaluation form is attached as Annex 1 [ PDF 123.7KB | 3 pages ].

C. Evaluation Results

Since participating in the evaluation of the course was not mandatory and since not all regular participants attended the last videoconference session, evaluations were obtained from 62 participants, representing 42.2% of the total number of course participants. Broken down by type of respondents, evaluations were obtained from 75.5% of the total TOT participants, and 29.9% of the total regular participants Table 4 [ PDF 80.5KB | 1 pages ].

1. Learning Activities Attended by Respondents

Respondents were asked whether they attended all, more than half, or less than half of the MFTOT learning activities. Overall, 54.2% reported attending all of the learning activities⁸, while 37.3% reported attending more than half. Not surprisingly, a greater proportion of TOT participants (63%) reported attending all of the learning activities. It is noted that up to 88% of regular-participant respondents reported attending more than half with 40% attended all the learning activities. Given the fact that accreditation and on-line tutoring were not part of the motivations for regular participants, the high level of attendance reflects their strong interest in the course content. Figure 6 [ PDF 80.5KB | 1 pages ]

2. Improving Access while Maintaining Quality

Within a relatively short period of the course announcement, a large number of applications and inquiries (over 100) were received. To maintain effective student support, only 40 TOT participants were selected while the rest joined as regular participants. The high number of applicants underscores the high level of demand for this type of training in microfinance.

Quality of Course Materials

The MFTOT’s success in attracting participants and improving access does not seem to have come at the expense of quality. Table 8 [ PDF 93.3KB | 1 pages ] below presents evaluation score on questions pertaining to the quality of the course. Based on the respondents’ perception, the course was highly successful in providing excellent course materials. The MFDL workbook and CD-ROMs were consistently ranked as the highest rated element of the course, both in the 1st and 2nd MFTOT evaluations (see Annex 2 [ PDF 83.8KB | 3 pages ] for the results of the first MFTOT evaluation). In the 2nd MFTOT evaluation, it, garnered an overall average rating of 4.47 (out of 5). The average rating by TOT participants, who presumably studied the materials more thoroughly, was significantly higher(4.67) than that by regular participants (4.28). In the open questions, the MFDL package likewise topped the list of elements which participants liked the most about the course.

The course was highly rated in terms of providing knowledge that is useful and relevant, obtaining an average rating of 4.24 and 4.31 for TOT and regular participants, respectively.

While these results are encouraging, some aspects of quality may warrant revisiting in future courses. First of all, the course was rated as fair in terms of providing information that was new to participants. The disaggregated ratings show that regular participants seem to have acquired more new information compared to TOT participants. This result can probably be expected given that majority of the TOT participants were selected from experienced microfinance professionals, many of whom hold managerial positions in their MFIs. See Table 6 [ PDF 78KB | 1 pages ].

Student Support

The course was rated fair with regard to the learning support provided by the DLC local facilitators, and tutors. It appears that support for the MFTOT differed among DLCs, based on their structure and the level of interest of DLC staff. Since some DLCs operate as cost centers while other DLCs are financed by the World Bank, the incentive for staff to be involved with the MFTOT varied. The experience during the recruiting period showed that DLCs in some countries did not assist in recruitment and the course was cancelled for those countries. In Lao PDR, where the videoconferences were held at the World Bank office, one local staff valued the course highly and actively supported the course leading to very successful outcomes.

As for the quality of student support by the tutors, the answers obtained from the evaluation were mixed, which is not surprising as one would expect each of the three tutors to have different strengths and weaknesses. Although in aggregate, the learning support provided by local facilitator was rated fair, it seems that the local facilitators were nonetheless key actors in the delivery of the course. The experience in Vietnam, where the tutor/local facilitators voluntarily organized learning activities a day before the videoconference sessions, illustrates how much more effective the course can be in the presence of highly motivated local facilitators. The face-to-face sessions organized in Viet Nam were particularly helpful for regular participants, who constituted the majority of participants. Since online tutoring was only available to the TOT participants, the face-to-face sessions provided an opportunity for the local facilitator to help regular participants with the course material. TOT participants likewise used the sessions as an opportunity to share model answers for the course assignments with the regular participants. Not surprisingly, in the case of Vietnam the local facilitators were rated very favorably. Table 7 [ PDF 95.6KB | 1 pages ]

Accreditation

The possibility of accreditation attracted a variety of high quality participants from different backgrounds who shared an interest, commitment and involvement in microfinance. Accreditation provided a strong incentive for participants to undertake the heavy course workload. In concert with the personalized online tutoring, peer learning environment, and networking, accreditation contributed to the MFTOT’s high completion rate. Accreditation provided the participants with the credentials necessary for their future activities in microfinance in the region. This would not have been possible if the accreditation were not backed up by the solid credibility and reputation of the accrediting organizations.

Drop out

The MFTOT was made available to participants free of charge and participants were free to drop out from the course without any consequences. Out of the 36 effective TOT participants who took the course, 3 dropped out in the middle of the course and one did not complete the final exam. The effective drop out rate was 12%, a relatively low rate when compared to the drop out rate of between 20-50% in other distance learning courses.

The reasons given for dropping out of the course were: 1) illness, 2) travel on job assignments, 3) language problem, and 4) time constraint due to work schedule and taking other distance learning course. Except for the language problem, it could be concluded that the reasons were mainly external to the course.

3. Balancing Access with Equity

While the MFTOT seems to have done well in terms of improving access without sacrificing quality, the evaluation results suggest that it has been less successful in terms of balancing access with equity. There appears to be two key obstacles to achieving equity in the MFTOT: the language barrier, and limited access for ruralbased microfinance practitioners.

Since the MDFL materials and the VCs were in English, in non-English speaking countries, it was clear that the beneficiaries of the course were the privileged groups who may not lack opportunities in capacity building. For those with a good command of English, the MFDL materials requires 10-20 hours of self-study and homework. In the case of participants for whom English is a second language, the study time significantly increased. In the VC sessions the language problem also manifested in the form of participants not understanding the English accents of other participants from different countries, which limited effective discussions.

In view of this, it is not surprising that an overwhelming majority --- close to 89% of the respondents ---- agreed that there was a need to localize the course materials. Since the underprivileged microfinance practitioners are rural based and may not have good command in English, in order to effectively reach them, the course will have to be translated into local languages. The request for localization even came from participants from countries such as the Philippines where English is widely used.

Since the VCs were offered through the GDLN Centers which are located in the capital cities of the countries, there were more urban-based participants in the course. Nevertheless, since the VCs were held only once a month, a number of rural-based participants managed to organize their time to successfully participate in all the VCs and complete the course.

4. Ensuring the Use of Appropriate Technology

The MFTOT combined three types of technologies – CD-ROM, online tutoring and discussion, and two-way interactive videoconferencing. Most of the course content and instruction was given using the interactive CD-ROM, with the course instructor managing the assignments and the exam by developing a “road map” with clearly identified deadlines for submission of the assignments and final exam as well as for providing feedback and grades. The online tutoring provided the necessary learning support. The videoconferences covered current and sometimes controversial microfinance topics and presented thought-provoking country case studies, thereby stimulating discussion and debate which spilled over into the online discussion between MFTOT participants.

As discussed previously, the course was rated highly in terms of the CD-ROM and somewhat less favorably in case of the online tutoring, although it should be noted that in the case of the later, the problem had less to do with technology and more with the effectiveness of each individual tutors.

With regard to the videoconference sessions, the technology used was rated favorably in the evaluation. This is not surprising as the GDLN system was designed to administer distributed classrooms, with resource speakers in one continent and participants in various developing countries. Through such facilities, participants were able to access high quality lectures and insights from internationally renounced experts, thus shortening their learning curves.

The participants, however, gave a fair rating with respect to the extent to which the VCs enhanced self-study and online learning. It is understandable that participants could have expected the VCs to explain the course materials as in a traditional classroom learning. However, the VCs were designed to complement the selflearning materials by highlighting key debating issues.

Notwithstanding the fair rating given to the VCs, a number of participants listed the interaction facilitated by the VCs as one of the elements they liked most about the course. In particular, two topics presented in the VCs, i.e., financial analysis and interest rate, were identified by the participants as two of the course’s strongest elements.

Apart from the VCs, answers to the open questions revealed that participants expected a greater opportunity to interact with the tutors as well as other participants through e-discussion forums.

Overall, therefore, it seems that the technologies adopted in the MFTOT were largely appropriate for the level of ICT infrastructure and skills of participants. Providing a clearer explanation of the VCs’ objectives and the inclusion of a moderated ediscussion forum could further improve the course.

5. Ensuring Long-Term Institutional Building

The MFTOT was successful in capitalizing on the strengths of development institutions in both developed and developing countries to organize the course. Overall, this element received an average rating of 4.33. The MFTOT benefited from the unique contributions of the three development organizations in managing the course. It also benefited from the participation of microfinance practitioners and experts in Asia who willingly and at times voluntarily contributed to the lectures and the organization of the course.

Tutors engaged for the 2^nd course were graduates from the 1^st course who could immediately utilize their skills in supporting new participants. The tutors themselves admitted that the opportunity to tutor others reinforced their own understanding of the subject matter and increased their confidence as a regional trainer.

Since more than half the TOT participants were from education/training institutions, the institutional building aspect was incorporated.

6. Overall Rating of the MFTOT

Despite the fact that some elements of the course did not receive an excellent rating, overall the results of evaluation indicate that participants found the course relevant and satisfactory. Overall, the course received an average rating of 4.18 from the respondents. The average rating given by new TOT participants was slightly higher at 4.22; regular participants gave an average rating of 4.13. Asked if they would recommend the MFTOT course to others, 88.71% of respondents answered affirmatively.

Download this Discussion Paper [ PDF 407.8KB| 38 pages ].

Comment(s)

There are [0] comment(s) for this entry. Post a comment.