Making Recommendations for Program Evaluation Based on Evaluability Assessment Results: A Brazilian Case

Topic: 
Education - Health

In Brazil, the National Program of Informatics in Education (ProInfo) was created to improve the quality of public schooling through the utilization of information and communication technology for education (ICT4E), with the goal of democratizing access to ICT. In December 2009, the UNESCO office in Brasilia contracted four consultants to do an evaluability assessment of the ProInfo and other ICT4E programs. I produced five products over an eight month period, and the other team members produced several preliminary products as well. The primary purpose of the evaluability assessment was to make recommendations for further program evaluation.

Evaluability Assessment in Action

Recommendations were grouped mainly by efficiency, effectiveness and impact. These represent three of the five DAC Criteria for Evaluating Development Assistance, of the Development Assistance Committee of the Organization for Economic Cooperation and Development (1991). Efficiency has two different connotations: (1) the provision of goods and services (inputs and outputs) at minimal cost and, in education, (2) proper student flows (minimal dropping out, grade retention and grade repetition, and correct age-grade correspondence). With regard to ICT4E, the first concern of the Ministry of Education must be the delivery of goods and services, for example IT equipment, as well as TV/DVD and projectors, television programming and signal, digital instructional objects, VHS tapes, DVD’s, broadband or another alternatives for broadcasting, and training.

Likewise, effectiveness has more than one connotation. A program is effective if it meets its objectives (intended outcomes). For international projects in the Latin America and the Caribbean region, this generally refers to capacity development. For its part, school effectiveness is evidenced by good academic achievement despite unfavorable socioeconomic indicators. If one thinks of effectiveness primarily in terms of capacity development, social capital, organizational learning, etc., one may decide to examine, among other things, communities of practice; communities of learning, research and practice; and learning networks, all of which typically involve internet-based communication. The digital collaborative learning environment called eProinfo provides a platform for teachers and others to share their work and compare notes, and can be researched using web analytics. Digital records can be used to track communications, either directly by website or through a meter for e-mail. Also recommended was triangulation, using methods such as questionnaires, interviews, focus groups and observation, to assess knowledge, attitudes and practice.

Impact in education is defined mostly in terms of student learning. Although it is often asserted that the impact of ICT4E cannot, or has not, been measured, Zwingenberger (2009), in a review of 26 research studies on the impact of the use of multimedia instructional resources on learning published between 1991 and 2004, concluded that the effect size was in a narrow band around 0.30. School level analysis using the national assessment, Prova Brasil, which utilizes item response theory to guarantee the equalization of tests applied at different times (sharing selected items), may make possible the analysis of trends related to phenomena discovered by on site observation and interviews. One school in the capital of the western state of Acre has demonstrated particular strengths in ICT4E; its gains in mathematics and Portuguese from 2005 to 2009 were found to be correspondingly greater than the national average. Numerous schools may be examined nationwide in such a context. One limitation to such analysis, however, is the cross-sectional nature of the comparisons. At the same time, Projeto GERES (Longitudinal Study of a School Generation) has been analyzing achievement in a small sample of schools, following the same students over several years; focusing on these schools may permit even more robust conclusions regarding program impact.

To what degree does proper utilization of ICT4E improve school efficiency and effectiveness, and consequently student learning (impact)? One question asked of students taking the Prova Brasil is, “Is there a computer in your house?” Sub-items also ask about internet access. Through analysis of results by school, the relationship between affirmative responses and performance on the Prova Brasil may be ascertained; socioeconomic status may also be controlled statistically. The possibility that some students who have no computer or internet access at home compensate for this shortcoming at school, or at a local LAN (Local Area Network) house, may be investigated. For example, such as the one in Acre, when the school has a blog and teachers promote its educational utilization, does achievement improve? Brazil’s youth are in the vanguard of internet utilization, according to research in Ibero-America by the Telefonica Foundation (Telefonica, 2013). To what extent has the program helped educators to use this phenomenon to promote learning?

Additionally, in our field visits, it was observed that in some schools the computers are used mainly by the teachers. In instruction, they may sometimes be used for teaching computer science or as an aid to classroom lectures. As the evaluability assessment was being conducted, a new experiment, “One Computer per Student,” was underway in a small sample of schools nationwide. The program has since been expanded, and it now appears that it may be appropriate to compare these two initiatives.

Evaluator Reflections

To date, the recommendations of the EA have not been implemented. Staff changes at a critical time in the project stymied efforts to present the EA findings to key decision makers. Perhaps if the Secretariat of Distance Education (subsequently abolished, with ProInfo now under the Secretariat of Elementary and Secondary Education) had participated more and feedback loops had been shorter and more frequent, the findings would have been seen as pertinent for immediate application. On the other hand, comprehensiveness might have been sacrificed.


References:

Organization for Economic Cooperation and Development. (1991). Principles for evaluation of development assistance. Retrieved from http://www.oecd.org

Telefónica. (2013). Telefónica Global Millennial Survey, today’s young adults: The global leaders of tomorrow. Retrieved from http://survey.telefonica.com/survey-findings/Telefonica,%202013

Zwingenberger, A. (2009), Die Wirksamkeit multimedialer Lernmaterialen. Kritische Bestandaufnahme und Metaanalise empirischer Evaluationsstudien. Munich, Waxman.

Share this