The Southeast Asian Ministers of Education Organization, Regional Centre for STEM Education, with support from Chevron Corporation in partnerships with SEAMEO RECSAM, CPRN, KSP, and TEDC, continued spearheading the Lecture Series on Introduction to Program Effectiveness Research which aims to provide a broad overview of how to conduct rigorous program effective research in education. The third topic of the said series was entitled “Lecture 3: Estimating and Interpreting Treatment Effects” held on October 8, 2021 via Zoom. Hosted by Ms. Yaoowalak Jittakoat, the webinar commenced at 6:00 (GMT +7 Bangkok Time) and was concluded with an estimated total number of 300 participants globally.
Dr. Brian Rowan, Emeritus Professor of Education, Emeritus Research Professor at Institute for Social Research, University of Michigan, served as the main speaker for the said event. On the third installment of the lecture series, he focused on how to estimate and interpret various types of program effects in order to examine the confirmation status of a given program logic model. The focal point of his discussion was the conceptual understanding of the overall topic rather than a statistical analysis.
During Dr. Rowan’s talk, he defined the different kinds of treatment effects discussed in the causal inference literature, specifically the: (a) the intent to treat (ITT) effect; (b) the effect of actually receiving treatment (TOT); (c) conditional average treatment effects (CATE); and (d) the average causal mediation effect (ACME). He emphasized that these different treatment effects can be used to examine the confirmation status of a program logic model. Moreover, he provided demonstrations of these treatment effects for deeper understanding of the participants.
Dr. Rowan recalled the program effectiveness research as “examining of the confirmation status of a program logic model, where the logic model consists of a set of hypotheses about how a service delivery model (the processes then affect a set of final program outcomes.” He also added that program logic model for an intervention, through professional learning communities (PLCs), is used to improve student learning. This model has three components, namely: 1) a service delivery model defining the “treatment” to be given to enrolled units; 2) an implementation model that describes the expected response as a result of the treatment; and 3) a set of final outcomes the program targets to achieve at the end of the intervention.
During the open forum, Dr. Rowan instilled to the attendees that “When you (attendees) conduct a research, you work extensively for the missing data. You can never force someone to comply with research protocols. There are many ways of performing statistical estimation among variables in light of missing data.”
He also affirmed that “Interventions can be unique… It is the uniqueness of the set of services you deliver compared to the ones being delivered in the normal world.” In line with Assumption vs. Sample of Population in research works, he stated that the former is usually a sample of population and there is a larger population that would have been (but they were not included.) If one has a population of everyone, the parameter is generated and not just an estimate.” He ended his talk by reminding everyone to keep on collecting (relative data) even in time of non-compliance of the group in order to accomplish the set outcomes of the chosen intervention.
The last topic for the lecture series will focus on “How to Report on Program Impacts” anticipated to happen on October 22. Meanwhile, the first two topics were entitled “Some Logical Underpinnings of Program Effectiveness Research” and “Research Designs to Create Counterfeits” conducted on August 27 and September 9, 2021, respectively.
The LCC Secretariat expresses its appreciation to the organizers for their invitation and facilitation of a comprehensive and rich lecture series on program effectiveness research. Convergence such as this brings together different literacy stakeholders and research enthusiasts which indicates call for an action. Through this, the importance of data-driven interventions and boosting of research activities in the local communities are instilled to everyone yielding to a better nation building through the power of literacy.
Jem Beryline Bualat