Call for Proposals: Indonesia-Based Research Organization to Support an Impact Evaluation Design and Conduct a Baseline School Survey
|Project||High-Touch High-Tech Pilot in Junior Secondary Schools in Indonesia|
|Report to||ADBI task manager|
Urban or peri-urban areas; specific location to be determined
|Contract Duration||March – December 2022|
Differentiated teaching, in which students are taught according to their actual levels of understanding and provided tailored learning paths, is key to ensuring learning. However, student foundational skills are often widely distributed. Implementing differentiated teaching to the whole distribution of abilities is challenging even for highly experienced, skilled, and motivated teachers. For average teachers, attempting to do so would cause them to spend virtually all their time teaching only foundational skills; they would have no time to teach higher-order skills.
Technology can help address this constraint. Recent evidence shows that personalized technology-aided instruction programs yield large absolute learning gains for all students, with greater relative gains for academically weaker students. More importantly, such programs can cover the whole distribution of students’ initial skills. With technology’s help to develop students’ foundational skills, teachers can have more time to help students develop higher-order skills.
A recent development in technology-aided differentiated teaching is the high-touch high-tech (HTHT) approach. The HTHT approach is comprised of two components:
- A high-tech approach equips students with foundational skills. In this component, students interact directly with computer software. The software assists teachers in teaching foundational skills in two ways: (i) it improves the learning of students in the whole initial distribution of foundational skills; and (ii) it reduces the amount of time teachers must spend teaching foundational skills by providing test results, diagnostics, and learning modules.
- A high-touch approach equips students with higher order skills. Since teaching these skills requires close interaction between teachers and students, together with appropriate pedagogical techniques, this component is comprised of a series of training programs for teachers to enable them to implement high-touch teaching.
Although this approach has become common in high-income countries, it has yet to be widely implemented in low- and middle-income countries. More rigorous evidence for its value is needed, especially in cases where teacher skills and motivation are low.
Indonesia’s Ministry of Education, Culture, Research, and Technology (MoECRT) is currently developing a technology-aided differentiated teaching system for the country’s whole primary and secondary level education. MoECRT is interested in exploring a high-tech direct-to-student approach pilot to complement the country’s in-house development.
The Asian Development Bank Institute (ADBI), in partnership with MoECRT, the Asian Development Bank (ADB), Tanoto Foundation, and J-PAL Southeast Asia (SEA), has engaged an education technology (EdTech) company and a local research organization as consultants for designing and contextualizing this HTHT approach for 7th grade mathematics since 2021. The approach mainly consists of two components: (a) developing an AI-based software and training program for students’ foundational mathematics skills and training teachers on its use (high-tech); and (b) developing another teacher training program on pedagogical approaches to foster students’ higher-order skills (high-touch). These components will be field-tested with students and teachers at selected schools so that these interventions will be ready for large-scale implementation in selected public schools for a whole school year. An impact evaluation will also be conducted to evaluate the impacts of these interventions on student achievement.
Through this call for proposals, ADBI is looking for a research organization based in Indonesia, or a consortium whose lead organization has legal status in Indonesia, to contribute to the impact evaluation design and to implement a baseline survey of schools, teachers, and students in June–September 2022. The organization will closely collaborate with the organizing partners, the other consultants hired for the project, selected schools, education districts, and local stakeholders.
Scope of Work
The evaluation method will be a randomized control trial (RCT). The project will indicatively include 100 treatment and 50 control schools to be surveyed. In each school, randomly selected students will be assessed. The geographical coverage of the evaluation will be urban or peri-urban areas in districts that will be determined later. The impact evaluation will focus on the following research questions:
- What is the impact of the high-tech high-touch approach on teacher motivation, teaching practice, and teacher efficacy?
- What is the impact of the high-tech high-touch approach on students’ foundational mathematics skills?
- What is the impact of the high-tech high-touch approach on students’ higher order skills?
- Are there any heterogeneous impacts of the high-tech high-touch approach based on student socioeconomic status, sex, and initial mathematics skills level?
- What is the additional impact of the high-touch intervention over the high-tech intervention alone?
- What is the additional impact of the high-touch intervention on students’ non-cognitive skills and passion for learning?
The consultant will contribute to the impact evaluation design, based on a theory of change developed prior to the baseline survey. This includes developing necessary survey instruments. The consultant will then play a leading role in implementing the survey. Specifically, the consultant will:
- Play a contributory role in the theory of change, sampling strategy, and overall evaluation design led by ADBI, J-PAL SEA, and in collaboration with other partners;
- Play a contributory role in the preparation of registering the study in an RCT registry, such as the AEA RCT Registry;
- Lead the preparation, submission, and acquisition of ethics review approval from a reputable university in Indonesia
- Lead the drafting, field-testing, and finalization of survey instruments;
- Lead the recruitment and training of enumerators;
- Lead the implementation of the baseline survey in the control and treatment schools;
- Play a contributory role in analyzing data and writing a baseline report.
The tentative impact evaluation timeline is below. Note that the timeline has minimal flexibility.
- Theory of change, sampling strategy, and evaluation design: March/April 2022
- Sampling: April 2022
- Survey instruments and field testing: April/May 2022
- Baseline survey: July/August 2022
The implementation of the teacher training programs, HTHT interventions, and endline survey is not included in the scope of this call for proposals.
Prior to implementing the survey, the consultant must assist in applying for an ethics review and receiving such approval from a reputable university in Indonesia.
- Final survey instruments and operational manuals. The instruments will be co-designed with ADBI and J-PAL to measure student foundational and higher-order mathematics skills and subjective outcomes. The survey instruments must be pilot tested at least twice in around five schools with a similar context as the survey target schools. From previous experience, the time needed for a team of two enumerators to collect the above data is two working days per school. (Deadline 31 May 2022).
- Evidence of ethics review board approval. (Prior to survey implementation).
- Survey report, covering aspects such as enumerator recruitment and training; issues faced during implementation and the solutions; and overall sample sizes. The maximum length of the report is 10 pages. The report is due 1.5 months after the survey is completed. (Deadline 15 September 2022).
- Cleaned and ready to use datasets of all schools and individuals that participate in the survey. The student and teacher datasets must be designed to allow merging between each other, and with the school dataset. The baseline survey must be designed such that the dataset could be merged at the individual level and school level with the endline survey that will be implemented in the future. The datasets must be in Stata format and accompanied by a document that describes all the variables and a method to merge different datasets (within a survey and between surveys). (Deadline for first submission: August 2022; Final submission September 2022)
Each deliverable will be reviewed by ADBI. Feedback, if any, will be provided within two weeks of submission. The revised deliverable which incorporates the feedback must be submitted within two weeks after receipt of feedback.
The task leader reports to the ADBI task manager on a weekly basis.
Interested organizations or consortia should submit a technical proposal written in English, with normal fonts (e.g. Times New Roman, 12-pt, 1.5 spacing) and 1-inch page margins on an A4 paper setting.
The proposal should be a maximum of 10 pages in length and contain the following three sections:
- Proposed general approach to deliver the scope of work, which describes indicative survey methods, instruments, and fieldwork plans (max 2 pages)
- Proposed number and composition of enumerator teams, enumerator and supervisor recruitment strategy, and approach to enumerator training, data entry, and cleaning strategy (max 5 pages)
- Organization profile, list of relevant experience, including a list of relevant publications, composition of the team that will undertake the consultancy, and profiles of the proposed team leader and team members (max 3 pages)
As part of the proposal submission, the following must be attached (English is preferred, but Bahasa Indonesia is accepted if getting a translation is time consuming or requires additional cost):
- Evidence of legal status in Indonesia (if a consortium, then the lead organization’s legal status in Indonesia)
- Evidence of independently audited financial statements for the last three years (if a consortium, then the financial statements of the lead organization)
- Statement confirming the implementation of standard operating procedures related to: (i) child protection; (ii) data safeguards; (iii) integrity/anti-corruption; and (iv) staff code of conduct. ADBI may request of a copy of such policies prior to contract signing.
The aim of the proposal is to demonstrate whether the consultant would be able to deliver the scope of work. As such, a financial proposal is not needed at the submission stage.
Interested parties should submit their proposal and attachments as a single PDF file to: Junko Mitsuhashi, on 4 February 2022 by 5:00pm Tokyo time. Please use: “HTHT Baseline Survey Proposal_[Firm Name]” in the email subject line.
Proposals will be assessed based on the following criteria:
- Extensive experience in conducting research in Indonesia, with education research experience highly preferred
- Extensive experience in managing and implementing primary data collection, especially in schools or households
- Experience in designing impact evaluations, with experience in conducting randomized control trials highly preferred
- Experience in working with donor agencies, international organizations, or charitable foundations
- Experience in establishing, and maintaining productive working relationships with government officials and/or employees
- Track record, skills, and experience of the task leader and team members with regards to: (i) research, specifically related to randomized control trials; (ii) program management, and; (iii) English academic writing and speaking skills
Inquiries should be directed to Daniel Suryadarma.