Call for Proposal:
Final Evaluation Consultant for MECIHO Project
June – August 2024
1. Background and Context
Building upon the success of Kopi Nande Project, Lutheran World Relief (LWR) was awarded a grant from the Starbucks Foundation to implement a 36-month MECIHO (Multisectoral Collaboration for Inclusive Health & Opportunities for Women) Project in six coffee-producing communities in Karo District, North Sumatra Province, Indonesia to address the challenges faced by the communities in hygiene, water, and sanitation as well as to empower women in leadership roles to enact positive changes in their communities. The consulting service, for which this Terms of Reference (ToR) is provided, is required to conduct the final evaluation for MECIHO.
The project is using innovative methodologies of awareness-raising and behavior change, enhance existing government-led programs on health and hygiene, and using a women-led approach for improved health and hygiene in the communities. The result of the study will help the organization in improving its strategy in women empowerment and WASH issue. The final evaluation will also include end line study, where the project will compare the evaluation result with the baseline.
The ToR outlines a brief description of the project, the objectives and scope of work, and required qualifications. The contract duration is for three (3) months, starting from the date of contract signing.
Working in six coffee-producing communities of the Karo District of North Sumatra, Indonesia, the MECIHO program aims to improve the water, sanitation, and hygiene (WASH) conditions of the community and individual household level through innovative, participatory means led by women. The MECIHO program's theory of change is based on the belief that women's leadership in health and hygiene matters. Thus, the project strengthens and infuses existing women's groups with new skills and knowledge on health, hygiene, and leadership, empowers them to disseminate the skills and knowledge to their respective communities, and encourages them to step up in the village decision-making process so that they can voice their aspirations. In doing so, they will take on a greater role in transforming their communities into healthier places to live. This women-led approach will create sustainable behaviour change by combining trainings and promotion with tangible improvements in the targeted communities. When communities begin to see the benefits of these improvements, they will be more likely to prioritize health and hygiene.
The goal of MECIHO Project is to improve the health, hygiene, and living conditions of households and communities in six target communities in the Karo District of North Sumatra. The six communities are: Sarimanis, Tanjung Barus (Barusjahe Sub-district), Jaranguda, Cinta Rakyat (Merdeka Sub-district), Nang Belawan, and Ndokum Siroga (Simpang Empat Sub-district).
The project has two inter-related outcomes, which are theorized to contribute to the goal. The two outcomes are:
1. Improve the mechanisms and systems that support the advancement of healthier homes and better living conditions.
MECIHO promotes "small doable actions" to reinforce behaviors that support healthy homes and communities. WASH campaign messages will be refined based on findings from WASH mapping. The project trains and supports WASH promoters in both schools and communities and produce a radio campaign with messaging around hand washing and latrine use, promotion of opportunities to install private latrines, and opportunities for participation such as contests and call-in shows. In addition, the project also initiates and strengthens garbage banks to promote waste management at the village level.
2. Strengthen women's leadership on and engagement in governance initiatives.
MECIHO provides leadership training and training of trainers (ToT) in hygiene promotion and waste management along with opportunities to engage in local governance processes for village-level Women's Champions. The Champions and other community members conduct WASH mapping to assess current hygiene practices and behaviors, functionality of existing facilities, WASH barriers, and gender and social norms
2. Evaluation Scope
The objectives of the evaluation are to assess:
• The achievement of projects' outcomes based on their respective set of indicators/ targets to date as well as the projects key evaluation questions.
• The project performance based on OECD-DAC evaluation criteria.
• Analyze the project approach, strengths and weaknesses; issue and challenges of major components of this project and provide recommendation for future
The assessment will be based on OECD-DAC criteria given below.
• Relevance or appropriateness of the project: IS THE INTERVENTION DOING THE RIGHT THINGS?
To assess the extent in which the project interventions goals and implementation are aligned with beneficiary and stakeholders needs by investigating if they view the interventions as useful and valuable. Where applicable, the assessment of relevance should be analyzed based on its four elements of analysis:
• relevance to beneficiary and stakeholder needs,
• relevance to context,
• relevance of quality and design, and
• relevance over time.
• Coherence: HOW WELL DOES THE INTERVENTION FIT?
To assess the compatibility of the project intervention with other interventions in the target geography, be it interventions delivered by government institutions or other development actors. Coherence also looks at the consistency of the project intervention with relevant international norms and standards especially in the context of water, sanitation, and hygiene (WASH) ang gender empowerment. It should be analyzed based on two dimensions:
• Internal coherence – how well are the project interventions align with the wider policy frameworks of the local or national institutions responsible for implementing the development interventions in the Karo region. The assessment should look at how the project interventions harmonized with local development activities or if there are any duplications, how they complement. Internal coherence also provides a lens for assessing inclusion.
• External coherence – how well are the project interventions align with external policy commitments such as the SDGs and how these are taken into consideration in the interventions design and implementation. External coherence is also concerned with how well the interventions connect with interventions implemented by other development actors in the Karo region, if there are any.
• Effectiveness: IS THE INTERVENTION ACHIEVING ITS OBJECTIVES?
To measure the extent to which the project interventions achieved or likely to achieve its desired outcomes and results. The assessment should provide insights into:
• The extent the project has attained its planned results by comparing its actual achievement with its targets
• An evaluation of the process by which this was done
• Which project components were decisive or important to the process
• Whether their were any unintended effects
Effectiveness is different from impact as it is more concerned with the most closely attributable effects, while impact examines higher-level effects and broader changes. Assessment of effectiveness should also look at how well it allowed for participants participation in the implementation of the project interventions and developing their ownership and accountability especially during the process evaluation. It should also look at how the project evolve and adapt and how the design changes, if at all, based on feedback from stakeholders, emerging results, and changes in context.
• Efficiency: HOW WELL ARE RESOURCES BEING USED?
To measure the extent to which the intervention delivers, or is likely to deliver, results in an economic and timely way. The assessment should look at whether the project use of resources justifies the results it achieved. The focus of this assessment should be in
• Operational efficiency – How well are resources used during implementation. It reviews if human and financial resources were used as planned or redirected as needs change. It should also looked at how well the project managed its risks.
• Timeliness – The assessment should look at whether and to what extent the results were achieved within the intended timeframe. It should review and check if the timeframe and work plans were realistic and appropriate, and if the changes to the timeline are necessary given the many external factors that could affect the implementation of the project.
• Impact: WHAT DIFFERENCE DOES THE INTERVENTION MAKE?
What were the higher level changes or effects both positive and negative or intended and unintended on the community, beneficiaries, gender and village groups/SHGs; What are the community/beneficiaries' perspective on the immediate and intermediate effects (quotes can be collected to provide evidence on participant's perspective). To what extent is the project contributing towards achieving transformational change especially within the context of WASH?
• Sustainability: WILL THE BENEFITS LAST?
Measure the extent to which the project results and impacts or positive outcome at the community level are sustainable in the longer terms (quotes can be collected that can provide evidence on participant's perspective).
The total population covered by this study is 12,216 with 6,013 male and 6,213 females.
3. Objective of the Engagement and Scope of Work (SoW)
The evaluations of the MECIHO Project will rely on data collected as described in this TOR. Each data collection activity will form the basis for measuring outcomes for program beneficiaries as well as non-beneficiaries, which is critical for assessing program impact. The Consultant should work in close coordination with Corus team to carry out the study. The Consultant is expected to assist in the evaluation by collecting primary and secondary data needed by the evaluation. The SoW of the Consultant includes a base tasks, which are a set of activities that the Consultant is expected to deliver and optional task, which is a set of activities that could be exercised by Corus.
Evaluation Design
The evaluation of MECIHO Project will use Propensity-Score Matching (PSM), a quasi-experimental method used to calculate the impact of the intervention by estimating the difference in outcome between beneficiaries and non-beneficiaries of the program. Since the selection of treatment villages were non-randomized, PSM compare treatment effects across program beneficiaries and matched non-beneficiaries. Matching of is done by observing and analyzing a range of observed characteristics. A propensity score is computed for each observation. Once all of the scores are computed, the observations in the treatment group can be matched with observations in the control group that have the closes propensity score. Through this method, a counterfactual is constructed.
The pool of observations where the control group will be drawn will come from an additional six villages. These six villages will potentially be included in the next phase of the MECIHO project.
Base tasks
These tasks include primary and secondary data collection that include:
Secondary Data
Desk Review
The Consultant will complete a desk review of key program documentation including project design documents, monitoring and progress reports, and other relevant publications. The purpose of the desk review is twofold:
i. to better understand program operations on the ground.
ii. to become familiar with existing analytical frameworks used to evaluate such types of programs; and
iii. to extract information needed to report against key evaluation questions.
Photo Voice and Real Reel Analysis
The Consultant will complete a review of the output of the Photovoice and Real Reel activity. The purpose of both activities is to add additional qualitative insights into the study. The Consultant is expected to include the analysis in the final report.
Primary Data
The Consultant is expected to lead a team of data collectors that will collect primary data needed by the evaluation. Primary data will be collected through quantitative and qualitative method as described below.
Quantitative Study
Survey will be administered in both the treatment and control villages. During the baseline, the survey was able to complete xx number of interviews. For the end line, a sample size of xxx is proposed for each group thereby resulting in xxx total interviews. The evaluation firm can propose a different sample size as long as satisfy the requirements of 95% confidence level and 5% margin of error. The preference is to use probability proportional to size sampling (PPS) in identifying samples but alternative method can be suggested as well.
The draft questionnaire in English will be provided to the Consultant after Contract signing. The Consultant is expected to translate, pre-test, and finalize the questionnaire. Enumeration using KoboCollect app or any survey app running on Android devices is the preferred method over a pen-and-paper survey.
Qualitative Study
The Consultant is expected to conduct qualitative study as well. The qualitative study is intended to provide better insights into the changes that happened in the community as well as to inform policymakers and implementers about how the project were implemented and maintained. As such, the qualitative study is envisaged to go deep into the discussions rather than go wide and cover more villages. The Consultant is expected to implement two approaches: FGD and KII. The proposed number for each method per project is given below:
FGD = 2
1. Women leaders in the treatment group
2. Women leaders in the control group
KII = 6
1. Representative of a successful garbage bank group
2. One female leader in the treatment group (not included in the FGD)
3. One female leader in the control group (not included in the FGD)
4. One male leader in the treatment group
5. One male leader in the control group
6. One district level government official
Qualitative data shall be captured using key notes using the template that will be provided.
Description of Tasks
Task 1 – Write Inception Report and Work Plan
The Consultant shall submit an inception report and work plan that includes the following:
• Understanding of the engagement and tasks required
• Substantive input into design of draft data collection instruments, methodology and overall study approach
• Team composition and recruitment plan
• Numbers, dates, duration, and location of training sessions; including composition of training teams, agenda, preparation of logistics, and reporting arrangement to the LWR Evaluation Coordinator
• A list of permits and letters required to conduct fieldwork and when those will be obtained
• Expected duration of survey implementation and work plan for Field Teams including supervision plan
• Expected date and duration of the Data Cleaning activities, including delivery dates of the different datasets
Task 2 – Prepare for field work
Preparation for field work entails a number of subtasks that must be completed in order to ensure the quality of data that will be collected. It includes:
• Recruitment and selection of field teams and data processing teams.
Interviewers and facilitators will be responsible for collecting the data in the field. Interviewers that will be selected must have demonstrated experiences in administering surveys or facilitating discussions. Each field team must be headed by a field supervisor which will oversee the over-all management of field work of the team including backchecking and other data quality processes on field. Data processing team are responsible for ensuring that data collected in the field are free from errors prior to analysis. They conduct the data transformations as required by the Consultant.
• Prepare research permits
In the event that the community or municipality will require permits for field work, it is the responsibility of the Consultants to obtain these permits.
• Finalization of data collection instruments
All data collection instruments must be tested before being used. Testing procedures could be pilot testing or cognitive pretesting. The objectives of the testing are:
o allow the Consultant to familiarize itself with the instruments,
o test the logistics (such as field mobilization and demobilization, potential locations of basecamps) and time necessary for the surveys
o test the draft questionnaires. Testing the draft questionnaires will include identifying any exercises/questions that are not working as currently framed.
• Familiarize with the use of KoboCollect or alternative
LWR prefers data collection using KoboCollect. Consultant must familiarize themselves with the use of the software before field work.
• Training of Interviewers and Facilitators
LWR strongly prefers that all field team members be trained in one central location. If Consultant proposes training in multiple locations, the proposal should detail how the service provider will ensure consistency across training locations and common understanding of questionnaires and data collection protocols.
Note that the Consultant is required to begin fieldwork within one week of the end of training. If there is a delay of more than one week, the consultant will be required to do a one-day refresher training to all interviewers at its own expense.
A short training report should be submitted by the Consultant within a week after the training.
Task 3 – Collect Data
Data collection effort should always follow government advisories in preventing the spread of COVID-19. In the event that face-to-face data collection cannot be conducted due to prevailing government restrictions, the Consultant should attempt to collect data using indirect method. Indirect method could include interviews over mobile or social messaging apps.
• Data Collection (Quantitative)
LWR will provide the Consultant with a list of households to interview and a replacement strategy that should be strictly followed. The households will be the same households that were visited during the baseline. Upon arriving in each village, the survey team should go to the village hall and attempt to verify whether the listed households still reside in the village. Field teams should preferably interview the same respondent who was interviewed during the baseline (for budgeting purposes, the Consultant should assume that 20% of respondents will be revisited).
The field teams are expected to adhere to the respondent tracking process:
1. If the respondent is not home during the time of the interview: The interviewer should return at a different time and if the respondent still cannot be reached during this second attempt, the interviewer must make a third attempt at a different time/day to find and interview the respondent.
2. If the respondent moved village but within the same district: The interviewer must trace the respondent within the district and conduct the interview.
3. If the respondent moved outside the district: Replace the respondent with another one according to the replacement strategy provided by LWR.
Ideally, no respondents from the previous survey round should be replaced. However, replacements should be addressed on a case-by-case basis. The Consultant should inform LWR about any such instances before reaching a decision.
• Data Collection (Qualitative)
Appropriate venues to conduct FGDs must be identified and booked prior to the activity. Invitations for participants to participate must be sent out a couple of days before the discussion. In the event that, less than 4 participants showed up for the FGD, the activity will still continue, with the facilitator adopting KII approaches. For KIIs, appointment must be booked in advance and reconfirmed a couple of days before the interview. Data recording, if allowed, and extensive note-taking should be done for each activity.
• Data Quality
The Consultant is responsible for collecting the highest quality data possible and subjecting interviewers, facilitators, and data to great scrutiny to ensure quality. It is recommended Consultant implement three kinds of data quality checks.
o Observation: Supervisors accompany the interviewers while they are doing the interview to observe and support interviewers. With this type of checking, supervisors will find out if the interviewer understands interview questions, effectively conveys them, and can coach the interviewer on how to improve if problems arise. Supervisors must accompany interviewers for at least 10% of interviews. The Consultant is expected to submit a report tracking the observation made.
o Supervisors are required to reinterview the respondents over mobile phone and ask some of the (sample) questions in the questionnaires to the same respondents. The purpose of this procedure is to verify that interviewer actually met the respondents and conducted the interview. If there are significantly different answers between the two interviews, supervisors should clarify the difference. If the difference is because the interviewer wrongly posed the question or misunderstood the question, supervisors should brief the team to clarify the misunderstanding. If any interviewer is found to be falsifying or manipulating data, he/she must be fired and replaced. Replacement interviewers must re-interview all respondents interviewed by the interviewers who falsified data. The Service Provider must ensure that at least 5% of interviews are audited (i.e., households revisited). The Consultant is expected to submit a report Detailing the result of the re-interviews.
o Checking the completed interview: supervisors will check 20% of completed interviews to identify errors made by the interviewers. The checks can be done through the KoboCollect online facility and should be done while the team is on the field interviewing.
• Methodological Report on field work implementation
The Consultant shall write a methodological report of no more than 5 pages (excluding annexes) detailing implementation of the fieldwork, any issues and problems that arose and how they were managed.
.
Task 4 – Data cleaning, analysis, and draft report
Data processing team is responsible for starting the data cleaning process as soon as they receive data and other supporting documents from the field. This process includes checking the accuracy of data received, such as respondents' identification and linkages among questions (meaning the skip patterns and relationships across questions). If data processing staff requires clarification from the field and the team is still in the field, they should clarify discrepancies with the field team.
Data processing team should also quality check the data captured qualitatively. The notes will be compared to the recording done during the activity to ensure that all important points are captured in the notes.
Data tables will be generated for quantitative data. Latent level of analysis will be done for qualitative data where relevant notes are tagged and categorized.
All answers or notes that are in the local language must be translated in English.
The Consultant is expected to first submit a draft report for comments and review of LWR. Clean raw data sets for quantitative data including frequencies and cross-tabulations must be submitted along with the draft in CSV format, and when available, Stata 11 format. FGD and KII notes along with a summary of key qualitative findings must be submitted in English and in Word format.
Task 5 – Final Report submission
The Consultant must submit the final report addressing all the comments made by LWR. A tracking table of comments made and responses must be included in the Annex.
Optional Tasks
The optional task can be called upon by LWR. If LWR decides to exercise this option, the Consultant will be informed before or during preparation for field work. The workplan and budget for the optional task should be given as separate items in the Consultant proposal. This task and its associated activities are:
1. Case studies
Combined with the quantitative and qualitative data, the Consultant will prepare two or three brief case summaries covering the following issues:
o development situation before the project,
o reflection on the various stages of the project,
o engagement with women leaders and village leaders (challenges and areas for improvement)
o project highlights and successes, including capacity-strengthening gains.
o Beneficiary in spotlight, highlighting one project beneficiary per village
Each case study should be no longer than 5 pages. A format for the case studies will be provided by LWR.
The Consultant should budget for 3 additional beneficiary interviews. The beneficiary details will be provided by LWR. Interview of the additional beneficiary should coincide with the data collection workplan for the base task.
2. Expand Evaluation to a dose-response approach
This methodology aims to apply a dose-response approach to systematically evaluate the impact of doses (levels) of interventions on the development outcomes of a project. By employing this approach, we seek to understand the optimal level of resources or interventions required to achieve the project developmental objectives as measured by changes in selected key indicators. If this option is called, the Consultant will also collect data in the six villages covered by Kopi Nande, a precursor project of MECIHO.
o Analysis
• Construct dose-response curves for each key indicator, illustrating the relationship between the dose of resources or interventions and the observed impact.
• Utilize statistical methods to analyze trends and patterns in the data.
o Identification of Doses
• Determine the optimal dose level at which the project achieves the maximum positive impact (if any) on the key indicators.
• Calculate any inflection points or thresholds indicating a significant change in the dose-response relationship.
4. Timeline
The field work is expected to start by first week of June and the first draft report should be submitted by second week of August.
5. Qualification and application procedures
The evaluation seeks for a person meeting the following requirements:
Qualification:
• Bachelor's Degree in in Social Sciences, Statistics, Economics, or other related field.
• If Bachelors Degree holder, minimum 10 years of working experience in evaluating project and in the field of agriculture value chain/WASH/ women empowerment. If Masters degree holder or better, minimum of five years working experience.
• Experience in using KoboCollect as evaluation Survey data collecting tools is desired
• The Consultants' independence from any organizations that have been involved in designing, executing or advising any aspect of the intervention that is the subject of the evaluation.
Skills:
• Writing and communication will be in English, and he/she must have excellent communication skills in English. He/she should also be able to communicate in Bahasa Indonesia while data collection in the project area;
Competencies:
• Demonstrates commitment to LWR's mission, vision and values;
• Highly knowledgeable of participatory monitoring and evaluation processes and experienced in evaluation of projects on WASH and women empowerment would be an added advantage;
• Ability to deliver quality report within given time frame;
The Consultant will submit their proposal with 3 reference from previous work.
The evaluator consultant will submit their proposal with 3 reference from previous work to JobsIndo@corusinternational.org before March 6th 2024.
Final Evaluation Consultant for MECIHO Project
June – August 2024
1. Background and Context
Building upon the success of Kopi Nande Project, Lutheran World Relief (LWR) was awarded a grant from the Starbucks Foundation to implement a 36-month MECIHO (Multisectoral Collaboration for Inclusive Health & Opportunities for Women) Project in six coffee-producing communities in Karo District, North Sumatra Province, Indonesia to address the challenges faced by the communities in hygiene, water, and sanitation as well as to empower women in leadership roles to enact positive changes in their communities. The consulting service, for which this Terms of Reference (ToR) is provided, is required to conduct the final evaluation for MECIHO.
The project is using innovative methodologies of awareness-raising and behavior change, enhance existing government-led programs on health and hygiene, and using a women-led approach for improved health and hygiene in the communities. The result of the study will help the organization in improving its strategy in women empowerment and WASH issue. The final evaluation will also include end line study, where the project will compare the evaluation result with the baseline.
The ToR outlines a brief description of the project, the objectives and scope of work, and required qualifications. The contract duration is for three (3) months, starting from the date of contract signing.
Working in six coffee-producing communities of the Karo District of North Sumatra, Indonesia, the MECIHO program aims to improve the water, sanitation, and hygiene (WASH) conditions of the community and individual household level through innovative, participatory means led by women. The MECIHO program's theory of change is based on the belief that women's leadership in health and hygiene matters. Thus, the project strengthens and infuses existing women's groups with new skills and knowledge on health, hygiene, and leadership, empowers them to disseminate the skills and knowledge to their respective communities, and encourages them to step up in the village decision-making process so that they can voice their aspirations. In doing so, they will take on a greater role in transforming their communities into healthier places to live. This women-led approach will create sustainable behaviour change by combining trainings and promotion with tangible improvements in the targeted communities. When communities begin to see the benefits of these improvements, they will be more likely to prioritize health and hygiene.
The goal of MECIHO Project is to improve the health, hygiene, and living conditions of households and communities in six target communities in the Karo District of North Sumatra. The six communities are: Sarimanis, Tanjung Barus (Barusjahe Sub-district), Jaranguda, Cinta Rakyat (Merdeka Sub-district), Nang Belawan, and Ndokum Siroga (Simpang Empat Sub-district).
The project has two inter-related outcomes, which are theorized to contribute to the goal. The two outcomes are:
1. Improve the mechanisms and systems that support the advancement of healthier homes and better living conditions.
MECIHO promotes "small doable actions" to reinforce behaviors that support healthy homes and communities. WASH campaign messages will be refined based on findings from WASH mapping. The project trains and supports WASH promoters in both schools and communities and produce a radio campaign with messaging around hand washing and latrine use, promotion of opportunities to install private latrines, and opportunities for participation such as contests and call-in shows. In addition, the project also initiates and strengthens garbage banks to promote waste management at the village level.
2. Strengthen women's leadership on and engagement in governance initiatives.
MECIHO provides leadership training and training of trainers (ToT) in hygiene promotion and waste management along with opportunities to engage in local governance processes for village-level Women's Champions. The Champions and other community members conduct WASH mapping to assess current hygiene practices and behaviors, functionality of existing facilities, WASH barriers, and gender and social norms
2. Evaluation Scope
The objectives of the evaluation are to assess:
• The achievement of projects' outcomes based on their respective set of indicators/ targets to date as well as the projects key evaluation questions.
• The project performance based on OECD-DAC evaluation criteria.
• Analyze the project approach, strengths and weaknesses; issue and challenges of major components of this project and provide recommendation for future
The assessment will be based on OECD-DAC criteria given below.
• Relevance or appropriateness of the project: IS THE INTERVENTION DOING THE RIGHT THINGS?
To assess the extent in which the project interventions goals and implementation are aligned with beneficiary and stakeholders needs by investigating if they view the interventions as useful and valuable. Where applicable, the assessment of relevance should be analyzed based on its four elements of analysis:
• relevance to beneficiary and stakeholder needs,
• relevance to context,
• relevance of quality and design, and
• relevance over time.
• Coherence: HOW WELL DOES THE INTERVENTION FIT?
To assess the compatibility of the project intervention with other interventions in the target geography, be it interventions delivered by government institutions or other development actors. Coherence also looks at the consistency of the project intervention with relevant international norms and standards especially in the context of water, sanitation, and hygiene (WASH) ang gender empowerment. It should be analyzed based on two dimensions:
• Internal coherence – how well are the project interventions align with the wider policy frameworks of the local or national institutions responsible for implementing the development interventions in the Karo region. The assessment should look at how the project interventions harmonized with local development activities or if there are any duplications, how they complement. Internal coherence also provides a lens for assessing inclusion.
• External coherence – how well are the project interventions align with external policy commitments such as the SDGs and how these are taken into consideration in the interventions design and implementation. External coherence is also concerned with how well the interventions connect with interventions implemented by other development actors in the Karo region, if there are any.
• Effectiveness: IS THE INTERVENTION ACHIEVING ITS OBJECTIVES?
To measure the extent to which the project interventions achieved or likely to achieve its desired outcomes and results. The assessment should provide insights into:
• The extent the project has attained its planned results by comparing its actual achievement with its targets
• An evaluation of the process by which this was done
• Which project components were decisive or important to the process
• Whether their were any unintended effects
Effectiveness is different from impact as it is more concerned with the most closely attributable effects, while impact examines higher-level effects and broader changes. Assessment of effectiveness should also look at how well it allowed for participants participation in the implementation of the project interventions and developing their ownership and accountability especially during the process evaluation. It should also look at how the project evolve and adapt and how the design changes, if at all, based on feedback from stakeholders, emerging results, and changes in context.
• Efficiency: HOW WELL ARE RESOURCES BEING USED?
To measure the extent to which the intervention delivers, or is likely to deliver, results in an economic and timely way. The assessment should look at whether the project use of resources justifies the results it achieved. The focus of this assessment should be in
• Operational efficiency – How well are resources used during implementation. It reviews if human and financial resources were used as planned or redirected as needs change. It should also looked at how well the project managed its risks.
• Timeliness – The assessment should look at whether and to what extent the results were achieved within the intended timeframe. It should review and check if the timeframe and work plans were realistic and appropriate, and if the changes to the timeline are necessary given the many external factors that could affect the implementation of the project.
• Impact: WHAT DIFFERENCE DOES THE INTERVENTION MAKE?
What were the higher level changes or effects both positive and negative or intended and unintended on the community, beneficiaries, gender and village groups/SHGs; What are the community/beneficiaries' perspective on the immediate and intermediate effects (quotes can be collected to provide evidence on participant's perspective). To what extent is the project contributing towards achieving transformational change especially within the context of WASH?
• Sustainability: WILL THE BENEFITS LAST?
Measure the extent to which the project results and impacts or positive outcome at the community level are sustainable in the longer terms (quotes can be collected that can provide evidence on participant's perspective).
The total population covered by this study is 12,216 with 6,013 male and 6,213 females.
3. Objective of the Engagement and Scope of Work (SoW)
The evaluations of the MECIHO Project will rely on data collected as described in this TOR. Each data collection activity will form the basis for measuring outcomes for program beneficiaries as well as non-beneficiaries, which is critical for assessing program impact. The Consultant should work in close coordination with Corus team to carry out the study. The Consultant is expected to assist in the evaluation by collecting primary and secondary data needed by the evaluation. The SoW of the Consultant includes a base tasks, which are a set of activities that the Consultant is expected to deliver and optional task, which is a set of activities that could be exercised by Corus.
Evaluation Design
The evaluation of MECIHO Project will use Propensity-Score Matching (PSM), a quasi-experimental method used to calculate the impact of the intervention by estimating the difference in outcome between beneficiaries and non-beneficiaries of the program. Since the selection of treatment villages were non-randomized, PSM compare treatment effects across program beneficiaries and matched non-beneficiaries. Matching of is done by observing and analyzing a range of observed characteristics. A propensity score is computed for each observation. Once all of the scores are computed, the observations in the treatment group can be matched with observations in the control group that have the closes propensity score. Through this method, a counterfactual is constructed.
The pool of observations where the control group will be drawn will come from an additional six villages. These six villages will potentially be included in the next phase of the MECIHO project.
Base tasks
These tasks include primary and secondary data collection that include:
Secondary Data
Desk Review
The Consultant will complete a desk review of key program documentation including project design documents, monitoring and progress reports, and other relevant publications. The purpose of the desk review is twofold:
i. to better understand program operations on the ground.
ii. to become familiar with existing analytical frameworks used to evaluate such types of programs; and
iii. to extract information needed to report against key evaluation questions.
Photo Voice and Real Reel Analysis
The Consultant will complete a review of the output of the Photovoice and Real Reel activity. The purpose of both activities is to add additional qualitative insights into the study. The Consultant is expected to include the analysis in the final report.
Primary Data
The Consultant is expected to lead a team of data collectors that will collect primary data needed by the evaluation. Primary data will be collected through quantitative and qualitative method as described below.
Quantitative Study
Survey will be administered in both the treatment and control villages. During the baseline, the survey was able to complete xx number of interviews. For the end line, a sample size of xxx is proposed for each group thereby resulting in xxx total interviews. The evaluation firm can propose a different sample size as long as satisfy the requirements of 95% confidence level and 5% margin of error. The preference is to use probability proportional to size sampling (PPS) in identifying samples but alternative method can be suggested as well.
The draft questionnaire in English will be provided to the Consultant after Contract signing. The Consultant is expected to translate, pre-test, and finalize the questionnaire. Enumeration using KoboCollect app or any survey app running on Android devices is the preferred method over a pen-and-paper survey.
Qualitative Study
The Consultant is expected to conduct qualitative study as well. The qualitative study is intended to provide better insights into the changes that happened in the community as well as to inform policymakers and implementers about how the project were implemented and maintained. As such, the qualitative study is envisaged to go deep into the discussions rather than go wide and cover more villages. The Consultant is expected to implement two approaches: FGD and KII. The proposed number for each method per project is given below:
FGD = 2
1. Women leaders in the treatment group
2. Women leaders in the control group
KII = 6
1. Representative of a successful garbage bank group
2. One female leader in the treatment group (not included in the FGD)
3. One female leader in the control group (not included in the FGD)
4. One male leader in the treatment group
5. One male leader in the control group
6. One district level government official
Qualitative data shall be captured using key notes using the template that will be provided.
Description of Tasks
Task 1 – Write Inception Report and Work Plan
The Consultant shall submit an inception report and work plan that includes the following:
• Understanding of the engagement and tasks required
• Substantive input into design of draft data collection instruments, methodology and overall study approach
• Team composition and recruitment plan
• Numbers, dates, duration, and location of training sessions; including composition of training teams, agenda, preparation of logistics, and reporting arrangement to the LWR Evaluation Coordinator
• A list of permits and letters required to conduct fieldwork and when those will be obtained
• Expected duration of survey implementation and work plan for Field Teams including supervision plan
• Expected date and duration of the Data Cleaning activities, including delivery dates of the different datasets
Task 2 – Prepare for field work
Preparation for field work entails a number of subtasks that must be completed in order to ensure the quality of data that will be collected. It includes:
• Recruitment and selection of field teams and data processing teams.
Interviewers and facilitators will be responsible for collecting the data in the field. Interviewers that will be selected must have demonstrated experiences in administering surveys or facilitating discussions. Each field team must be headed by a field supervisor which will oversee the over-all management of field work of the team including backchecking and other data quality processes on field. Data processing team are responsible for ensuring that data collected in the field are free from errors prior to analysis. They conduct the data transformations as required by the Consultant.
• Prepare research permits
In the event that the community or municipality will require permits for field work, it is the responsibility of the Consultants to obtain these permits.
• Finalization of data collection instruments
All data collection instruments must be tested before being used. Testing procedures could be pilot testing or cognitive pretesting. The objectives of the testing are:
o allow the Consultant to familiarize itself with the instruments,
o test the logistics (such as field mobilization and demobilization, potential locations of basecamps) and time necessary for the surveys
o test the draft questionnaires. Testing the draft questionnaires will include identifying any exercises/questions that are not working as currently framed.
• Familiarize with the use of KoboCollect or alternative
LWR prefers data collection using KoboCollect. Consultant must familiarize themselves with the use of the software before field work.
• Training of Interviewers and Facilitators
LWR strongly prefers that all field team members be trained in one central location. If Consultant proposes training in multiple locations, the proposal should detail how the service provider will ensure consistency across training locations and common understanding of questionnaires and data collection protocols.
Note that the Consultant is required to begin fieldwork within one week of the end of training. If there is a delay of more than one week, the consultant will be required to do a one-day refresher training to all interviewers at its own expense.
A short training report should be submitted by the Consultant within a week after the training.
Task 3 – Collect Data
Data collection effort should always follow government advisories in preventing the spread of COVID-19. In the event that face-to-face data collection cannot be conducted due to prevailing government restrictions, the Consultant should attempt to collect data using indirect method. Indirect method could include interviews over mobile or social messaging apps.
• Data Collection (Quantitative)
LWR will provide the Consultant with a list of households to interview and a replacement strategy that should be strictly followed. The households will be the same households that were visited during the baseline. Upon arriving in each village, the survey team should go to the village hall and attempt to verify whether the listed households still reside in the village. Field teams should preferably interview the same respondent who was interviewed during the baseline (for budgeting purposes, the Consultant should assume that 20% of respondents will be revisited).
The field teams are expected to adhere to the respondent tracking process:
1. If the respondent is not home during the time of the interview: The interviewer should return at a different time and if the respondent still cannot be reached during this second attempt, the interviewer must make a third attempt at a different time/day to find and interview the respondent.
2. If the respondent moved village but within the same district: The interviewer must trace the respondent within the district and conduct the interview.
3. If the respondent moved outside the district: Replace the respondent with another one according to the replacement strategy provided by LWR.
Ideally, no respondents from the previous survey round should be replaced. However, replacements should be addressed on a case-by-case basis. The Consultant should inform LWR about any such instances before reaching a decision.
• Data Collection (Qualitative)
Appropriate venues to conduct FGDs must be identified and booked prior to the activity. Invitations for participants to participate must be sent out a couple of days before the discussion. In the event that, less than 4 participants showed up for the FGD, the activity will still continue, with the facilitator adopting KII approaches. For KIIs, appointment must be booked in advance and reconfirmed a couple of days before the interview. Data recording, if allowed, and extensive note-taking should be done for each activity.
• Data Quality
The Consultant is responsible for collecting the highest quality data possible and subjecting interviewers, facilitators, and data to great scrutiny to ensure quality. It is recommended Consultant implement three kinds of data quality checks.
o Observation: Supervisors accompany the interviewers while they are doing the interview to observe and support interviewers. With this type of checking, supervisors will find out if the interviewer understands interview questions, effectively conveys them, and can coach the interviewer on how to improve if problems arise. Supervisors must accompany interviewers for at least 10% of interviews. The Consultant is expected to submit a report tracking the observation made.
o Supervisors are required to reinterview the respondents over mobile phone and ask some of the (sample) questions in the questionnaires to the same respondents. The purpose of this procedure is to verify that interviewer actually met the respondents and conducted the interview. If there are significantly different answers between the two interviews, supervisors should clarify the difference. If the difference is because the interviewer wrongly posed the question or misunderstood the question, supervisors should brief the team to clarify the misunderstanding. If any interviewer is found to be falsifying or manipulating data, he/she must be fired and replaced. Replacement interviewers must re-interview all respondents interviewed by the interviewers who falsified data. The Service Provider must ensure that at least 5% of interviews are audited (i.e., households revisited). The Consultant is expected to submit a report Detailing the result of the re-interviews.
o Checking the completed interview: supervisors will check 20% of completed interviews to identify errors made by the interviewers. The checks can be done through the KoboCollect online facility and should be done while the team is on the field interviewing.
• Methodological Report on field work implementation
The Consultant shall write a methodological report of no more than 5 pages (excluding annexes) detailing implementation of the fieldwork, any issues and problems that arose and how they were managed.
.
Task 4 – Data cleaning, analysis, and draft report
Data processing team is responsible for starting the data cleaning process as soon as they receive data and other supporting documents from the field. This process includes checking the accuracy of data received, such as respondents' identification and linkages among questions (meaning the skip patterns and relationships across questions). If data processing staff requires clarification from the field and the team is still in the field, they should clarify discrepancies with the field team.
Data processing team should also quality check the data captured qualitatively. The notes will be compared to the recording done during the activity to ensure that all important points are captured in the notes.
Data tables will be generated for quantitative data. Latent level of analysis will be done for qualitative data where relevant notes are tagged and categorized.
All answers or notes that are in the local language must be translated in English.
The Consultant is expected to first submit a draft report for comments and review of LWR. Clean raw data sets for quantitative data including frequencies and cross-tabulations must be submitted along with the draft in CSV format, and when available, Stata 11 format. FGD and KII notes along with a summary of key qualitative findings must be submitted in English and in Word format.
Task 5 – Final Report submission
The Consultant must submit the final report addressing all the comments made by LWR. A tracking table of comments made and responses must be included in the Annex.
Optional Tasks
The optional task can be called upon by LWR. If LWR decides to exercise this option, the Consultant will be informed before or during preparation for field work. The workplan and budget for the optional task should be given as separate items in the Consultant proposal. This task and its associated activities are:
1. Case studies
Combined with the quantitative and qualitative data, the Consultant will prepare two or three brief case summaries covering the following issues:
o development situation before the project,
o reflection on the various stages of the project,
o engagement with women leaders and village leaders (challenges and areas for improvement)
o project highlights and successes, including capacity-strengthening gains.
o Beneficiary in spotlight, highlighting one project beneficiary per village
Each case study should be no longer than 5 pages. A format for the case studies will be provided by LWR.
The Consultant should budget for 3 additional beneficiary interviews. The beneficiary details will be provided by LWR. Interview of the additional beneficiary should coincide with the data collection workplan for the base task.
2. Expand Evaluation to a dose-response approach
This methodology aims to apply a dose-response approach to systematically evaluate the impact of doses (levels) of interventions on the development outcomes of a project. By employing this approach, we seek to understand the optimal level of resources or interventions required to achieve the project developmental objectives as measured by changes in selected key indicators. If this option is called, the Consultant will also collect data in the six villages covered by Kopi Nande, a precursor project of MECIHO.
o Analysis
• Construct dose-response curves for each key indicator, illustrating the relationship between the dose of resources or interventions and the observed impact.
• Utilize statistical methods to analyze trends and patterns in the data.
o Identification of Doses
• Determine the optimal dose level at which the project achieves the maximum positive impact (if any) on the key indicators.
• Calculate any inflection points or thresholds indicating a significant change in the dose-response relationship.
4. Timeline
The field work is expected to start by first week of June and the first draft report should be submitted by second week of August.
5. Qualification and application procedures
The evaluation seeks for a person meeting the following requirements:
Qualification:
• Bachelor's Degree in in Social Sciences, Statistics, Economics, or other related field.
• If Bachelors Degree holder, minimum 10 years of working experience in evaluating project and in the field of agriculture value chain/WASH/ women empowerment. If Masters degree holder or better, minimum of five years working experience.
• Experience in using KoboCollect as evaluation Survey data collecting tools is desired
• The Consultants' independence from any organizations that have been involved in designing, executing or advising any aspect of the intervention that is the subject of the evaluation.
Skills:
• Writing and communication will be in English, and he/she must have excellent communication skills in English. He/she should also be able to communicate in Bahasa Indonesia while data collection in the project area;
Competencies:
• Demonstrates commitment to LWR's mission, vision and values;
• Highly knowledgeable of participatory monitoring and evaluation processes and experienced in evaluation of projects on WASH and women empowerment would be an added advantage;
• Ability to deliver quality report within given time frame;
The Consultant will submit their proposal with 3 reference from previous work.
The evaluator consultant will submit their proposal with 3 reference from previous work to JobsIndo@corusinternational.org before March 6th 2024.
0 Comments:
Posting Komentar