Terms of Reference
Final Project Evaluation
Strengthening Public services through the Empowerment of women-led Advocacy and social audit networKs (SPEAK)
(Project Period: 2018-2021)
Background
Endemic poverty in Bojonegoro District and Semarang City suggests that structural factors or "poverty traps" are deeply entrenched in society. Structural poverty theories refute the understanding of poverty as "case poverty" referring to individuals who are poor because of "some quality peculiar to them"[1], and shift the focus on the phenomenon of social exclusion, or conditions that tend to generate and perpetuate chronic poverty. Political freedoms and transparency guarantees are now widely acknowledged as integral components of effective poverty reduction strategies, and governance reforms have become inseparable of poverty reduction strategies.
Most governance reform strategies are based on the assumption that the development of free and fair electoral systems combined with the strengthening of watchdog and supervisory institutions, together with regulatory and institutional changes promoting better availability of information on administrative functioning, will result in greater political freedoms and transparency guarantees. Institutional and legal reforms are expected to bring about greater transparency, public participation, and therefore greater political responsiveness and accountability, leading to social change. However, institutional and legislative reforms can only bring about political opportunities, which will not translate into social change without addressing the capability of citizens to cease such opportunities. Regulations on freedom of information without public demand for information will not translate into greater transparency of public institutions. Availability of information does not guarantee greater public participation in policy making or monitoring, and electoral freedom does not guarantee better political representation and accountability either.
Strengthening Public services through the Empowerment of women-led Advocacy and social audit networKs (SPEAK) is a co-funded program between the European Union and Hivos that aims to institutionalize more constructive and effective community participation in planning and implementing regional budgets to overcome poverty. Perkumpulan IDEA as the implementing partner, and its associates are investing in the capacity of local women groups to advocate for gender-responsive and inclusive budgets, and to facilitate social audit as to improve accountability of public procurement and public services through community-based monitoring. The program will aim to strengthen 2 local CSO and 5 women-led groups in order to develop a network of over 240 women, empowered to represent 550,000 economically marginalized men and women in Semarang City and Bojonegoro District.
The overall objective of the action is to institutionalize constructive and effective citizen participation in the planning and implementation of local budgets to better address structural poverty.
There are two specific objectives designed to mutually contribute to the program objectives, as follows:
· Specific objective 1: Empower CSOs and local women groups to advocate for gender-responsive and inclusive budgets. Despite recent improvements in local budget management, focus on poverty reduction and gender-responsiveness into development planning and budgeting remains insufficient. Therefore, advocacy is needed to improve budget allocation towards priority programs by engaging citizens to actively participate in sectoral planning and budget planning. The proposed action will strengthen the representation of economically marginalised groups in the planning and budgeting process, by developing sectoral networks and empowering women groups to facilitate access to information and participation in public forums and planning processes.
· Specific objective 2: Improve accountability of public procurement and public services through community-based monitoring. Public procurement in Indonesia is prone to corruption, and actually reflects collusion and intervention of vested interests from the planning and budgeting stages. Budgets and procurement plans therefore tend to be biased towards interests and preference of business elites rather than actual needs of economically vulnerable populations, who depend most on public spending. The proposed action will attempt to address such gap by developing in parallel the institutional and social foundations of public participation. The action will build the capacity and regulatory framework required to facilitate public access to information and constructive communication with local administrations, while at the same time developing the capacity of local women groups to organize local communities and facilitate community participation in complaint management and project monitoring. Finally, the action will contribute to raise awareness on the impact of corruption on public health and education, and on the importance of public participation in corruption prevention and monitoring through developing a network of 12 local journalists to report on corruption in public procurement, and investigate on citizen journalists reports.
Purpose & Objectives of Evaluation
The main purpose of this final independent evaluation is to assess the extent to which the project objectives have been achieved and at assessing the impact of the project particularly in strengthening the quality of political representation and integrating public participation in the making of social, economic, and political change.
The focus will also be on assessing the expected impact of the interventions (either positive or negative) and the sustainability of the project's beneficiaries and the local partners' strategy and capacity to sustain them. It will also look at strengths and weaknesses, opportunities and challenges and any external factors that have affected the achievement (or non-achievement) of the outcomes and the delivery of the outputs.
The final evaluation also promotes accountability for achievement of the project objectives through the assessment of results, effectiveness, processes and performance of the partners involved in project activities according to SPEAK indicators. It will also identify lessons learnt and good practices for both accountability and learning for possible similar interventions in the future.
Consequently, the evaluation is also expected to make detailed recommendations for major stakeholder groups to promote sustainability and long-term impact to the beneficiary communities.
To achieve the above mentioned purposes, this final evaluation will focus and address the following key questions:
● SPEAK project's approach including project management, monitoring and evaluation system, coordination mechanism among various stakeholders (including government institutions, private sectors, CSOs, local partners etc.) and how effective this has been;
● The extent to which SPEAK's strategy can address issues related to:
o Good governance through integrating the social and political aspects of accountability and strengthening the capacity of public officials, reforming regulatory frameworks, developing new accountability procedures or strengthening new supervisory institutions organizations;
o The limited capacities & capabilities of CSOs to hold their leaders accountable and reach instrumental freedoms;
o The limited representation of economically marginalised groups in the planning and budgeting process
o Availability of solid social and institutional infrastructure to allow for effective public participation in complaint handling, follow up and monitoring.
o The limited capacity of local journalists to report on corruption in public procurement, and investigate on citizen journalists reports.
● The extent to which the SPEAK project has achieved its specific objectives and outputs:
● Project's experiences that can be learned with regards to promoting social and gender inclusiveness in facilitating public participation, building trust and improve communication between stakeholders (multi-stakeholders' approach) and develop a conducive regulatory framework of transparent procurement, community right to access information and participate in the budget design, and systematic public monitoring of public procurement;
● An initial assessment of the project's indirect impacts
● Provide concrete recommendations for future design and implementation of the project based on evaluation findings and conclusions
● Present recommendations for sustainability, lessons learnt and case studies of good practices
The project evaluation must provide evidence-based information that is credible, reliable, and useful and should align strongly with the SPEAK key indicators. The evaluation team is expected to follow a participatory approach. Close coordination with the SPEAK project team is also key for this assignment.
Scope of Project Evaluation
The project evaluation is meant to assess the project performance and achievements vis-à-vis the project's objectives, indicators and results, as well as the impact of the project to the beneficiaries. The evaluation should be based on the six evaluation criteria from the Development Assistance Committee of the Economic Cooperation and Development (OECD/DAC) – Relevance, Coherence, Effectiveness, Efficiency, Impact, and Sustainability.
Relevance
The final evaluation will review and assess the appropriateness of the SPEAK' concept and design, the relevance of the project's outputs, and the extent to which they have contributed toward the overall development objective. Relevance means the degree to which the objectives of the project remain valid and pertinent either as originally planned or as subsequently modified owing to changing circumstances within the immediate context and external environment of that project.
Some of the guiding questions might include, but not limited to:
● To what extent did the Project achieve its overall objectives?
● What and how much progress has been made towards achieving the overall outputs and outcomes of the Project (incl. contributing factors and constraints)
● To what extent were the results (impacts, outcomes, outputs) achieved?
● Were the inputs and strategies identified, and were they realistic, appropriate, and adequate to achieve the results?
● Was the Project relevant to the identified needs?
Coherence
The final evaluation will better capture linkages, systems thinking, partnership dynamics, and complexity in the project implementation.
Some guiding questions include:
● To what extent did the program capture the multi-stakeholders' coordination specifically with local CSOs, (women) infomediaries[2], women-led community groups, local administrations, local journalists, and citizen journalists, to institutionalize constructive and effective citizen participation in the planning and implementation of local budgets to better address structural poverty.
● To what extent is this program relevant to contributing to the government's commitment to international norms and standards?
Performance
There are three criteria for performance that the final evaluation will need to address:
● Effectiveness - the extent to which the project achieved its immediate objectives or produced its desired outcomes;
Some of the guiding questions might include, but not limited to:
● Was the Project effective in delivering desired/planned results?
● To what extent did the Project's ME mechanisms contribute in meeting project results?
● How effective were the strategies and tools used in the implementation of the Project?
● How effective has the Project been in responding to the needs of the beneficiaries, and what results were achieved?
● What are the future strategies and issues?
● Efficiency - the optimal transformation of inputs into outputs;
Some of the guiding questions might include, but not limited to:
● Was the process of achieving results efficient?
● Are there more efficient ways and means of delivering more and better results (outputs and outcomes) with the available inputs?
● Could a different approach produce better results?
● How was the project collaboration with the local administrations, journalists, and other relevant stakeholders?
● What are the strengths, weaknesses, opportunities, and threats of the project implementation process?
● Timeliness of inputs and results.
The evaluation shall:
● Assess whether the SPEAK project has so far produced its outputs effectively and efficiently;
● Identify the major factors that have facilitated or impeded the progress of the project in achieving its desired results.
● Assess whether Government inputs, at national and local levels, were sufficient and how they should be improved, if necessary.
Success
The three criteria of success that the final evaluation will need to address are impact, sustainability and SPEAK's contribution to change happened at the social actors.
● Impact
Impact refers to the results of the project that are assessed with reference to the development objectives or long-term goals of the SPEAK project. In this sense, impact represents changes in a situation, whether planned or unplanned, positive or negative; that the project brings about.
Some leading questions to assess the impact are:
o What has happened as a result of the project?
o What real difference that the project has made to the ultimate beneficiaries?
o What changes have been observed in relation to the objectives of the intervention?
o Are there unintended impacts including consideration of different segments of society?
o What interventions and approaches delivered the impact? What are key contextual features for these e.g. gender, poverty, ethnicity, etc.?
● Sustainability
Sustainability is the durability of positive project results after the termination of the technical cooperation channeled through that project.
Some leading questions to assess the sustainability are:
o To what extent will the project's benefits/impact continue after they have ended?
o What are the major factors that will have or will influence the continuity of the project's benefit?
o How effective was the exit strategies and approaches to phase out assistance provided by the project, including contributing factors and constraints?
o How were capacities strengthened at individual and organizational level, including contributing factors and constraints?
o Describe the main lessons that have emerged
Relevance, coherence, effectiveness, efficiency, impact and/or sustainability are the evaluation criteria against which the intervention will be assessed.
The evaluator may adapt the evaluation criteria and questions but the changes should be agreed and reflected in the inception report.
Evaluator may suggest and make necessary changes on the questions upon approval by the evaluation manager.
The conclusion should be comprehensive and balanced, and highlight the strengths, weaknesses and outcomes of the project. They should be well substantiated by the evidence and logically connected to the final evaluation findings. They should also respond to the key evaluation questions and provide insights into the identification of and/or solutions to important problems or issues pertinent to project beneficiaries and donors.
Evaluation Approach and Method
The evaluation will require the collection of both qualitative and quantitative data which include but are not limited to:
- Review of documents related to the project, including the initial project document, progress reports, technical assessments and reports, project monitoring and evaluation documents.
- Review of technical products (training manuals, tools, guidelines, etc.) and other publications used or developed by the project.
- Review of other relevant documents, if any
- Internal meetings and interviews with Project Manager, technical specialists, and other project staffs
- Consultations and/or interviews with relevant stakeholders involved, including government representatives, local communities, CSOs, and other relevant stakeholders
- Interview project beneficiaries (FGDs and individual interviews with members of community groups, associations, etc.)
- Observation
For the Final Evaluation, the consultant(s) shall determine the appropriate sample size for data collection efforts in consultation with SPEAK Project team with maximum margin of error 5%. These figures will take into considerations the activities carried out by the project. The sampling criteria and sampling methodology will be clearly described in the final report of the consultant.
Ideally, consultant(s) will visit these project sites and interview the beneficiaries and partner organizations on the relevant project objectives. Given the COVID-19 situation, all tasks are expected to be done and delivered online or with limitation of conducting workshops/FGDs in compliance with government regulation. At the same time, the timeline should allow for flexibility (and patience), keeping in-check the availability, situation, and sensitivities of the participants. Implementing strict health protocol and provide health & safety measurement is mandatory if direct/face-to-face workshops/FGDs/interviews is selected by the consultant.
It is important in gathering data that there are opportunities for minority groups to provide feedback in a safe space.
Deliverables
The Consultant(s) will produce the following deliverables:
| Deliverable | Content | Timing | Payment Schedule |
| Contract signing | 25% | ||
| Meeting agreement | Kick-off Call The SPEAK project team will schedule a contract kick-off meeting with consultant to discuss expectations for the assignment and clarify remaining technical and/or logistical questions | Half-day meeting, within 2 days after contract signing |
|
| Revised ToR | Revision/Improvement of ToR, as appropriate The Consultant(s)' first task will be to revise and strengthen the Term of Reference outlined above based on its experience and good practice. The updated ToR will be submitted to the SPEAK project and if accepted, included in the contract to be assigned by the Consultant(s) | Within 3 days of contract signing |
|
| Inception Report (including evaluation matrix)* | Inception Report Consultant(s) will provide an Inception Report (in English and Bahasa Indonesia) consisting of: - Description of the conceptual framework that will be used to undertake the evaluation; - Elaboration of the methodology proposed in the ToR with changes required; - Details of the data required to answer the evaluation questions, data sources by specific evaluation questions, data collection methods, sampling and selection criteria of respondents for interviews; - Detailed work plan for the evaluation indicating the phases in the evaluation, their key deliverables and milestones; - List of key stakeholders to be interviewed and the tools to be used for interviews and discussions; - Outline for the final evaluation report | Within 3 days of contract signing |
|
| - | Midpoint Check-in Consultant(s) will conduct a midpoint check-in with the SPEAK project team to update progress and communicate any major challenges or concerns regarding the data collection. | Midpoint data collection |
|
| Powerpoint Presentation and Validation workshop | Presentation of initial findings & Validations Workshop During the final days of data collection, the Consultant(s) will be expected to prepare a summary debrief presentation of the initial findings. The Consultant(s) will also be expected to design and facilitate a daylong validation workshop with the SPEAK project team, partners and key stakeholders to gather feedback on initial findings and discuss potential recommendations. | Within 3 weeks of the Inception Report |
|
| Draft Final Report + Executive Summary | Draft Final Report + Executive Summary (in English) Full report covering all items detailed on Scope of Project Evaluation with detailed attention to lessons learned and recommendations and with annexes minimally including list of persons interviewed, summary of field visits, lists of documents reviewed, questionnaire and summary of results, co-financing matrix and leveraged resources, etc. | Within 1 week of validation workshop | 25% Upon satisfactory approval |
| Draft Case Studies | Draft Case Studies (in Bahasa Indonesia & English) Minimum of 1 strong case studies which can be published as evidence of project outcomes/impact | ||
| Final Report & Evaluation Matrix | Revised report with detailed information on how all received comments have (and have not) been addressed in the final review report. Final report should be submitted in Bahasa Indonesia and English.
| Within 1 week of receiving comments on draft | 50% Upon satisfactory approval |
Evaluation report.
The production of the evaluation report will be the liability of the evaluator(s) covering all the evaluation questions, objectives and areas outlined in this ToR.
Together with SPEAK's Project Manager, Project Officer and DMEL Coordinator, the consultant will be responsible for coordinating the evaluation exercise, including coordinating draft of final report to a minimum of 2 external readers.
During the research process, the evaluator will keep the above-mentioned people up to date and agree on changes to the methodology as and where appropriate.
The evaluation report shall be as followed:
● Be produced in English and Bahasa Indonesia and should be simple in expression (jargon free).
● The Executive summary should include brief description of project, brief section on methodology, most text for conclusions and summary of recommendation.
● The text should be an A4 paper size in Museo Sans Rounded 300 or Calibri font size 10.
● The evaluator(s) will be liable to submit at least two hard copy and an electronic version of the evaluation report in Word and PDF Version by the agreed deadline.
● The final product should be fully referenced, with findings clearly linked to evidence.
Format for Evaluation Report
The evaluation reports should be written in clear and to the point language and not be longer than 65 pages, excluding the appendices. The format should be as follows as a minimum:
• Executive summary
This is a summary of the evaluation with special emphasis on the main findings, conclusions, leanings and recommendations. (1-3 pages maximum).
It is VERY important that the executive summary of the evaluation is clear, synthetic and collects all of the relevant information as it is the part that the majority of readers will pay closest attention to.
• Introduction
Contextualization of the evaluation exercise, objectives of the evaluation, main criteria analysed with corresponding findings, presentation of the evaluation team and the process of the evaluation and its limitations. (3 pages maximum).
• The intervention and the context
Description of the evaluated intervention, its objectives, its logic, history, organisation and stakeholders. This part of the report should analyse the evolution of the political, economic and social context of the intervention zone throughout the evaluation period, paying special attention to aspects of development and humanitarian action. (max of 5 pages)
• Methodology
This includes a description of the methodological approach of the evaluation, details of the sample, the methods and tools used for collecting the data as well as an assessment of the evaluation process. (max of 3 pages)
• Analysis of the information gathered
This part presents the analyses that have been carried out from consulted information, secondary sources and the information that has been gathered. Based on the evaluation assessment proposed, the information gathered (primary) and consulted (secondary) should be analysed depending on the evaluation questions collected in the terms of reference (TOR). (max of 5 pages)
• Results of the evaluation
This includes a description of the relevant evidence for responding to specific questions about the evaluation. The evaluation team should answer the key questions developed in the TOR; we understand that these responses are a result of the evaluation. They should be based on evidence, that is more than just empirical data, and the possible inferences that could be carried out. Based on this evidence the arguments, opinions and interpretations that the evaluation team consider appropriate for answering the key questions will be developed. It is important that this section is well-linked with the data analysis section. (max of 35 pages)
• Conclusions
Assessment of the intervention and its results based on the evaluation criteria, the organisation's performance standards and their policies. (3 pages)
• Lessons learned
This part presents good practices or aspects to avoid stemming from the conclusions that could be used in other interventions or in a broader context. (3 pages)
• Recommendations.
This part contains ordered, practical, applicable and prioritised proposals that help improve the Management of the project or even future interventions. They can also be directed towards developing different kinds of policies and strategies. (4 pages)
• Appendixes/Annexes
These present the relevant information that has been consulted or generated for the evaluation, as well as complementary data about the process.
As a minimum requirement, the appendixes should present the following information:
- Terms of Reference (ToR)
- The work proposal, including the methodology and the tools used.
- A list of people who participated in the different evaluation activities including respondents
- The data gathered: from physical support to databases, in cases where they have been developed.
- Evaluation tools developed
- Consulted bibliography.
- Technical appendix for methodology assessment.
Consultant Qualifications
The following are the selection criteria for the team of consultant(s):
● Solid experience and proven track record of carrying out project evaluations
● Demonstrable relevant academic and practical experience in qualitative and quantitative research methodology, evaluation design and implementation
● Proven experience in good governance, gender and human rights issues, with good program design skills, including capacity to prepare logical, coherent and consistent documents;
● Capacity to work in a multi-disciplinary team, including coordinating technical input into a framework;
● Proven experience of using participatory tools as a means of data collection for project evaluation
● Excellent reporting and communication skills
Timeframe and budget
The fieldwork should be completed in no more than 2 weeks (exclude assignment days for the pre and post field work). The draft of final report should be presented no later than 5 weeks after commencement of the contract. Maximum budget for the entire assignment is IDR 204,233,367. The budget offered will be subject to negotiation after selection of the team of consultant.
Guiding principles
The SPEAK Project expects (team of) consultant(s) for this activity to adhere to the American Evaluation Association Guiding Principles for Evaluators:
● Systematic Inquiry: Evaluators conduct systematic, data-based inquiries
● Competence: Evaluators provide competent performance to stakeholders
● Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process
● Respect for People: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders
● Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation
Please consult the website of the American Evaluation Association (http://www.eval.org/p/cm/ld/fid=51) to find the Guiding Principles for Evaluators in their entirety. Note that any intellectual property or materials produced or any files or information obtained by the consultant while carrying out this contract are the sole property of the SPEAK Project and will not be disclosed to others. This includes the preliminary and final evaluation report. The consultant must work in close consultation with the SPEAK Project team and in accordance with the Terms of Reference.
Expression of interest
The overall proposal will be judged based on scoring criteria below:
| Technical Criteria | Technical Sub-Criteria | Maximum Points |
| Technical Approach (5 pgs) | - Description of the approach and method proposed for the requirements listed above | 40 |
| Personnel (2 pgs) | - Description of qualifications and unique skillset of each team member, as well as balance and complementary of skills within the team - Description of home-office and support personnel, as relevant - Provide a maximum of three-page CV of each candidate proposed (non-page count Annex) | 30 |
| Management Plan (2 pgs) | - Description of how the consultant/s will support assessment team before, during, and after fieldwork (e.g. preparation, validation workshop, stakeholders' interviews, etc.) - Description of the consultant/s' home office support throughout assessment activities, including how it intends to ensure technical quality of all deliverables, maintain regular client communication, and comply with all contractual requirements | 15 |
| Institutional Capacity and Past Performance (3pgs) | - Description of consultant/s experience conducting capacity assessment and development activities in Indonesia - Description and previous examples of capacity to quickly replace/add staff or retain staff to deliver the require tasks in the contract | 15 |
| Total: 12 pages | Maximum points (Minimum score for technical compliance: 70) | 100 total |
The expression of interest should also contain Financial Quotation. A financial quotation should be submitted in IDR with VAT. The financial quotation should be based on all estimated costs of the assignment including:
● Consulting fees to be charged;
● Projected logistical and other expenses to be incurred including costs of travel and accommodation.
Submission date
Interested candidates are requested to submit an electronic copy of their expression of interest/ proposal to recruitment@hivos.foundation later than 29 November 2021.
Please put in email subject: SPEAK_EU_Final-Evaluation_Name of consulting company
0 Comments:
Posting Komentar