END OF PROJECT EVALUATION

REQUEST FOR CONSULTANTS TO END OF PROJECT EVALUATION

The Adventist Development and Relief Agency (ADRA) is a global humanitarian organization with a mission to work with people in poverty and distress to create just and positive change. ADRA Somalia belongs to the worldwide network, comprised of more than 130 supporting and implementing country offices. ADRA Somalia is seeking to recruit a consultant to conduct End of Project Evaluation in the following regions of Somalia: Puntland (Bari and Mudug); Galmudug (Mudug and Galgaduud); SouthWest State (Bay and Bakool); Hir-Shabelle (Hiran and Lower Shabelle); Jubbaland (Lower Jubba region and Gedo); Banadir (Mogadishu); and in Somaliland (Awdal, Togdheer, Sool and Sanaag).

1. Purpose and Objective of the EOP EVALUATION Consultancy

The assessment will be carried out through interactive, participative methods involving a range of project stakeholders. This end of project evaluation (EOPE) focuses on the processes, tasks or activities executed, and achievements realized during the entire implementation period spanning the period from March 2019 to the present, including the impacts following the intervention.

The overall purpose of the evaluation (EOPE) is to assess the relevance of the project design, scope, implementation, results, and the achievements of project objectives. The EOPE will collate, analyze, and extract credible and applicable information that would facilitate decision-making while incorporating lessons learnt, challenges faced, and best practices obtained during implementation, which will further inform the programming strategy in the subsequent programming phase 2024-2025 in response to the SOWASDIP-II priorities.

Broadly, the EOPE will strategically review performance judged against the project indicators compared with baseline information and overall project targets. The evaluation will assess the impact of the project and sustainability of results including its contribution to capacity development and the achievement of sustainable development goals. The evaluation will also be required to deliver on the following:

1. Assess the relevance and coherence of the project’s design in relation to the theory of change and country needs and how the design aligns with the National Development Plan objectives or, where possible, Sector Strategic Objectives and initial analysis of the underlying root causes.
2. Assess how the project is perceived and valued by the target groups (target communities).
3. Identify the contributions of the project to the SDGs, Somali NDP, the MoEWR, MoH and ADRA’s objectives, and its synergy with other projects and programs in the area.
4. Review the effectiveness and coherence of the project outcomes. Have the project outcomes been building on or strengthening each other in order to reach the objectives?
5. Review the strategies on sustainability and impact of project outcomes, analyzing their effectiveness, unexpected results, and factors affecting implementation, as well as the use of management tools and alignment to the Theory of Change.
6. Review the institutional set-up, capacity for project implementation, coordination mechanisms and the use and usefulness of management tools including the project monitoring tools and work plans.
7. Assess the implementation efficiency of the project, including the scope of the project. Would the efficiency have been better with a smaller project scope?
8. Identify lessons and potential good practices for the key stakeholders.
9. Provide strategic recommendations for the different key stakeholders to improve programming towards the attainment of desired objectives.
10. Identify the impact of the project (what has ‘it’ changed in the lives of the target population?).
11. Scope of Work

The scope of work is broad and must address purpose and objectives, the key evaluation questions (enumerated below), deliver on the activities outlined in the Clause 6 (Timeframe) as well as ensure the following tasks:

1. Familiarize with the local WASH context in Somalia, past and ongoing water and sanitation and hygiene-related projects, the WASH Sector Strategic Plans, and the SOWASDIP project document.
2. Review project proposal, logframe, all the relevant project reports, studies, and surveys on the WASH sector in Somalia and synthesize to ensure that collected data is focused and useful to the EOPE.
3. Review tools and data collection and sampling plans with ADRA before embarking on field work. Where possible, utilize electronic data collection tools (Kobo) and databases accessible in Excel. Relevant data analysis software such as SPSS may be used as appropriate.
4. Capture and document in a tabular form technical details of each infrastructure (for water system & sanitation etc.) and the population feasibly reached by the system.
5. Review the crosscutting themes including gender and non-discrimination, and just transition to environmental sustainability incorporated in the relevant sections to the extent possible.
6. Participate in an independent pre-visit meeting with Sida prior to the fieldwork. Upon finalization of the evaluation, participate in a briefing meeting with Sida on the findings and recommendations.
7. Participate in meetings with SOWASDIP’s respective line ministries and incorporate their views and opinions about the project, its design, implementation, results, achievements and shortfalls.
8. Examine the project’s MEAL framework, evaluate the project log-frame and the overall performance.
9. Assess whether the overall design strategy and implementation approaches delivered in a relevant, effective, efficient manner and with impact and complementarity with other actions.
10. Prepare and submit a comprehensive but concise, well-structured EOPE report with findings that include visual aids (e.g. graphs, charts, and maps). All data and survey tools will also be annexed.
11. Scope of Work

The scope of work is broad and must address purpose and objectives, the key evaluation questions (enumerated below), deliver on the activities outlined in the Clause 6 (Timeframe) as well as ensure the following tasks:

1. Familiarize with the local WASH context in Somalia, past and ongoing water and sanitation and hygiene-related projects, the WASH Sector Strategic Plans, and the SOWASDIP project document.
2. Review project proposal, logframe, all the relevant project reports, studies, and surveys on the WASH sector in Somalia and synthesize to ensure that collected data is focused and useful to the EOPE.
3. Review tools and data collection and sampling plans with ADRA before embarking on field work. Where possible, utilize electronic data collection tools (Kobo) and databases accessible in Excel. Relevant data analysis software such as SPSS may be used as appropriate.
4. Capture and document in a tabular form technical details of each infrastructure (for water system & sanitation etc.) and the population feasibly reached by the system.
5. Review the crosscutting themes including gender and non-discrimination, and just transition to environmental sustainability incorporated in the relevant sections to the extent possible.
6. Participate in an independent pre-visit meeting with Sida prior to the fieldwork. Upon finalization of the evaluation, participate in a briefing meeting with Sida on the findings and recommendations.
7. Participate in meetings with SOWASDIP’s respective line ministries and incorporate their views and opinions about the project, its design, implementation, results, achievements and shortfalls.
8. Examine the project’s MEAL framework, evaluate the project log-frame and the overall performance.
9. Assess whether the overall design strategy and implementation approaches delivered in a relevant, effective, efficient manner and with impact and complementarity with other actions.
10. Prepare and submit a comprehensive but concise, well-structured EOPE report with findings that include visual aids (e.g. graphs, charts, and maps). All data and survey tools will also be annexed.

Key Evaluation Questions

The evaluator shall examine the following key issues:

1. Relevance and strategic fit

• Is the project relevant to achieving Sida’s strategy for Somalia and ADRA’s strategic objectives along with other pertinent regional and global commitments?
• Is the project designed based on a thorough analysis of the underlying root causes of the development objectives it is aiming to contribute to?
• Is the project relevant to the specific needs women and girl beneficiaries?
• How well does the project complement and fit with other ongoing programmes and projects? What are the links established with other activities of other development aid agencies?
• Has the design clearly defined outcomes, outputs and indicators with baselines and targets?
• Was the project design expressed in the theory of change realistic and comprehensive?
• Did the project design include an integrated an appropriate strategy for sustainability?
• Was the implementation approach valid and realistic?
• Has the project addressed gender issues as identified in the project document?
• Were any useful lessons learned that could aid in enhancing the project's design and execution?
• To what extent have the project objectives and their related outcomes and outputs been achieved?
• Has the project successfully built or strengthened an enabling environment (systems, policies, people's attitudes, etc.)?
• Which contributing or challenging factors influenced the project’s success in attaining its targets?
• What, if any, unintended results of the project have been identified or perceived?
• Does the (adopted) intervention model used offer an opportunity for replication or scale up?
• How efficiently have resources (human resources, time, expertise, funds etc.) allocated used to provide the necessary support and to achieve the broader project objectives?
• Was the scope of the project within the limit of the available resources? Were the available technical and financial resources adequate to fulfil the project plans?
• How efficient was ADRA’s monitoring framework and activities? Did the monitoring framework allow for adjustments and adaptation of the work plan and budget?
• Was ADRA’s coordination of project implementation with other relevant programmes and authorities effective?
• In the management and governance arrangement of the project, is there a clear understanding of roles and responsibilities by all parties involved?
• Was the M&E strategy used useful for management, learning and accountability? Is relevant information systematically collected and collated?
• Has the project created good relationships and cooperation with the relevant national, regional, local level government authorities and other relevant stakeholders who implemented the project?
• Did the project receive effective administrative and technical support (i.e. staff time) as well as - policy support from ADRA’s national and regional office?
• In your assessment, what was the intended long-term impact of this action? How is intended long-term impact expressed in relation to the design, implementation and follow-up of the project?
• To what extent have the men and women target population benefited from the project outputs and outcomes? Has the project changed or is likely to change their lives in any meaningful way?
• Is the programme strategy and programme management steering towards impact and sustainability? Has the project exit strategy worked?
• Has the project built the capacity of people and national institutions or strengthened an enabling environment (systems, policies, people's skills, attitudes, curricula, and trainings etc.) in order to contribute to systemic change that will continue to make a difference after the project is finalized?
• How has the sustainability of the project, in context of the national responses and project stakeholders, responded in moving forward the project results?

1. Validity of design
2. Project effectiveness
3. Efficiency of resource use
4. Effectiveness of management arrangements
5. Impact orientation and sustainability
6. Proposed Methodology

This EOPE is a summative qualitative evaluation of the implementation and performance of the SOWASDIP Project using participatory methods. The adopted approaches must (a) identify and interpret key themes that emerge from the qualitative data to understand project impacts; (b) systematically examining the content by stakeholders, uncover underlying meanings and patterns; and (c) explore the narrative analysis of the project's holistic impact. The collected data will be collated, analyzed, and interpreted in a coherent and systematic manner, using detailed data/information analysis method(s) and in correlation with indicators as designed in the project Logframe.

1. Deliverables
   1. An Inception Report prior to the commencement of the EOPE explaining the detailed evaluation design and the implementation plan, that shall be approved by ADRA and Sida.

The following services and outputs are expected for the end of project evaluation (EOPE):

1. Detailed and comprehensive tools to be used to capture the information required for the EOPE. The tools should be discussed and agreed on before utilization and thereafter annexed to the report.
2. Debrief session and power-point presentation to ADRA; and Pre-meeting & debriefing with Sida.
3. Comprehensive draft and final EOPE report. A compulsory annex to the EOPE report is the M&E plan summarizing how data on indicators in the project log-frame were garnered and reported.
4. Soft copy of the database of the primary data analyzed and used in the report.

The draft final report should be submitted in soft copy to ADRA (and SIDA) within 2 weeks of completing field work. The final EOPE report to be submitted (in soft and 1 signed bound hard copy) 2 weeks after receiving consolidated review comments from ADRA/Sida. ADRA’s will approve the report if it meets organizational standards. The report should be concise not exceeding 30 pages (excluding the Annexes) and should contain all relevant parts (e.g., executive summary, findings, lessons learnt, recommendations and annexes).

1. Timeframe

The consultant will be engaged for 31 working days between June 2024 and July 2024. A breakdown of time allocation for the assignment is presented below. The consultant may, however, revise and redistribute the allocated time as deemed appropriate but must work within the provided timeframe.

No.

Activity Description

Duration (Days)

Location

Responsibility/Remarks

1

Briefing by ADRA/ SIDA

0.5

Nairobi

Consultant

2

Literature Review

1.5

Nairobi/Mogadishu

Consultant

3

Development of the EOPE survey tools

1

Nairobi

Consultant

4

Preparation of Inception MTE Report

2

Nairobi

Consultant

5

Presentation & submission of Inception Report to ADRA/Sida (Pre-meeting SIDA)

1.5

Nairobi

Consultant/ADRA& Sida

8

Data Collection & Field visits (2 consultants going to different locations). Including travels between different states.

12

Various

Principal & assistant Consultant & ADRA support team – in different states

9

Data Analysis

4

Central Location

Consultant

11

EOPE report (Draft)

6

Virtual

Consultant

12

Presentation to Stakeholders (ADRA / SIDA / MoEWR, MoH, etc.) & final reviews

2

Mogadishu/Nairobi

ADRA/MOEWR/Sida

Consultant

13

Submission of Final EOPE Report

0.5

Nairobi

Consultant

TOTAL (DAYS)

31

1. Profile of the Consultant

The principal consultant must hold a Masters degree in a relevant field from a recognized institution with proven technical experience. Teams or firms with a Somali national consultant are preferred. Besides having a good experience of working in this sector, the consultant must also have:

• A minimum of 7 years and extensive experience in research including designing and conducting large scale (WASH) studies, inception studies, baseline, midline and final evaluations.
• Proficiency in participatory methodologies for monitoring and evaluating WASH sector work in developing countries, with a familiarity with the Somali context.
• Proven experience in qualitative and quantitative data collection methods.
• Ability to analyse information, evaluate options and to think and plan strategically, and present complex information in a concise and convincing manner, using graphics and innovative forms of communication.
• Experience of working in a politically insecure environment.
• Familiarity, recent previous work of a similar nature will be an added advantage.