Project Background 

In September 2021, Internews began implementing the DRL-funded “Cultivating Local Independent Media in the Balkans (CLIMB)” project in Albania, BiH, Kosovo, Montenegro, North Macedonia, and Serbia. Implemented in partnership with data analytics company Pikasa, the CLIMB project aims to improve access to useful and objective information in the Balkans by supporting local independent media outlets and enhancing media literacy in the region through mutually reinforcing experimental activities that address both the supply of and demand for high-quality information about issues of public importance. The current end date of the project is September 30, 2023.  

The project focused on the following objectives:   

• Objective 1: Independent local media outlets improve the quality, relevance, and appeal of local journalism so that the supply of objective information is expanded. 
  • Activity 1.1: Train Media Practitioners to Leverage Audience Data Analytics  
  • Activity 1.2: Track and Understand Changing Narratives 
  • Activity 1.3: Provide Comprehensive Technical Assistance and Grants to Independent Media 

• Objective 2: Strengthened demand for objective, fact-based, and reliable information in local communities by fostering an expanded domestic audience for it. 
  • Activity 2.1: Support Grants Pool to Test Experimental Formats that Promote Media Literacy 
  • Activity 2.2: Host Balkans Open Space Gathering 
  • Activity 2.3: Provide Partnership Grants to Amplify the Reach of Media Literacy Content 

Through this assistance, the project anticipated the following expected results:   

• Strengthened ability of at least six media partners to analyze data to inform editorial decisions, content formats and business strategies 
• Increased capacity of independent media partners to engage target audiences through creative new approaches, across different formats, channels and niche audiences 
• Greater exposure of content producers to new, replicable methods and formats for promoting media literacy 
• Increased collaboration and knowledge-sharing between independent media and the media literacy community around new approaches to expanding the demand for high-quality, objective information  

Evaluation Purpose and Objectives: 

The purpose of this final evaluation is to comprehensively assess the performance, results, impact, and sustainability of the CLIMB project. The evaluation aims to provide a thorough analysis of the project’s effectiveness in achieving its objectives and expected results. 

The evaluation will specifically focus on the following areas: 

• Effectiveness: Evaluate the extent to which the project has successfully achieved its stated objectives and expected results. Examine the project’s adaptability and responsiveness to the evolving needs of the target countries and beneficiaries. 
• Impact: Assess the impact of the project activities on beneficiaries, including media outlets, individuals and communities. Examine whether the project has contributed to improving media literacy levels and access to objective information in the region.  
• Sustainability: Evaluate the sustainability of project results and the capacity of local stakeholders to continue initiatives started under the project beyond its duration. Assess the extent to which the project has strengthened the capacity of independent media outlets and media literacy actors to sustain their efforts in providing high-quality content and promoting media literacy. Identify potential challenges and opportunities for long-term sustainability. 

Scope of work  

Expected Activities and Outputs 

The consultant responsible for the evaluation will be expected to produce deliverables within a methodology and timeline that is mutually agreed upon with the project staff. The evaluation should include actionable recommendations for future programming or adjustments based on his/ her findings. The evaluation process should consist of three main phases: (1) developing the methodology, (2) data collection, and (3) analysis and report writing. While the evaluation should take into consideration all CLIMB beneficiaries and partners, the evaluator may want to single out case studies for more in-depth assessment. The evaluator will exercise discretion in the handling of sensitive and proprietary information. 

Developing the Methodology: 

• Outline the evaluation framework and methodology used to conduct the evaluation. The framework should include, but is not limited to, the evaluation questions listed in the following evaluation matrix. This matrix is not exhaustive and may be modified in discussion with the project team. A completed version of the matrix should be incorporated into the evaluation methodology submitted by the evaluator. 

EVALUATION QUESTIONS	RELEVANT DATA SOURCES	SUGGESTED DATA COLLECTION METHODS	EVALUATOR PROPOSED DATA COLLECTION AND ANALYSIS METHODS
EFFECTIVENESS: How effective was the project at successfully achieving its stated objectives and expected results, while adapting to the needs of the target countries and beneficiaries, where needed?
In what ways did external factors affect the achievement of the program’s objectives, if at all? If so, in what ways did the program respond to these factors?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Representatives from participating media literacy organizations/initiatives Project narrative reports and M&E data	Document review Interviews with relevant CLIMB program staff, data analytics partner staff,  and consultants Interviews with staff/content producers from partner media outlets and media literacy initiatives	To be listed as part of proposed evaluation methodology.
How effective were the project’s activities at increasing media partners’ use of audience analytics to inform their daily decisions, editorial choices, content formats, and business strategies?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Project narrative reports and M&E data Data analytics partner reports	Document review Interviews with relevant CLIMB project staff, data analytics partner staff, and consultants Interviews with media practitioners involved in the project Analyze qualitative data to assess their understanding of audience analytics, adoption of data-driven decision-making, and strategies to integrate analytics into their daily workflow	To be listed as part of proposed evaluation methodology.
How effective was the project’s support in strengthening the ability of media partners to produce content that preempts and counters disinformation and manipulative narratives?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Project narrative reports and M&E data Data analytics partner reports Sample of content produced by media partners with project support	Document review Interviews with relevant CLIMB project staff, data analytics partner staff, and consultants Interviews with partner media outlets	To be listed as part of proposed evaluation methodology.
How effective were the project’s experimental media literacy grants (Activity 2.1) and partnership grants (Activity 2.3) at increasing the reach of media literacy initiatives in local communities in the target countries?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Representatives from participating media literacy organizations/initiatives Project narrative reports and M&E data	Interviews with relevant CLIMB project staff Interviews with content producers and media representatives involved in experimental media literacy projects and partnership projects Document review (analysis of project narrative reports, grant proposals and reports) Analysis of relevant M&E data (results from surveys distributed to project participants, reach and impact metrics)	To be listed as part of proposed evaluation methodology.
IMPACT: How significantly did the project activities impact its beneficiaries, including media outlets, individuals, and communities?
To what extent did the trainings/mentorship, grant support, and analytics services provided by the project help media partners under Obj. 1 to improve the quality of their content?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Project narrative reports and M&E data Sample of content produced by media partners with project support Data analytics partner reports	Document review Interviews with relevant CLIMB project staff, data analytics partner staff, and consultants Interviews with staff from partner media outlets Analyze quantitative and qualitative data from partner outlets including review of a sample of content produced with project support, as well as audience trends, content reach, content consumption patterns on platforms such as Pikasa and Google Analytics	To be listed as part of proposed evaluation methodology.
To what extent did the trainings/mentorship, grant support, and analytics services provided by the project help media partners under Obj. 1 to improve the relevance of their content to their target audiences?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Project narrative reports and M&E data Sample of content produced by media partners with project support Data analytics partner reports	Document review Interviews with relevant CLIMB project staff, data analytics partner staff, and consultants Interviews with staff from partner media outlets Analyze quantitative and qualitative data from partner outlets including review of a sample of content produced with project support, as well as audience trends, content reach, content consumption patterns on platforms such as Pikasa and Google Analytics	To be listed as part of proposed evaluation methodology.
To what extent did the trainings/mentorship, grant support, and analytics services provided by the project help media partners under Obj. 1 to improve the appeal of their content to their target audiences?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Project narrative reports and M&E data Sample of content produced by media partners with project support Data analytics partner reports	Document review Interviews with relevant CLIMB project staff, data analytics partner staff, and consultants Interviews with staff from partner media outlets Analyze quantitative and qualitative data from partner outlets including review of a sample of content produced with project support, as well as audience trends, content reach, content consumption patterns on platforms such as Pikasa and Google Analytics	To be listed as part of proposed evaluation methodology.
To what extent did the partnership grants (Activity 2.3) and Open Space Gathering conference (Activity 2.2) contribute to increased collaboration and knowledge-sharing between independent media and the media literacy community?	Relevant project staff, and consultants Representatives from participating partner media outlets Representatives from participating media literacy organizations/initiatives Relevant conference speakers and participants Post-conference survey responses Project narrative reports and M&E data	Interviews with CLIMB project staff Interviews with content producers involved in partnership projects, as well as conference speakers and participants Document review (analysis of project narrative reports, partnership grant proposals, grant reports)	To be listed as part of proposed evaluation methodology.
Did the project have any unintended results/outcomes?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Representatives from participating media literacy organizations/initiatives Project narrative reports and M&E data	Document review Interviews with relevant CLIMB program staff, data analytics partner staff, and consultants Interviews with staff/content producers from partner media outlets and media literacy initiatives	To be listed as part of proposed evaluation methodology.
SUSTAINABILITY: To what extent are the results of the project sustainable?
Did any media partners adopt practices as a result of the project that they plan to continue beyond the life of the project?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Project narrative reports and M&E data Data analytics partner reports	Document review Interviews with relevant CLIMB project staff, data analytics partner staff, and consultants Interviews with media practitioners involved in the project Analyze qualitative data to assess their understanding of audience analytics, adoption of data-driven decision-making, and strategies to integrate analytics into their daily workflow	To be listed as part of proposed evaluation methodology.
Are any of the supported media literacy approaches expected to be replicated after the end of the project?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Representatives from participating media literacy organizations/initiatives Project narrative reports and M&E data	Interviews with relevant CLIMB project staff Interviews with content producers and media representatives involved in experimental media literacy projects and partnership projects Document review (analysis of project narrative reports, grant proposals and reports) Analysis of relevant M&E data (results from surveys distributed to project participants, reach and impact metrics)	To be listed as part of proposed evaluation methodology.
If any changes do not appear to be sustainable, why not?	Relevant project staff, data analytics partner staff, and consultants Representatives from participating partner media outlets Representatives from participating media literacy organizations/initiatives Project narrative reports and M&E data	Document review Interviews with relevant CLIMB program staff, data analytics partner staff,  and consultants Interviews with staff/content producers from partner media outlets and media literacy initiatives	To be listed as part of proposed evaluation methodology.

Data Collection: 

• Desk-based compilation and assessment of relevant existing project materials, documents and data, at the discretion of the evaluator (potentially including project proposals, narrative reports, Pikasa’s data analytics reports, results from the M&E framework and indicator-related pre- and post-training tests) 
• Hold 10-15 online meetings/interviews with a selection of project management staff, implementing partner staff, consultants and trainers, participating media partners, and other stakeholders throughout the project’s target countries to evaluate project outcomes. The list of interviewees and interview questions should be drafted and agreed upon in consultation with the project team.  

Analysis and Report Writing: 

• Assess all data and insights collected to address the questions in the evaluation framework, triangulating data to validate findings, where relevant 
• Produce a concise report (no more than 20 pages) that would include: 
• Executive summary, highlighting the evaluation’s main findings, conclusions and recommendations (approximately one page) 
• Main findings from the analysis of the project’s performance in terms of its effectiveness, impact and sustainability, drawing on specific examples from project beneficiaries, target audiences, and partners 
• A list of key recommendations that are specific, action-oriented, and tailored to all relevant stakeholders 
• Provide an initial debrief to the CLIMB project team on the core findings from the evaluation and incorporate any suggestions as needed into first report draft 
• Update the report as required, based on feedback on an initial draft from the CLIMB project team 

Deliverables 

• Initial debrief with Internews staff on the core evaluation findings (before first draft is submitted) 
• Final evaluation report  

Qualifications/Experience 

• Bachelor’s degree in social sciences, international development, evaluation, or a related field OR equivalent professional experience  
• Demonstrated experience with conducting mid-term and final external project evaluations, including monitoring and evaluation experience, application of interviewing techniques, qualitative and quantitative data collection and analysis, research and statistical analysis, etc. 
• Knowledge of media and political environment in the Western Balkans region 
• Excellent inter-cultural communication skills/cultural sensitivity 
• Required fluency in English and preferred fluency in at least one local language spoken in the Balkans 
• Upholds Internews’ Core Values and demonstrates commitment to fostering a culture of Belonging, Dignity, and Justice 

Estimated timeframe: Approximately 18-25 days according to estimated timetable below. The final evaluation report must be submitted 45 days before the project end date. Specific deadlines for each required task will be determined in agreement with the selected consultant upon hire. 

REQUIRED TASKS	ESTIMATED LOE	DEADLINE
Conduct desk research of existing project materials	3-4 days	TBD
Finalize research methodology, evaluation framework, list of interviewees, and interview questions.	3-4 days	TBD
Conduct interviews with key informants	4-7 days	TBD
Analysis of information collected	5-6 days	TBD
Debrief with Internews staff on evaluation findings	1 day	TBD
Completion and submission of first draft following initial debrief	1-2 days	TBD
Finalize evaluation report based on Internews’ comments	1 day	TBD

Application Requirements

Interested parties should apply by August 20, 2023, to [email protected] , with attention to Project Director, Samra Campara, who will be leading and directly managing the evaluator. Applications should be written in English and include the below documentation: 

• A two-page statement of interest/draft proposal, highlighting how your methodological approach will address the scope planned for this evaluation and how your profile meets criteria  
• Updated CV/resume of evaluator, including contact information for two professional references 
• Evidence of previous evaluation work or similar services (e.g., link to or PDF attachments for evaluation reports) 
• A short cost justification and/or cost proposal (in USD), including all costs, including transportation, translation services, etc.