The context

Relief Applications and The Alternatives Factory conducted an external evaluation of ITCILO’s 2024 Social Protection training portfolio for the SPGT Programme. This external evaluation assessed the design, delivery, and outcomes of selected online, blended, and face‑to‑face Social Protection trainings delivered by ITCILO in 2024. The scope covered standard evaluation criteria (relevance, coherence, effectiveness, efficiency, impact, sustainability) and alignment with ISO 29993:2017 learning‑services standards and ITCILO’s Plan–Do–Check–Act (PDCA) quality‑management cycle. The work included desk research, data extraction from MAP/eCampus and course materials, participant survey, key‑informant interviews with ITCILO staff and institutional clients, focus group discussions with learners, and the development of five case studies showcasing good practices and outcomes.

Methodology

• Desk Review: Analysis of training documents, course materials, and post-training evaluations to establish context and inform data collection tools.
• Online Survey: Sent to over 1,800 participants across modalities, collecting data on demographics, training design, learner support, delivery preferences, and outcomes.
• Key Informant Interviews (KIIs): Conducted with ITCILO staff, programme managers, and institutional clients to gain strategic and operational insights.
• Focus Group Discussions (FGDs): Held with diverse training participants to explore learning experiences, application, and impact.
• Case Studies: Five in-depth cases were selected to illustrate concrete examples of training uptake and organizational impact. Data from these sources were triangulated using statistical analysis (e.g. ANOVA, regression models) and thematic coding to generate evidence-based findings and recommendations.

Technology Used

• Quantitative analytics: Python 3.11 (JupyterLab); pandas, numpy, scipy, pingouin (reliability; KMO/Bartlett/Cronbach), statsmodels (OLS/logit/ordinal), scikit‑learn (validation & diagnostics); matplotlib & seaborn (figures). Version logging ensured full reproducibility.
• Qualitative support: Gemini (interview transcription); NotebookLM (document querying). Final coding/interpretation done manually with evaluator cross‑checks.
• Reproducibility & QA: Version‑pinned environment, documented notebooks (.ipynb), seeded analyses, codebook, and full pipeline/log archives.

In partnership with The Alternatives Factory, Relief Applications brings stakeholder-centred evaluation leadership together with a reproducible, ISO-aligned analytics stack—turning training data into auditable evidence and actionable improvements across programmes.