An impact evaluation of the Northern Ireland European Social Fund Programme 2014-2020

Key findings

A review of NI European Social Fund (ESF) performance has identified the following headline findings:

• the programme has been delivered consistently against aims and objectives
• in terms of management and governance, it is considered effective and has received praise from beneficiaries. The Managing Authority (MA) has demonstrated pragmatism, fairness and collaboration in its approach
• for Priorities 1 and 2 beneficiaries were selected via a ‘call’ process that improved considerably between Call 1 and Call 2. The process under Call 2 is regarded as both effective and efficient
• project targets are being met or surpassed, and financial performance is aligned with expectations.

The information below summarises the evaluation findings, aligned to the key questions posed in the Terms of Reference:

Have the needs of the area, as defined in the European Social Fund Programme for Northern Ireland, changed since the assistance was approved and to what extent is the strategy still relevant?

The needs of the area had been improving significantly at a ‘macro’ level; but economic inactivity remained a challenge. Since the Covid-19 pandemic, several years of labour market progress have been reversed and the NI economy faces major challenge. Support with respect to employability and skills remains as relevant (and perhaps more so) now as when the assistance was approved.

What progress has been made toward achieving the quantified targets for expenditure, outputs, and results?

Progress towards 2023 targets has been rapid, with significant achievement across each Priority Area.

What progress has been made toward measuring ‘qualitative’ outcomes of the programme?

This evaluation aimed to test qualitative outcomes via a representative sample of participants. It is regrettable that email contacts were not readily available and that the process to try and secure email addresses was complex and ultimately disappointing. Difficulties in securing participant consent to take part in the survey were further exacerbated by the impact of Covid-19. However, the results from the survey are indicative of qualitative impacts being made via the programme. Significant proportions of respondents indicated that confidence and ‘softer’ skills had benefitted from their participation.

It was also evident from the conversations with beneficiaries that many are assessing their own qualitative impacts. While this is welcome, there is no consistency of approach across the programme. On a positive note, the use of case studies is a valuable resource that brings the qualitative impacts to life.

What progress is being made toward achieving the planned impacts of the European Social Fund Programme for Northern Ireland?

Survey respondents noted a positive impact on participants’ behaviours and skills, a key impact of the Programme. The survey also supports an assessment that notes how 89.5 percent of respondents said the NI ESF Programme had an impact in helping them get employment – 63.2 percent said it helped ‘A Lot’ – with only 5.3 percent saying it had no impact at all.

Almost three quarters (74.3 percent) of respondents agreed that without the NI ESF Programme they would not have entered employment or education/training, with only 8.6 percent disagreeing. A follow-up question asked if participants believe the NI ESF increased their chances of finding employment or undertaking education/training, 91.7 percent of respondents agreed that it had increased their overall chances.

These findings point toward the NI ESF displaying high levels of additionality and personal impact. The consultation process revealed a similar result, with NI ESF beneficiaries stating that without the NI ESF Programme their project would be significantly smaller and more limited in scope, resulting in fewer positive outcomes.

Have the agreed cross-cutting themes and horizontal principles of gender equality and equal opportunities - and sustainable development in particular - been integrated successfully into the European Social Fund Programme for Northern Ireland?

The MA and beneficiaries are committed to promoting equality of opportunity for everyone and will ensure that no individual will be treated less favorably on the grounds of religious belief, political opinion, race, age, marital status, sexual orientation or disability or any other unjustifiable grounds. These principles are well embedded into Northern Ireland’s government and business environments. The Evaluation Team’s review of the project activities indicates that they are available to all eligible individuals across NI and the border area.

What has been the added value of the European Social Fund Programme in Northern Ireland?

In addition to considerable success against ‘hard’ metrics, the NI ESF Programme has delivered added value through offering a ‘wrap around’ support approach to cohorts of unemployed people who are particularly distant from the labour market and who have complex needs. The evaluators were particularly struck by anecdotes of beneficiaries providing intensive mental health and wellbeing supports to vulnerable participants and going ‘above and beyond’ for participants with significant personal, societal and financial barriers to taking part in programmes.

What value for money does the European Social Fund Programme in Northern Ireland offer?

Costs per participant are €2,232, which compare favourably with projections for the whole NI ESF Programme (€3,789 per participant). 'Value for money' is also conservative, estimated by calculating a wage level for the people that have gained employment following participation. With 11,690 participants having gained employment, a conservative estimate suggests a wage impact of €212m across one year. Grant Thornton calculates that for every €1 of NI ESF funding, an additional €0.77 in wages has been generated in a year.

Recommendations

This review has highlighted the following issues for consideration in any successor programme.

Evolution, not revolution

The rapid success and notable achievements of the NI ESF provide a working template for future programmes. It was evident that the changes made to the application process between Call 1 and Call 2 of the programme have been successful. The value for money achievements and key success factors should be the goal for any successor scheme. A key strength of NI ESF has been the flexibility of the programme and the ability for beneficiaries to invest time and energy into relationship building and provision of tailored, demand driven supports. In the context of an exceptionally challenging ‘post Covid-19’ labour market, there are likely to be complex employability challenges emerging. Any successor scheme should therefore maintain the successful principles of the NI ESF while being sufficiently able to evolve to address emerging labour market challenges.

Match funding matching service

Match funding can prove difficult and administratively burdensome to secure. Typically, a match funder will also require a bespoke application and monitoring information. It could be more efficient if match funders were signed up to a central fund by the MA and matched to appropriate beneficiaries. It would further enhance efficiency for beneficiaries to apply to one ‘pot’ and, during the course of a programme, provide one version of monitoring information.

Real time evaluation

Appointing evaluators at the beginning of the programme who could follow the process through, and develop monitoring and impact assessment templates in support of beneficiaries is worthy of consideration. At a minimum, given the constraint to undertaking the participant survey, there should be an opt in/out clause inserted to participants’ programme inductions that confirms that the MA and beneficiary can share their name, telephone number and email address to third party organisations for the purposes of evaluation.

Maintain a relationship-building approach

People availing of services can identify as having a set of complex needs, which might then change over a period of time. A key element of NI ESF’s success comes from the relational approach to engaging with participants rather than a transactional approach. This approach should be maintained and enhanced through developing a consistent, programme wide, means of tracking ‘distance travelled’ among participants.

Remove Level 1 cap

The inability for beneficiaries to train beyond Level 1 appears counterproductive. The Level 1 cap should be removed, allowing beneficiaries to offer Level 2 / 3 qualifications in-house.

Raise the apprenticeship targeted age

The age cap for apprenticeships is deemed to be unnecessary and it is understood that the Department for the Economy is giving consideration to addressing the age cap for apprenticeships.

Equal status with mandated programmes

Frustration at the process whereby participants on NI ESF Programmes can be moved to ‘Steps 2 Success’ was clearly evident from the consultation process. This is leading to instances where participants are starting NI ESF courses and then having to leave. It is recommended that participants receive a full deferral from a mandated course if they are already on an NI ESF Programme course.

Recognition of ‘soft skill’ accomplishments and broader ‘hard outcomes’

In addition to the ‘hard outcomes’ which are recognised in the NI ESF Programme (that is, paid employment and progression into training/education), there is a broad spectrum of outcomes that could be considered as success measures. For example, people progressing into voluntary work and other sustainable activities that build social inclusion and employability skills (e.g. health and wellbeing groups, support groups, local interest groups) are not counted in the ‘hard metrics’. Consideration should be given to capturing these outcomes as successes. Likewise, the journey towards employability and the improved range of ‘softer’ skills and behaviours that was evident throughout the evaluation should be captured. Grant Thornton echoes the Education and Training Inspectorate Chief Inspector’s Report which stated that “across the projects, there is a need for better measurement of progress made by participants, in particular their development and application of the softer skills”.

Full report

The full report is available:

• Impact evaluation of the NI ESF Programme 2014-20