An evaluation is an important work for charities and voluntary organisations, whether its a service user evaluation or as large as a full-service evaluation.

Quality Matters will work closely with services to achieve great results and provide high-quality, well-evidenced reports. But how many organisations know where to start asking questions about evaluations? This article will help answer how to start thinking about an evaluation and how Quality Matters can help your service. 

1. We are planning a new programme, how should we be evaluate it?

This is a great position to be in. By engaging your consultant at the planning phase you will be in the best position to get a high quality evaluation. An evaluator will work with you to agree an outcomes framework. This is essentially agreement on what change you are planning to create and a way of measuring this.

An evaluator will work with you to agree an outcomes framework.

Often services understand the general change they wish to make, i.e. in the example of a programme where the goal is that people to understand about sustainability and make small lifestyle changes to be more sustainable, however this is not quite specific enough to test. Developing an outcome framework should assist you to understand exactly what change you want to see, i.e. reducing household power consumption by 15%. Once you have developed clarity on the exact change you wish to make, this can have useful knock-on effects on your programme design. In this example you need to be able to measure power consumption, so each household will have to have some simple technology to do this. Once you have agreed how this will occur (and be funded), the programme has just developed an interesting new component – participants can now measure their own change. That could greatly improve the process.
Often when people clarify their outcomes, they realise the programme needs to be adapted slightly or that other supports for service users need to put in place if service users are really going to be successfully in achieving the specific positive outcomes.
On the other hand, If you don’t plan what data you will collect at the beginning of the process this will lead to real challenges and lesser quality evaluation at the end of the programme.

Interested in Outcome Measurement? Read our blog article on choosing your outcome targets.

2. What does process and summative evaluation mean?

A process evaluation helps you review how you well your work is going for those involved, this is generally focused on their experience and on improving the way you operate. A process review is useful for new projects. It involves all stakeholders reflecting on what is working and what could change.
A summative evaluation looks at programme effectiveness, i.e. what changed, and potentially the value of this change. These types of evaluations can also assist in improving the programme, however their primary focus us to test whether the programme has an adequate rationale to continue. The conclusion will be based on whether it is considered effective, i.e. did it create positive change and value for money. This article has a focus on these types of evaluations, which are dependant on good outcome data.

3. What’s a Social Return on Investment (SROI)?

An SROI is a type of summative evaluation that reviews the change that occurred for each stakeholder group, whether planned or unplanned, negative or positive. It does not from the perspective of stakeholders not based on what the service planned to happen. It then goes one step further and seeks to value this change using euros as a way of establishing a value for each outcome, even ones that don’t have a readily available price such as, for example: “an increased feeling of safety”. SROI considers other important things, such as: how much change would have occurred anyway, how much is the programme under review responsible for the change that occurred and what effect and value will this change have for stakeholders over the coming years. Including these factors in the evaluation, make SROI into an interesting and deep narrative.
An SROI comes with a set of “accounts” or an outcome map detailing the change experienced by all stakeholder groups and the value of the change. This map is useful in assisting the service to deciding what actions that they undertake are the most valuable and where they should focus attention in order to create the most value for their services users.
Finally, SROI provides you with an assessment of the value of your work against the costs of running it. SROI is the story of your organisation and the change it made, using money as a way of describing value. SROI speaks to the values of organisations that understand that often what counts most: opportunity and connection for instance, is frequently undervalued. SROI addresses this challenge directly and in a robust manner, by counting this change and valuing it.

Interested in Social Return on Investment? Read more about adding SROI methods to your work.

4. What makes a good evaluation?

The best evaluations all include and comparison of pre and post data. This means that you need to decide at the beginning of the process what data you will collect and then collect this at the end of the process to evaluate the change that occurred.
Psychologists, such as Daniel Kahneman, have proven that people are not very good at assessing how they were feeling or behaving retrospectively. For this reason an evaluation that occurs at the end of an intervention and asks people about the change they experienced is of much less quality than one that asks people the same questions at the beginning and end of an intervention (evaluators will look for surveys and tools that have been used before and are considered robust tools to assess change).
Of course quotes and narrative about the change also help bring the data to life. But stories without data do not create confidence in the reader.

5. What happens if you have started your programme? (e.g. “Our clients say we work, but we don’t have any data”)

This is a really common challenge facing many services and there are a number of reasons why this is so common. There is often pressure to beginning programme quickly, everyone is action focused at the onset of a programme. Often people don’t reflect on the importance of being able to show value until much later in the process and of course planning and collecting outcome data costs money. Also many people, having not engaged evaluators before simply think – ‘that is something that happens at the end’.
So what to do?
Well the first thing is to waste no time in creating some pre and post data measures for new services users or programmes, as described above. The other approach is to undertake a forecast SROI using the information you have from past programmes to describe what you expect the change its valuation to be.
The outcome from developing a forecast SROI will be a deep understanding within the team of how you think you create value. This will be formed through thorough engagement with past stakeholders. From this process an outcome framework will be developed that will help you assess whether you made this change. The outcome map can then be adapted to provide data on project outcomes and value.
This method draws from past data and experience, provides an short term narrative on value, as well as a longer term very robust approach to data collection.

6. What’s the best way to hire a consultant?

We always recommend on open tender, or if time is short, getting a number of consultant proposals to ensure you have a good match. In order to help you decide which proposal best suits you, let them know the objectives, timeframe, and provide a range for the price. If you also give a marking schedule that involves at least 15% for value of money or price, then you are going to get comparable proposals. If you don’t mention a budget range, you are likely to get an amazing proposal for more than you can afford and cheap one that doesn’t do all of what you want.