Developing a monitoring and evaluation framework

Use this page to learn about how to create a monitoring and evaluation framework.

Developing a monitoring and evaluation framework helps you work out which pieces of information to collect to evidence your story of change.

It’s useful to look at this once you’ve read our guidance on the evaluation process and planning your evaluation.

It’s good practice to include people who will be collecting the data when you develop your framework. You could also involve:

• the people and communities you work with
• volunteers
• trustees
• partner organisations
• funders.

It’s best to write your framework before your project starts so you can make sure you’re collecting appropriate data from the beginning.

The example below is based on a monitoring and evaluation framework we’ve created. It’s for a project looking to improve access to employment for people leaving prison.

Be clear about what you’re trying to achieve. Most organisations are driven by a clear sense of shared purpose and mission, but this isn’t always captured on paper.

This can lead to problems when you’re deciding what data to collect for your evaluation.

You can use an evaluation planning triangle or a theory of change to get started.

Set out what you do. This may be an activity (such as a workshop), a service (a helpline), or a campaign. We suggest you don’t include internal processes such as fundraising or recruiting volunteers here.

Example

Activity indicators are specific pieces of information that you collect to keep track and report on the work you’ve delivered.

Activity indicators give you information about:

• what you delivered
• who you delivered them to
• whether the people or organisations you delivered them to were satisfied and thought your work was good quality.

There will be some activities you’re more interested in gathering information on than others.

Set realistic priorities for measurement and try to limit the number of indicators you set.

Many funders and commissioners ask for specific profile information about the people you provide your services to. This may include:

• age
• gender
• sexual orientation
• ethnicity or religious belief.

More importantly, you’ll need this information to understand whether or not your service is accessible to everyone you intend to reach.

Outcomes are the changes that come about as a result of your work.

If you’ve developed an evaluation planning triangle or a theory of change, you’ll have created a list of outcomes.

Make sure you:

• define your outcomes clearly. Avoid woolly outcomes that aren’t clear. Such as ‘service users feel more empowered’. What does this empowerment mean to users? What changes for them as a result? The easier your outcomes are to understand, the easier they will be to measure.
• have described single changes. Outcomes containing more than one idea are difficult to measure.
• have not articulated big, complex changes. For example, instead of an outcome of ‘improved wellbeing’, think about whether you mean improved physical, mental or emotional wellbeing.

Outcomes should describe the changes you want to create, not the work you’re delivering.

Be cautious of writing outcomes that are about accessing your services.

If you’re running skills workshops in prisons, for example, there’s no real value in measuring the outcome ‘increased access to skills workshops'.

This doesn’t tell you anything about what changed for people as a result of your work.

There can be an exception to this if your users are very hard to engage, and getting them to access a service signifies a real change for them.

This might be the case with street homeless people, for example, who have never accessed a day centre and feel uncomfortable indoors.

Prioritising outcomes

Consider which outcomes are priorities for measurement – it’s unlikely you’ll be able to spend time measuring everything that changes. Priority outcomes may:

• be pivotal to your work
• represent the ‘end result’ of a chain of smaller changes
• unlock other outcomes
• be helpful for attracting funding or reporting to existing funders.

Example

Outcome indicators are specific, measurable pieces of information that you can collect to keep track of the difference your work is making.

They tell you whether or not you’re achieving your outcomes and how much change has occurred.

Setting outcome indicators is very useful for deciding what to measure and what not to.

Writing a clear list of indicators will make developing data collection tools (such as questionnaires) much easier.

Review your list of priority outcomes, and think carefully about what information you need to collect as evidence of change.

Your outcome indicators should:

• be expressed in neutral language. They might be phrased as ‘level of’, ‘number of’, ‘type of’ or ‘how often’. Unlike outcomes, outcome indicators shouldn’t use a word that suggests change, such as ‘increased’ or ‘improved’. This is because you need to collect the data before you know whether or not the intended increase or improvement has happened.
• be tailored to your context. There are online indicator banks, but they’re only helpful if the indicators are suitable for the type of work you’re doing, people you’re supporting and changes you want to create. Think carefully about your outcomes. What are the signs that you have been successful? How can you tell when you have made progress? The answers to these questions will generally make strong indicators.

It’s good practice to set at least two indicators to measure each outcome. When putting your indicators together you might do the following:

• Combine numbers (quantitative data) and descriptions or narrative (qualitative data) to give you a fuller picture of what has changed.
• Combine different perspectives. In our example project, to assess whether or not ex-offenders feel more confident in interviews, we could ask them about how anxious they felt about interviews. Since the project offers mock interviews to help people practice feeling confident, we could also ask the mock interviewer about what they thought of the ex-offender’s performance – how confident did they appear?
• Combine subjective and objective indicators. This can give a more robust picture of whether or not a change has happened. For example, if we wanted to know if people working in voluntary organisations had increased their skills around applying for grants, we could ask them how confident they would feel about making an application for a larger grant – this gives us a subjective view on whether or not the outcome has been achieved. A more objective measure could be the percentage of applications that were successful (although other factors could also influence this).

Some of the most important outcomes from your work may be ‘soft’ outcomes – internal changes to the way people feel or think that are impossible to observe directly.

For example, improved self-esteem, improved confidence, increased happiness or increased job satisfaction.

To set indicators for soft outcomes, think about how you would tell if the outcome had been achieved. You may be able to measure observable behaviour changes – sometimes known as setting ‘proxy indicators’.

For example, when self-confidence goes up, you might see changes in the way people talk, dress, or behave with others.

Self-confidence indicators might be readiness to make new social contacts, and how comfortable they are with speaking in a group or making a presentation.

Simply asking people if they think they know more than before will not necessarily give you meaningful or accurate information. It’s better to set more targeted indicators.

For example, a campaign that plans to increase awareness of local recycling facilities could set indicators about whether or not people know where to recycle different materials rather than asking about their ‘level of awareness’.

After drafting your outcome indicators, you may be left with a long list of information to collect. Think carefully about your priorities for measurement.

Which outcomes are most crucial to evidence? If you have set a large number of indicators for each outcome, which are the most interesting or revealing?

Consider whether you’re already collecting information that could be used for monitoring.

For example, if you’re using validated scales and tools, such as the CORE counselling tool, you can use the score as an indicator of overall mental health.

Check your list of outcome indicators. Will they give you accurate information about your outcomes?

If you collected evidence against your indicators, would it be easy for someone external to find issues in your data, or to discount it?

Your indicators will be the foundation for your questionnaire questions, your focus group topics, and your interview schedules so it’s important to make sure they’re measuring what you intend to measure.

Your impact (the broad or longer-term effects of your work) can be difficult to evidence.

It takes time to measure change over time, and a number of different actors and factors will be involved in creating significant, longer-term changes.

In our example project, the impact would be ‘Improved access to employment for people leaving prison’.

It’s easy to see how other factors, such as the work done with prisoners by similar organisations, the job market, and changes to government policy could all affect whether or not this longer-term change comes about.

This makes figuring out the specific role of the project more problematic.

To track impact in a meaningful way, you need to be able to demonstrate the role that your work had to play in creating longer-term change.

This involves understanding the role that other organisations and wider factors have (attribution).

It also involves understanding how much change would have happened if your organisation did not exist (sometimes referred to as the counterfactual, or deadweight).

Getting to grips with these issues often requires a more sophisticated approach to evaluation and a more detailed research design.

Generally, evidencing your organisation’s longer-term impact will need more time, skill and resources than evidencing your outcomes.

However, you may decide to set indicators for keeping track of your impact if you feel you can measure longer-term change in a meaningful way.

In this case, review your impact statement and think about which indicators you would need to collect information against.

In our example project, impact indicators for tracking improved access to employment for people leaving prison might be:

• percentage of ex-offenders entering the workforce after leaving prison
• how soon ex-offenders enter work after leaving prison
• the type of work ex-offenders take on after leaving prison.

To understand why your project is bringing about change, you need to identify the factors that really matter in the way you do things – your processes.

This may include looking at the way you deliver certain activities, or other aspects of your work.

Such as your relationships with other agencies or the way staff and volunteers work together.

It can be helpful to set specific indicators for processes. Examples might be:

• amount of team working
• extent and type of meetings with other agencies
• level and quality of contact with the media.

To complete your evaluation framework, you will need to decide on your information collection methods. Read more about choosing your evaluation methods.