Programme Evaluation

Evaluating programmes and activities means to measure if the objectives have been efficiently achieved and to identify the impacts of the actions taken. Clearly, evaluation focuses on quantitative outputs (e.g., number of training events held, number of students, number of visits in SMEs, number of conference participants, etc.) but evaluation also tries to find out whether the quality of the programme and activities was satisfactory to reach the objectives and goals.

Furthermore, evaluation looks into efficiency and management topics. Have the resources dedicated to the programme and activities been used in the most economic way, i.e., achieving the objectives with the minimum input? The evaluation of management practices and processes is described in detail in the chapter on "internal evaluation".

Note 22 Why should a support institution evaluate its programmes and activities?

How to plan the evaluation of your programmes and activities

The institution needs to plan the evaluation of its programmes because it has to ensure the systematic collection of data before, while and after the activity has taken place. Acting when everything has already happened is too late. A lot of information may not be (correctly) available anymore.

The programme will be evaluated in terms of:

·    Impact or contribution to achievement of strategic goals

·    Achievement of annual objectives

·    Financial and programme management

a)    Focus on the strategic goal and objective of the programme and its (sub)activities

List for the programme you plan to evaluate the annual objectives and expected outcomes/strategic goals.

b)    What do you need to ask in order to know if the objectives and goals were achieved efficiently?

This step is important to find out what you need to observe in order to see if your programmes were successfully achieving the outcome and related objectives.

Example:

Outcome(strategic goal): Improved purchasing practices of SMEs in the paper sector.

Objective: Train 20% of all buyers in SMEs in the paper sector in professional purchasing techniques.

Questions:

-      How many buyers have been trained in SMEs in the paper sector?

-      Did the buyers improve the purchasing practices of the SMEs they are working for?

-      Did the programme manager use the available resources in the most efficient way?

It can happen that a programme reaches the annual objectives but fails to make the desired impact. In the above example this would mean that the 20 % of buyers were trained but that the buyers did not improve the purchasing practices. This result of the evaluation indicates that you need to investigate the reasons for this unexpected behaviour and find valid solutions for the future.

c)    What information do you need to have in order to answer these questions?

Now, you have to list the information you would collect to be able to answer the questions.

Example:

Total number of buyers in SMEs in the paper sector trained in year one

Total number of buyers in SMEs in the paper sector in country x.

Improvement of purchasing practices in the SMEs where the trained buyers are working

Programme budget deviations

Programme work plan deviations

Complete programme documentation

d)    What are the indicators for achieving the objectives and goals?

From the above example you will see that the first two questions (number of buyers trained…) are clearly defined and you will already try to imagine how and from whom you could collect this information. How about the improvement of purchasing practices in the SMEs the buyers are working for? This information will not exist as such. You will have to find information that indicates that purchasing practices were improved. What could these indicators be?

First of all, improvement means that something is better in the current period than in the past period(s). This refers in our case to the period(s) before the buyer attended our training and the period after he attended the training.

Secondly, good purchasing practices refer to the activity of ensuring the right goods and services are delivered in the right quality at the right time and in the right place. Furthermore it means that the process of purchasing is managed in a professional and ethical way. How can we measure if the newly trained buyers succeeded in applying improved purchasing practices?

-      ratio of purchasing cost to turn-over in the SMEs was smaller than the average of the three years before (they were more cost efficient)

-      the number of complaints due to delayed or incorrect supplies was decreasing compared to the average of the past three years

-      the buyer felt better able to do his job than before attending the training

-      the buyer used new techniques and practices he/she learned in our training

-      the buyer increased the number of alternative suppliers in the supplier base

e)    Is this information and data available either in your institution or from other sources and at which cost?

The above example shows that for one objective or statement a whole series of indicators can be developed. Unfortunately, the support institution might not be able to get all the information that it would need for the indicators chosen. In the above example we can imagine that the SMEs might not want to inform us about their purchasing cost or their turnover and many SMEs might not know about the number of complaints or incorrect supplies. Whereas it seems more likely that the buyers can inform us about the changes they perceived or did not perceive after returning from our training event.

Evaluation should consider both subjective (perceived) and objective (fact based) as well as qualitative and quantitative (numbers) information. Therefore, try to get information from different sources and with different methods, e.g., telephone interview buyers and their colleagues regarding their experiences after the training, send a questionnaire inquiring factual data from the SME’s financial reports and desk research factual quantitative data from economic statistics or reports.

Information collection takes time and is costly, so the institution will have to focus on a few indicators that it considers valid to judge whether the objectives and goals have been achieved.

f)     How can you gather this information?

There will be both information being generated internally by the institution and externally by programme beneficiaries, partners, public authorities, etc.

The institution will have to fix the indicators and the information that it needs to collect per programme in advance. Depending on the needs the programme management will develop the plan how to collect the information. Typical means are client questionnaires, feed-back from sales people or client presentations, budget and cash-flow analysis, phone surveys, analysis of existing statistical data. For example most of the training institutions use a standard student questionnaire to evaluate the training event.

Make sure that the right information is available for all before/after comparisons. For example you want to measure the success of your awareness workshops. You need to collect information about the state of awareness “today” before starting your workshops. After having done all workshops you will collect the state of awareness again and then compare it to the former one.

Internal information can be drawn from a complete programme and activity documentation. The programme manager should ensure that the following information is available:

·    The programme plan

·    The programme work plan

·    The programme budget

·    Financial management throughout the programme incl. cash-flow calculations

·    All communication exchange with staff and external parties

·    All transaction receipts (invoices, banking receipts, cash receipts, purchasing orders, etc.)

·    All contracts especially referring to donors, clients, independent staff, event organisation, equipment

·    Chronological documentation of all activities, participants, questionnaires and related information

·    Documentation of sales and client acquisition activities

·    Contents developed and used within the programme (publications, learning materials, presentations, etc.)

·    Any other administrative documentation

External information:

·    Surveys done with clients, member, donors, etc.

·    Observations, case studies of certain activities done by external people

·    Experience exchange or feed-back events with clients, members, donors, etc.

·    Publications and reports from public authorities or other organisations

·    Newspaper articles, journal contributions, Internet

·    Public statistical office data, directories

·    Any other suitable source of information that allows to make conclusions regarding the impact of the programme and/or the achievement of objectives

Plan your evaluation with the help of InnoNets Workstation.

g)    Analysis

Once all the relevant information to calculate the indicators is available the responsible person(s) for the programme evaluation will look at the aspects of:

·    Quantitative achievement of objectives

·    Qualitative achievement of objectives

·    Programme impacts

·    Programme and activity management

Any outcome that concludes sub optimum management and/or non-achievement of objectives should be discussed with the programme management. Some findings will need to be further looked at. Within the evaluation community there are many ongoing discussions about the role of the evaluator, the purpose of evaluation, use of evaluation results, etc.

The evaluation of your programmes should be seen and conducted in a very constructive way. How can you learn from your activities and continuously improve what you do if you do not know about the effect and impact of your actions? Evaluation helps you to find out what works and what does not. Therefore, it is an invaluable management development instrument.

Note 23 How to use evaluation results

Workshop/Training feed-back questionnaire