Trade Support Institutions (TSIs) are increasingly under pressure from
their stakeholders to demonstrate that their services provide significant and
lasting positive changes for their clients. TSIs have traditionally focused on assessing
their efficiency, but not the impact of their work. Quantitative measures are necessary
and important, but they do not provide the information TSIs need to assess the benefits
of their services for their clients. ITC provides advisory and training
services to assist TSIs to assess the outcome and impact of their services.
Information
Organizations have been exploring
new ways to measure results by shifting the focus from examining how a service
is operated to considering the good that the service achieves. Evaluating the
impact of a service is a crucial activity for all project managers and,
furthermore, good performance measurement systems will assist managers to
identify areas of the service offer that may need improvement. For more information
and links to online resources related to impact assessment, please expand this
section. Donors too must
measure the impact of the resources they provide to organisations, and their
reporting needs are often different from each other. Many donor organisations prefer
to use a logical framework as it allows for correlations to be made between
activities, outputs and client satisfaction. Recently however, donors have been
requesting that more information be provided concerning the positive changes
that their resources instigate, so they can make better decisions on the allocation
of those resources. In response to this TSIs are beginning to assess and report
on the changes that are a direct result of the TSI’s service.
The Canadian International Development Agency
(CIDA) is one donor agency that provides tools to assist project managers to
track and report on the changes in behaviour of their clients. Templates of
these two tools can be found
here.
Data and Research
In 2004 the World Bank pre-empted the upcoming shift in
the way TSIs could assess results. The table below compares the key features of
“implementation monitoring” against the new approach, “result monitoring”.
ELEMENTS
OF IMPLEMENTATION MONITORING
|
ELEMENTS
OF RESULT MONITORING
|
| Description of the problem or
situation before the intervention |
Baseline data to describe the
problem or situation before the intervention |
| Benchmarks for activities and
immediate outputs |
Indicators for outcomes
|
| Data collection on inputs,
activities, and immediate outputs |
Data collection on outputs and
how and whether they contribute toward achievement of outcomes |
| Systematic reporting on
provision of inputs |
More focus on perceptions of
change among stakeholders |
| Systematic reporting on
production of results
|
Systemic reporting with more
qualitative and quantitative information on the progress toward outcomes |
| Directly linked to a discrete
intervention (or series of interventions)
|
Done in conjunction with
strategic partners
|
| Designed to provide information
on administrative, implementation, and management issues as opposed to
broader development effectiveness issues
|
Captures information on success
or failure of partnership strategy in achieving desired outcomes.
|
Source: World
Bank 2004 Report: “10 Steps to a Results-Based Monitoring
and Evaluation System"
In
order to be able to demonstrate their “value for money”, more and more TSIs are
beginning to measure the outcomes, the impact and effectiveness of their
programmes and services, rather concentrating solely on reporting on their
activities and consequent outputs. Advisory Services
ITC provides consultation to TSIs interested in
establishing an impact assessment framework for their organizations. The consultation focuses on:
- Impact assessment approaches in the context of
development
- Establishing and implementing an impact
assessment system
- Key elements of the impact assessment framework
- Analyzing and reporting the results.
Training
The TSI capacity building programme provides support to TSIs in
their efforts to improve their own efficiency and effectiveness by equipping
them with the knowledge and tools to measure their own results. These assessments allow them to:
- Justify the financial support of
governments and development partners by
documenting the positive effects achieved with their resources. If TSIs are not
in a position to demonstrate results, they will not win public support.
- Improve the effectiveness of services by analyzing why they did not achieve the desired results. If TSIs
cannot identify failures, they cannot correct and learn from them.
- Adapt or improve the portfolio of TSIs’ services by focusing on those producing the greatest benefits
for clients. If TSIs cannot identify success, they cannot reward it.
- Retain or attract new customers by demonstrating the benefits derived from the use of specific services.
The training provided under this module enhances the
skills of TSIs to measure the outcomes and impact of their services.
To do this, it provides information on how to:
- Formulate
the main elements of an impact assessment framework – including the
identification of relevant outcomes and impact measures
- Select
the most relevant and efficient data collection methods
- Customize
the impact assessment system to TSIs’ current portfolio of services
Translate the results of the services into
reports for governments and stakeholders.
Projects
ITC is providing support under
the PACT II programme to the African regional economic commissions of
COMESA, ECOWAS and ECCAS to implement and monitor trade related technical
assistance programmes: measuring the impact of the services is an integral part
of the programme. ITC is also providing advisory
services to Brazil’s Trade Promotion Organization, Apex-Brasil, to augment
their current recording and reporting systems through assisting them with the
development of relevant outcome and impact measures for their services.
ITC will launch a comprehensive performance management training
programme for TSIs in 2011. Impact assessment is one of the modules included in
this programme. More details to follow.