Doing participatory evaluation with community projects
Stronger Families Learning Exchange Bulletin No.6 Spring/Summer 2004 pp.10-13
Maya Haviland presents a broad model of the principles and stages of participatory evaluation.
This paper is drawn from the work of the Stronger Families Learning Exchange (SFLEX). It presents a broad model of the principles and stages of participatory evaluation including an example of how a community project undertook a cycle of participatory evaluation. It also discusses some of the learning by SFLEX about what it takes to support community projects to do participatory evaluation.
This paper outlines a model of participatory evaluation that has developed out of the SFLEX team's work with community projects. We would also like to share some of our learning from this work about how to support participatory evaluation processes in community-based projects.
SFLEX was established in 2002 to provide support and training in action research to Stronger Families and Communities Strategy (SFCS) projects which are community-based early childhood intervention and community development projects. At the time writing, these 50 community projects were spread across Australia and were funded by the Australian Government Department of Family and Community Services (FaCS) as part of the first phase of the Stronger Families and Communities Strategy. SFLEX is based at the Australian Institute of Family Studies and funded by FaCS as part of the Strategy, now in its second phase.
A condition of funding was that projects undertake action research as part of their work. This was so they could develop in response to their local context and contribute evidence about what it takes to do quality early childhood intervention in Australia. It was also seen as a good method of doing evaluation in an ongoing and formative way in community contexts.
What is the difference between participatory evaluation and action research?
Action research and participatory evaluation are complementary and aligned processes which use many of the same methods. However, action research can be much more than simply a tool for evaluation. For this reason we use the term "participatory evaluation" to talk about the aspects of action research that focus on evaluation. This paper is drawn from a large resource entitled Communities doing Participatory Evaluation, which has been developed by SFLEX. We have also prepared another resource about action research more broadly, called Doing Action Research with Community Projects.
What is participatory evaluation?
Participatory evaluation is a process through which people involved at various levels of a project engage in ongoing evaluation of the project and its effects. The focus of participatory evaluation is to actively engage those who the project is for in all aspects of the evaluation process - sharing control of planning, undertaking, analysing and applying learning from an evaluation process.
Participatory evaluation has a number of aims. The first and most important is to develop and improve a project through applying emerging learnings about the specific context of that project and in the process build skills and knowledge that empower sustainable action in the future. It further aims to communicate with external and internal stakeholders, and provides a method of accountability for the project.
Some of the key principles of participatory evaluation are as follows.
- It ensures the involvement of active participants, not just sources of information.
- It builds the capacity of local people to gather information, analyse, reflect and take effective action.
- It supports the joint learning of people involved in a project, including those who are involved at different levels and in different ways.
- It acts as a catalyst to help people commit to taking more effective action in a project or community context.
- Conventional evaluation and participatory evaluation differ in many ways. Some of the key points of difference are outlined in the following table (see Table 1).
| Conventional | Participatory | |
|---|---|---|
| Who | External experts. | Community members, project staff, facilitator. |
| What | Predetermined indicators of success, principally cost and production outputs. | People identify own indicators of success, which may include production outputs. |
| How | Focus on "scientific objectivity", distancing of evaluators from other participants; uniform, complex procedures; delayed, limited access to results. | Self evaluation; simple methods adapted to local culture; open, immediate sharing of results through local involvement in evaluation process. |
| When | Usually upon completion of project or programme; sometimes also mid-term. | More frequent, small-scale evaluations. |
| Why | Accountability, usually summative, to determine if funding continues. | Empower local people to initiate, control and take corrective action. |
| Source: Adapted from Estrella & Gaventa (1999) | ||
At the Stronger Families Learning Exchange we feel that participatory evaluation is an important strategy for working in and with communities to support social and family change. We have found that participatory evaluation allows evaluation processes to contribute directly to the effectiveness of a project and develops local skills and knowledge in the process. It is a good way to acknowledge that members of the community are the expert in their own lives and culture and are therefore best placed to explore their own realities, identify their own assets and define their own aspirations. We have found that participatory evaluation can develop rich pictures of the real world contexts in which community projects are working, and this knowledge can help to guide future practice and policy.
The six stages of participatory evaluation
The Stronger Families Learning Exchange have been working with over 46 different projects around Australia and we have found that participatory evaluation looks different in every context. There is no one way to do participatory evaluation.
However, we have observed that projects generally move through a set of six stages that make up a cycle of participatory evaluation.
The stages usually overlap and recur because, in practice, participatory evaluation is a fluid process that is embedded in the implementation of a project. Although it can be used to do overall project evaluation or evaluate specific aspects or activities within a broader project, we have observed that the process ideally passes through each of the six stages in one way or another.
An example of the six stages - Jalaris
An example of a cycle of participatory evaluation can be seen in the project with which I have been working in Derby, Western Australia. The Building Strong and Healthy Families in Derby project is run by the Jalaris Aboriginal Corporation and provides a range of support and community development activities to their local Aboriginal community. They also act as a resource centre and liaison point between the Aboriginal community and other groups in Derby. Late last year they completed a cycle of participatory evaluation with the support of SFLEX.
- Scoping their evaluation: Scoping an evaluation involves establishing what the project is aiming to do, how change might happen, what resources are available, what the principles underlying the project are, and any beginning stories that can be documented. In the case of the project in Jalaris, this was a slow process of clarifying the key themes that the project was working on by reflecting on activities they had done during the many years they had been living and working in Derby. The themes they decided to address were: providing support to their community in areas such as child nutrition; school attendance; parenting; substance abuse and family violence; poverty and budgeting; developing self esteem and the communities ability to identify and respond to their own issues.
- Focusing the evaluation: Jalaris focused the evaluation of the project on looking at which strategies were most effective, and what different sectors of the Derby community thought about the work they had done so far. It was decided to invite participation in evaluation from local Aboriginal families, employees of Jalaris, associated agencies and service providers and members of the local business community about what was working well and what else they would like to see as part of the project.
- Gathering information in Derby: Developing effective ways to gather information from these different groups of participants was quite a challenge. In the end a number of different mechanisms were used including: informal reflection sessions and conversations with project participants; semi-structured family surveys delivered by the family support workers with local aboriginal families; interviews with local businesses and agencies conducted by staff from SFLEX, and analysis of daily logs kept by the project coordinator. In participatory evaluation, the way that information is gathered, by whom and in what form is important as different methods can increase and decrease levels of participation. In this example, a variety of different methods were used in order to make sure that people felt comfortable and free to participate.
- Making meaning from information: Once some information has been gathered the process of making meaning begins. Doing this process in a participatory manner is a particular challenge, especially to those of us who are professional researchers. In our example, the process of making meaning happened in quite a fluid way. After each round of interviews for the family survey, the Jalaris workers would come back and debrief together. These debriefing sessions, including the initial analysis that happened as part of the conversation, would be documented. SFLEX worked with the Jalaris team reflecting on the information that had been gathered by collaboratively reading notes from interviews, daily logs and reflection sessions and having many conversations and discussions as part of the daily work of the project. Through these processes, key themes were drawn out and over time agreed meanings emerged.
- Communication: In conventional evaluation, communication is often the last stage of an evaluation process, but in participatory evaluation communication is an integral step towards making evaluation relevant and useful to the ongoing project. In the case of Jalaris' recent evaluation cycle, methods of communication included a draft report, creating a brochure with many pictures and quotes from the evaluation, and informal conversations with project participants about ideas that emerged from the evaluation. The final evaluation report and the brochure were consciously written in a language and format that were easily accessible to all the groups involved in the process so they could understand the findings and what they meant for future directions of a project.
- Applying learning: Applying the learning from the evaluation by putting into action new ideas or making subtle changes to existing strategies is the primary aim of participatory evaluation. This stage is both the end of a cycle, as well as the beginning of another cycle. For Jalaris, one of the key benefits of doing a cycle of evaluation was confirming support for the direction of their project from the local community. Out of the evaluation process, Jalaris realised there was much interest from families in a mobile health and family outreach caravan. With the evidence they gathered about people's wants and needs, the project has re-focused its resources on developing the mobile caravan, and renewed its efforts to gain further funding to support this aspect of their work.
Jalaris has now started planning a new cycle of evaluation on the development and implementation of the mobile caravan, looking at how this new strategy will achieve the projects original objectives.
Why focus on participation?
Projects usually have a range of people involved in them at different levels. These include: those who the project is for and the community in which they live; the project management team; the auspice agency; and the funding body. These people often have differing skills, interests and, perhaps most importantly, power.
In the Building Strong and Healthy Families in Derby project the process of evaluation has given families an opportunity to directly shape and contribute to the ongoing design and implementation of the project. It has also developed participants' skills in communicating about local issues in a way that can lead to tangible change. Families interviewed as part of the process commented that nobody had ever asked their opinion about local services before. In this way, participatory evaluation is a capacity building exercise in itself. In the context of Stronger Families Fund, participatory evaluation processes have become key strategies in implementing projects rather than simply being an added extra to document the project or meet accountability requirements.
The amount and quality of participation obviously varies throughout the life cycle of a project. Participation can be viewed as a continuum of intensity - from minimal involvement by key stakeholders to increased control and ownership by those who the project is for.
This is represented in the accompanying diagram (Figure 1).
Figure 1.
Source: Adapted from Rietbergen-McCracken & Narayan 1998.
In the case of Jalaris, levels of participation clearly increased as the process went on. Early on in the evaluation process, participation was focused on sharing information about the project and what it was doing. With the support of SFLEX, the evaluation moved towards consultation with people exchanging ideas and experiences about the issues the project is working on. In the end the evaluation process led to collaboration between different groups in establishing the new caravan project. As this cycle of evaluation did not quite reach the full handover of control to those who are involved in the project, we plan for the next round to be run with less outside support.
Lessons learned about supporting and facilitating this work
Through the Stronger Families Learning Exchange's ongoing work with the 50 Stronger Families and Communities Strategy projects we have learned a lot about supporting participatory evaluation processes. There are some key lessons that may be of use to anyone interested in undertaking or supporting participatory evaluation in the future.
Be flexible, adaptable and open to change
There is no one way to undertake participatory evaluation because it evolves within the community and context in which it is undertaken. Each community, whilst sharing common characteristics, is unique in terms of its culture, dynamics, politics, resources and social capital.
Aim to build capacity
Perhaps the most challenging part of participatory evaluation is being participatory. This means building relationships and committing time and resources to developing capacities and a learning environment where the skills and knowledge of all involved are acknowledged and put to use. The following are some things we have found useful.
- Building skills in reflective practice.
- Recognising that gaining widespread participation can be a very slow process as many strategies don't always work the first time.
- Providing training and support about participatory evaluation and participatory action research to as many people involved in a project as possible leads to better learning and improved projects.
- For the evaluation process to lead to changes and modifications in a project, there is sometimes a need to build capacities and knowledge about participatory processes within support organisations, such as auspice organisations and funding bodies.
Investing time and energy early on pays off later
We have found that investing time and energy in creating a learning environment early on in a project, lays good foundations for strong relationships and resilient projects. Although our support work with the Stronger Families Learning Exchange has proved to be useful throughout the lifespan of the projects we work with, where we have been able to provide early support and training to projects, they have gone on to integrate participatory evaluation processes in a deep way into their work.
We have found that repeating and reinforcing information about action research and participatory evaluation helps projects to gain the confidence to try strategies that might seem a little daunting at first. This includes planting many seeds and ideas. Different seeds will germinate in different contexts.
We have also found that training is not enough. Projects sometimes need intensive support to develop and maintain a learning culture. They also need support to quarantine time for reflection as part of their core work, which is not always easy.
Model creative and innovative approaches
Some of our greatest successes in supporting participatory evaluation have come from modelling creative approaches to action research and evaluation. Our examples have helped projects to gain confidence in doing new things themselves. For example, it is often hard for projects to get into the swing of documenting their work, which is a key part of any research and evaluation. At SFLEX we have given as much support as possible to projects around developing and trying out appropriate methods of documentation. We have found it is important to help people to value informal stories and anecdotes of change as well as working with them to develop more structured processes for documenting their work.
Streamline data collection and accountability requirements
SFLEX does not have a direct role in the monitoring and accountability of projects. However, we have found that it is important that funders' accountability and reporting requirements work as much as possible in concert with participatory evaluation processes.
Having complicated reporting and data collection requirements can limit the capacity of a project to do evaluation that focuses on improving the project. Sometimes information that would be very useful for improving practice is not gathered because information needed for reports to funding bodies necessarily takes precedence.
Engaging in negotiation with funding bodies early in a project about the information that is needed for accountability purposes is important and can help smooth the way to better participatory evaluation. SFLEX has at times been a useful neutral party able to broker such negotiations.
Expect and support change
Participatory evaluation and action research are geared toward causing change. Because the primary aim of this approach is to improve a project, the extent and implications of change that can result must be understood and anticipated. In our experience, participatory evaluation can spark change at the level of a project, auspice organisation, the wider community and for funding bodies.
Participatory evaluation can be an integral part of the development and implementation of local solutions to local issues. Whilst the methods used may be different in every community, the value of participatory evaluation, in terms of improved local knowledge and deeper ownership and participation by local people, is worth the effort it takes to find new ways to work in each new context.References
Estrella, M. & Gaventa, J. (1999), Who counts reality? Participatory monitoring and evaluation: A literature review, IDS Working paper 770.
Rietbergen-McCracken, J. & Narayan, D. (1998), Participation and social assessment: Tools and techniques, The World Bank, Washington, DC.
Maya Haviland wrote this article while she was a Senior Research Officer with the Stronger Families Learning Exchange at the Australian Institute of Family Studies. The paper was jointly presented by Maya Haviland and Richard Munt at the International Association of Public Participation (IPA2) Conference in Brisbane in May 2004. It has been adapted for publication in this edition of the Bulletin.
