Using video to evaluate impacts and communicate results.
stakeholder including Visual Participatory Evaluation (VPE) is a robust qualitative evaluation tool that enhances existing methodologies of program evaluation through the use of video. It utilizes the increasing availability and decreasing cost of video to provide a powerful new medium for the purposes of gathering and sharing evidence.
The method in brief:
Step 1. Undertake stakeholder mapping – only include those individuals who have been in the position to observe change. (We are not interested in opinions but in direct observations of impact).
Step 2. Interview a representative sample of participants and other stakeholders on DVD. (eg 20 respondents). Select a location of the interviewee’s choosing and include observational footage (eg of their home, or an activity important to the interviewee.)
Step 3. Transcribe the interviews in full.
Step 4. Edit the interviews down to key observations or experiences supported by evidence of change. (eg 20 interviews of 5 to 10 minutes each).
Step 5. Constitute a representative panel of stakeholders including funders, program directors, decision-makers, subject matter experts, participants.
Step 6. Facilitate a panel that views and debates each interview one by one, making reference also to the full interview transcripts for each. Allow a full day to view and discuss.
Step 7. Summarise the findings and observations of the panel into a written report.
Step 8. Use the footage to make a visual report on the outcomes of the program/project/change being evaluated.
VPE collects qualitative data through semi-structured interviews with project participants and other stakeholders such as service providers, families of participants, community leaders, partnering organisations. The key question asked is “What changes have you experienced/observed?”. Respondents are invited to nominate whatever change they have observed that they consider to be important, but they are challenged to provide their evidence for that observation. Hence “You say that people are happier since the new road was built. How do you know they are happier?” Respondents are asked to provide the concrete reason they have reached the conclusions they have. Other questions might include "what do you most value about this service?"
Building on the existing methodology of Most Significant Change (MSC), the interviews encourage participants to simply share their ‘stories’ of change. Depending upon the nature of their stake in the project (eg participant, service provider, partner, a witness to the impacted community), stories will differ and a range of perspectives on the perceptions and experiences of change is provided. Interviewees are then also asked “why” they think those changes have occurred, which is an important part of establishing causality. Participants may also be asked to share their concerns and ideas, the challenges they experience in participating in the program, and their aspirations for how program participants might benefit from the project and change their lives and those of their families and communities. The most important element of the method is that interveiwees (informants) are not specifically asked their opinion of the project and its impacts - rather they are asked to share the changes and impacts they have observed.
The ideal informant interviewed is one who was present in the context prior to the commencement of the program, during and after. For example, a nurse at a hospital might be ideally placed to observe the outcomes and impacts of a diabetes management program even though he is not a participant in the program. This is especially so if the nurse knew and had significant daily interaction with the program participants before, during and after the program.
There are cultural, physiological, and data quality reasons for incorporating visual images with dialogical based evaluation methodologies to capture and understand the changes experienced by project participants and other stakeholders. In particular where English is a second language, or verbal communication is challenging, visual images are important in providing a voice words that might otherwise be lost. Importantly, images provide rich visual information about the social and environmental context within which the participant lives which communicate more effectively than any written word.
VPE is ‘participatory’ at two levels. First it allows the informant to deliver their experiences directly to the camera. The informant has an increased measure of control over the messages conveyed, in particular in emphasis. VPE is also participatory at another level, in that a panel of stakeholders, in particular decision-makers themselves, is constituted to view and analyse the interview content and so contribute to the evaluation in a more direct way.
Once stories and images are collected, the footage is edited to discard less pertinent content and then provided as a DVD and used to facilitate workshops where a stakeholder audience, consisting primarily of funders and decision-makers, but including stakeholders to provide specialist perspectives/interpretations eg Indigenous representatives) actively engages with the filmed interviews to debate themes, learnings, meaning and significance. The Panel that views the interviews is also provided with a full hard copy transcript of the interviews, so that they can see for themselves if anything important has been edited out. This transparency is important to the validity of the method.
The VPE process engages the participant, evaluator and the project implementer / funder across sensory levels. There is a physiological argument for using images in sociological research in that the parts of the brain the that process visual information are evolutionarily older than the parts that process verbal information. Thus images evoke deeper elements of human consciousness than do words; exchanges based on words alone utilize less of the brain’s capacity than do exchanges in which the brain is processing images as well as words (Harper 2001: 13).
We have found that the combination of the use of visual images, participant control over the data collection and interpretation, collaboration between participant and researcher, and reflection of images and stories by the project funder / implementer have identified findings closely aligned with the lived experience of participants and revealed themes of change from both images and dialogue. However, the main advantage of VPE over other qualitative methodologies is the way in which it enables management and decision-makers to hear stories for themselves, directly from the stakeholders, rather than to have these mediated via a consultant report.
A written report is compiled after the panel has viewed the interviews containing the conclusions drawn by the panel about the impacts and critical success factors of the program being evaluated.
In all instances where we have used the VPE method, the client has asked us to use the footage to communicate the findings of the evaluation (a visual reporting) and to create an advocacy tool for the program.
There are a number of challenges to be considered regarding the utility of VPE: some natural talent for filmography, or the need for well developed film experience, the technological requirements for the filming and editing and the time required to edit the footage must be included in evaluation scoping. Widespread availability of smart phones and ipads, as well as simple, free editing programs, have improved accessibility and overcome some of the technical challenges and therefore the costs.
The VPE method was developed over a long period of trial and error, and has now developed into a mature methodology for the collection, production, analysis and communication of visual aspects and insights in evaluating social programs, change events and community change.
The method in brief:
Step 1. Undertake stakeholder mapping – only include those individuals who have been in the position to observe change. (We are not interested in opinions but in direct observations of impact).
Step 2. Interview a representative sample of participants and other stakeholders on DVD. (eg 20 respondents). Select a location of the interviewee’s choosing and include observational footage (eg of their home, or an activity important to the interviewee.)
Step 3. Transcribe the interviews in full.
Step 4. Edit the interviews down to key observations or experiences supported by evidence of change. (eg 20 interviews of 5 to 10 minutes each).
Step 5. Constitute a representative panel of stakeholders including funders, program directors, decision-makers, subject matter experts, participants.
Step 6. Facilitate a panel that views and debates each interview one by one, making reference also to the full interview transcripts for each. Allow a full day to view and discuss.
Step 7. Summarise the findings and observations of the panel into a written report.
Step 8. Use the footage to make a visual report on the outcomes of the program/project/change being evaluated.
VPE collects qualitative data through semi-structured interviews with project participants and other stakeholders such as service providers, families of participants, community leaders, partnering organisations. The key question asked is “What changes have you experienced/observed?”. Respondents are invited to nominate whatever change they have observed that they consider to be important, but they are challenged to provide their evidence for that observation. Hence “You say that people are happier since the new road was built. How do you know they are happier?” Respondents are asked to provide the concrete reason they have reached the conclusions they have. Other questions might include "what do you most value about this service?"
Building on the existing methodology of Most Significant Change (MSC), the interviews encourage participants to simply share their ‘stories’ of change. Depending upon the nature of their stake in the project (eg participant, service provider, partner, a witness to the impacted community), stories will differ and a range of perspectives on the perceptions and experiences of change is provided. Interviewees are then also asked “why” they think those changes have occurred, which is an important part of establishing causality. Participants may also be asked to share their concerns and ideas, the challenges they experience in participating in the program, and their aspirations for how program participants might benefit from the project and change their lives and those of their families and communities. The most important element of the method is that interveiwees (informants) are not specifically asked their opinion of the project and its impacts - rather they are asked to share the changes and impacts they have observed.
The ideal informant interviewed is one who was present in the context prior to the commencement of the program, during and after. For example, a nurse at a hospital might be ideally placed to observe the outcomes and impacts of a diabetes management program even though he is not a participant in the program. This is especially so if the nurse knew and had significant daily interaction with the program participants before, during and after the program.
There are cultural, physiological, and data quality reasons for incorporating visual images with dialogical based evaluation methodologies to capture and understand the changes experienced by project participants and other stakeholders. In particular where English is a second language, or verbal communication is challenging, visual images are important in providing a voice words that might otherwise be lost. Importantly, images provide rich visual information about the social and environmental context within which the participant lives which communicate more effectively than any written word.
VPE is ‘participatory’ at two levels. First it allows the informant to deliver their experiences directly to the camera. The informant has an increased measure of control over the messages conveyed, in particular in emphasis. VPE is also participatory at another level, in that a panel of stakeholders, in particular decision-makers themselves, is constituted to view and analyse the interview content and so contribute to the evaluation in a more direct way.
Once stories and images are collected, the footage is edited to discard less pertinent content and then provided as a DVD and used to facilitate workshops where a stakeholder audience, consisting primarily of funders and decision-makers, but including stakeholders to provide specialist perspectives/interpretations eg Indigenous representatives) actively engages with the filmed interviews to debate themes, learnings, meaning and significance. The Panel that views the interviews is also provided with a full hard copy transcript of the interviews, so that they can see for themselves if anything important has been edited out. This transparency is important to the validity of the method.
The VPE process engages the participant, evaluator and the project implementer / funder across sensory levels. There is a physiological argument for using images in sociological research in that the parts of the brain the that process visual information are evolutionarily older than the parts that process verbal information. Thus images evoke deeper elements of human consciousness than do words; exchanges based on words alone utilize less of the brain’s capacity than do exchanges in which the brain is processing images as well as words (Harper 2001: 13).
We have found that the combination of the use of visual images, participant control over the data collection and interpretation, collaboration between participant and researcher, and reflection of images and stories by the project funder / implementer have identified findings closely aligned with the lived experience of participants and revealed themes of change from both images and dialogue. However, the main advantage of VPE over other qualitative methodologies is the way in which it enables management and decision-makers to hear stories for themselves, directly from the stakeholders, rather than to have these mediated via a consultant report.
A written report is compiled after the panel has viewed the interviews containing the conclusions drawn by the panel about the impacts and critical success factors of the program being evaluated.
In all instances where we have used the VPE method, the client has asked us to use the footage to communicate the findings of the evaluation (a visual reporting) and to create an advocacy tool for the program.
There are a number of challenges to be considered regarding the utility of VPE: some natural talent for filmography, or the need for well developed film experience, the technological requirements for the filming and editing and the time required to edit the footage must be included in evaluation scoping. Widespread availability of smart phones and ipads, as well as simple, free editing programs, have improved accessibility and overcome some of the technical challenges and therefore the costs.
The VPE method was developed over a long period of trial and error, and has now developed into a mature methodology for the collection, production, analysis and communication of visual aspects and insights in evaluating social programs, change events and community change.
