Evaluating to Increase Impact

Evaluation appeals to my intense curiosity to figure out how and why things work and don’t work. Many high performing organizations that I have come to know generally use a combination of formal and informal evaluation to build learning loops towards increasing strategic success. That is, they hire experts to increase internal knowledge and staff capabilities and to work with them on particularly complex inquiries, where it is critical to understand the elements on which a program will fail or succeed. And, they look to their staffs to structure meaningful learning opportunities throughout the life cycle of programs. The story that I share today is about how one organization followed their curiosity and learned not only from the evaluation data but from the process itself.

Meet Community MusicWorks (CMW) in Providence, RI. This group of professional musicians is working to “build and transform” the urban community in which they live. They are collaborative, inquisitive and tenacious. In their quest to do the best possible work affecting the most lives, Community MusicWorks wanted to know if a key element of their core education program was having the “transformational” effect they hoped for on the lives of the young people that it touched.

Timing and Type of Evaluation
The program had just passed its tenth anniversary and CMW felt that it was both necessary and timely to measure impact. CMW wanted to know the children’s perspectives on who they were in the context of their world – did they have a motivated, curious and engaged inner life, and did they manifest qualities of persistence, musicianship, personal agency and participation in a wider world? – and the degree to which the CMW program had influenced this perspective. They were not sure how best to find the answer, and they recognized the importance of investing resources (money and people) given critical position of the program. So, they looked to long-time professional evaluator and educator Dennie Palmer Wolf for help. Her philosophy was that evaluation should generate capacity in a community; it should be dynamic, timely and not simply research based. This aligned with CMW values and expectations.

Evaluation Structure
A participatory evaluation was built using new methods to acquire data – most importantly, creating conversations between teacher and student where both learned. The data sets were analyzed, looking at language for patterns and phrasing to judge impact.  Conversational interviews were also conducted between CMW board members and parents of the students. These built relationships that had both an immediate effect of deepening the board’s understanding of the value of the organization’s work and a longer term effect of creating ongoing communication between the parents and the board members.

Increased Impact
The benefits to the community increased when CMW used what they learned through this evaluation to change the program design. They increased practicing, made progress markers clearer, encouraged greater family participation and created outside performing opportunities, all of which created more impact that was easily observable.

Initially, staff saw the evaluation process as being very time consuming in an already busy schedule and resisted it. As they moved through the process, they experienced the benefits and incorporated some of the methods for measurement and learning. They currently evaluate in six month intervals because of the insights gained.

The education director has become evaluation champion, having had formal training and having learned from the 2009 evaluation.  She now “consults” with other staff on design and whether or not to bring in outside expertise. For example, CMW is getting ready to evaluate their fellowship program and she will work with them on the evaluation design.

Elements of Success
How has Community MusicWorks made the evaluation work for them, and how this might be applicable to you?

• Clear line of inquiry – they knew what they sought to learn and how they would use the information.
  • Do you know what you don’t know? What is the most important thing to understand about how a program or service works or doesn’t? Can evaluation help you form and answer the question?
• Good timing – they had enough data to be meaningful and answer the questions at hand.
  • What is the quality of the data you have on hand?
• Appropriate process and priority for resources – they chose an outside evaluator for a more complicated inquiry who had a deep expertise and was geared for teaching and sharing in terms of knowledge and methodology, very compatible to CMW culture and values.
  • What’s right for your inquiry? Do you need outside expertise – to advise on what form the evaluation might take? To conduct the evaluation? Will you manage the evaluation process internally? What do you gain/lose with these approaches – expert advisor, outside expert for entire evaluation, internal evaluation?
• Significant involvement – they included important stakeholders in the inquiry (students, parents and the CMW staff and board) – so that learning would be comprehensive, cohesive and the processes learned would be integrated into the organization.
  • Evaluation can overwhelm a staff member and/or organizational team. The process needs to take into account who will drive the evaluation, who will need to be involved, and whether the workflow can be adapted to enable participation at the necessary levels and in a timely fashion.

Learn more and download Community MusicWorks’ final evaluation report here.