This post makes me just a little sad to write. Chorus America, a while ago, published the results of a study, which they say shows that people who sing in choruses are exceptionally good citizens. They then say that choruses should bring this information to the media, “to help establish an awareness of the personal and communal benefits of choral singing.” Here’s their press release about the study, and here’s the study itself. (The quote comes from the end of the study.)
So why am I sad to talk about this? Because the study suffers from an elementary misuse of statistics. But it’s so eager, so hopeful, and so well-intentioned that it’s almost painful to tell the poor choral people that their work is flawed. I feel like I’m telling a lovely child that I can’t take her to the zoo today.
The claims the study makes are very strong. Here are excerpts from the press release:
If you enjoy singing with your neighbors, congregation, or classmates, you’re taking an increasingly popular path to a successful life….Seventy-eight percent of choral singers indicated they “at least sometimes” volunteer their time in their community, while only 50% of the general public say the same….Choral singers donate 2.5 times more money to philanthropic organizations than the general public….Ninety-six percent of choral singers surveyed who are eligible voters said they vote regularly in national and local elections; only 70% of the general public cites the same level of participation.
Choral singers, in other words, are better citizens.
What’s wrong with this? Well, first, if you find that choral singers participate more in civic activities, you’ve loaded the dice, because simply by singing in a chorus, they’re participating in something. Your sample, in other words, is — by its very nature — a sample of people who already participate in at least one civic activity. It might not be remarkable to find that they participate in other things as well. (If I told you that that people who buy season tickets to the NFL like sports more than their neighbors do, would you be surprised?)
But that’s only the start of the trouble. The biggest problem is the control group (so to speak) for the study, the people whom choral singers are compared to. It’s the general public! Or in other words, the entire population. Choral singers vote more often than the general public does? They give more to charity? They volunteer more? Well, so what? Suppose I told you that choral singers commit fewer murders than a random sample from the population at large. You wouldn’t be surprised. You might say, “But of course that’s true, because we’re pretty sure that, by and large, the kind of people who commit murder don’t sing in choruses.”
So maybe the truth about voting, volunteering, and charitable giving is similar. Maybe the facts cited in the study aren’t really facts about choral singers, but instead facts about the demographic that choral singers come from.
Maybe these aren’t facts about choral singers, but instead facts about the demographic that choral singers come from. Maybe the kind of people who sing in choruses — measured by standard demographic traits, like income, education, or occupation — are already the kind of people who vote, volunteer, and give to charity. Or maybe, to look at this a different way, people who participate in any community activity are more likely to vote and give to charity. Which would mean choral singers weren’t exceptional, because they were behaving exactly the way any people who get out of the house to do anything in their community would behave.
What the study should have done is so elementary that — to sing this song again — it makes me wince to set it forth. The study should first have found out what kind of people sing in choruses, what their other demographic traits might be, and how, as a group, people from that demographic fare in all the measures that we might want to apply to choral groups.
Then theh study should have looked at people who participate in other community activities, and find out their characteristics. Only then should it have looked at choral singers, and found out if they — as a distinct group within their demographic, and as compared to others from that demographic who take part in community activities — have any special traits.
As I said, all this is elementary. The conclusions of the Chorus America study might in fact be true. Maybe choral singers really are exceptional citizens. But Chorus America hasn’t even come close to proving that.
(I ascribe this lack of skills and judgment, by the way, to the same eagerness about the arts that makes arts advocates sometimes think uncritically in other ways. See, for instance, my post about the arts bailout, my Wall Street Journal essay on the same subject, and my post about the none too grounded proposals that came out of last summer’s mammoth arts conference in Denver.)
You’re right on, Greg. Unfortunately, misuse of data helps make everyone skeptical of all data, even when it is presented more or less correctly. And we wonder why nearly half of the country thinks the earth is only 6,000 years old and that Ben Franklin invented electricity. The study did inspire me to make my own completely unsupported assertion that whiskey drinkers make good citizens, too, but only those drinking Irish whiskey… 🙂
Cheers.
Interesting post. I propose to introduce some strictly anecdotal evidence. If you get a chance, check out this short video — ahamoment.com/pg/moments/view/4649 — it doesn’t get at the quality of the person singing, but rather the effect of the music. It’s one woman’s “aha moment” when she realized what singing in a chorus meant to her. I think you’ll enjoy it.
Thanks,
jack@ahamoment.com
“Maybe choral singers really are exceptional citizens. But Chorus America hasn’t even come close to proving that.”
It’s important to distinguish between the following statements:
1. Choral singers are exceptional citizens.
2. What makes choral singers exceptional citizens is the fact that they sing in a chorus – i.e., that they’re “taking an increasingly popular path to a successful life,” as the Chorus America article says.
#1 is arguably proven by the study (I am still skeptical, but I’ll grant some amount of proof). #2 definitely isn’t.
Downer post, Greg! j/k
I am glad that you’re on top of these things and are able to approach these issues with a critical frame of mind. I took basic statistical research methods as part of my undergrad, and even with this basic stats background I could tell these results were concluded upon a faulty design. You would think a national organization such as Chorus America would have a little better grasp of statistical methods than your average undergrad.. Here’s hoping..
But you’re right…it is hard to disregard their findings because their intentions are so.. well intended! Thank you for being the voice of reason in this matter and shedding some unbiased light on the matter.
Ah, yes, the old “correlation does not imply causation” rule. Seems so simple, yet is so misunderstood. Michael is right. This kind of thing leads the general public to think that statistics are unscientific crap that can mean “whatever you want it to mean.” This is very sad, given the amazing power of statistics to explain things when used correctly.
I was principal analyst for Chorus America’s report and want to respond to Greg’s post. First of all, Greg raises a concern that is always valid. Anytime you do a research study about a psychographic such as choral singing, you have to be concerned that it’s a proxy for a demographic category. In our experience, when differences are this striking, it almost never turns out that they can be fully explained by demographics alone. But even when demographics do not explain the results, you still can’t necessarily claim causation merely by asking questions of the type we did because the psychographic may be self-selecting for people who tend to be similar in other ways. For these reasons, we did not claim choir participation causes people to vote, become more charitable, or volunteer more often–in fact, we were explicit about this on the first page of the report, when we said: “While this research does not prove that choruses cause singers to gain attributes that are characteristic of success, the data–especially from parents and teachers surveyed–make the connections overwhelmingly strong.” And in our view, the existence of these connections is worth noting even if no causal relationship is proven. For example, if choristers are more civic-minded across a wide range of activities than other Americans, there are clear potential implications for government and corporate decision-makers about choirs as potential constituencies. Of course, the civic engagement of chorus members may seem like stating the obvious to those in the know, but clearly it’s not obvious to everyone who needs to know it (it may not even be obvious to choristers themselves, the vast majority of whom do not regard their singing primarily as a form of civic duty), which is why these findings were considered newsworthy by many. At the end of the day, the point we were making with these results was simply, quoting again from page 1 of the report itself: “If you’re searching for a group of talented, engaged, and generous community members, you would do well to start with a chorus.”
In any event, for us the data on volunteering, charity, and voting was just follow-up on work done by another research group for Chorus America in 2003 that came to the same conclusions we did. The bulk of the current report was actually about the impact of choir participation on children, and in this area concerns about potential demographic proxies were more directly addressed. We found, for example, that among children there was no correlation between choir singing and traditional defining demographic categories like income or ethnicity, yet we found a myriad of positive differences between children who are in choirs and those who are not. To deal with the self-selection biases that could still explain these results, we asked parents whose children sing whether they attributed and dated each of these possible positive changes in their children to when they started singing in choirs, and got resoundingly positive responses. We then asked educators (the vast majority of them not arts educators, and therefore, in the current zero-sum economic environment, not necessarily motivated to be positive about choirs) about these areas (and more) and got even more overwhelmingly positive responses. The benefits cited by both parents and educators were, in fact, so broad and, in many cases, choir-specific, that they cannot be dismissed as simply a proxy for extracurricular participation of any kind. But even with all of this evidence suggesting causal linkage, we still did not claim causation in the report.
One can, of course, decide to discount the observations of large percentages of parents about their children, or large percentages of educators about their students, but it is not obvious to us that scientific rigor requires one to do so; at most it argues–as nearly all research does–for further research using the same or different methodologies that either affirms or calls into question the original claims. In the meantime, the results that are available are published so that appropriate action can be taken, rather than waiting–and causing children to wait–for definitive results that, given the nature of most academic research, are not likely to be considered definitive any time soon.
Tom de Boor
Grunwald Associates
This is a great example of someone shooting off their mouth getting blasted. I’m so glad I read it!
First the OP looks down on the “poor choral” people, patting their head and “feeling bad” about how he’s about to destroy their illusions. So high above them is he and his knowledge of statistics! And in his beneficence (even though it hurts him to do so, he continues in the name of truth — how brave!) he kindly steps off of his high plateau and is gracious enough to bestow upon them the light of his knowledge (I go a bit overboard, but traces of this mentality are practically bleeding through the characters).
Then, enter a well articulated response from someone in the trenches – a person who doesn’t seem to need the OP’s grace whatsoever, who seems cognizant of the implications of his study and it’s limitations, and who is genuinely kind enough (or understanding enough) to not point out the embarrassment that the original post is. He’s too kind, in fact, which is why I’m typing now.
Some of us are lucky enough to get a general eduction. Shooting off our opinions about something we learned in “basic statistical research methods”, as one person put it, and thinking that that elementary knowledge gives us the ability to somehow debunk professionals working and applying methods in the real world (not from an armchair) is ungracious and arrogant. And, as an American, I find this one symptom of an attitude that gives us the international reputation of being loud obnoxious asses, which is embarrassing and unfortunately somewhat true.
I realize the OP has previous experience in statistical analysis, but he nevertheless comes off with the same attitude, which is perhaps even worse (or at best, less forgivable).
People who supposedly know better (professionals, professors, scholars, whatever) can be thwarted by anyone (even a child wanting to go to the zoo). But this thwarting usually comes in the form of questions that shake assumptions, not the sort of half-cocked, shooting-from-the-hip, condescending ego-ut-deus ex machina I’ve witnessed here today.
Brilliant show, though.
Greg’s post inspired me to read the Chorus America report in detail. There are certainly a few sentences in the introduction and in the recommendation sections where the case is overstated and the authors slip into claims that participation in a choral group causes one to be a great citizen. Overall, however, the researchers do not succumb to what I would classify as “undergraduate level” errors in research design and statistical analysis, and while there are holes in the work, there is very little “magical thinking” presented.
The press release is another story, and is certainly overzealous in its claims, but the report just seems a bit muddy and incomplete. Much of what is described in this report about choristers and childhood participation begs the question, “Does participation in choral activities produce any of these effects, or do people with these characteristics simply select choral singing as an activity?” Perhaps the research design could have been planned to shed more light on this question. Two of the big questions I see left open are:
• How much of the difference in civic involvement between adult choristers and general population is explained by the higher than average level of church attendance for choristers? Other research has found that many of the same civic traits are predicted by church attendance, and the Chorus America report states that 67% of choristers regularly attend church. The findings could be confounding the two variables. A comparison of the answers of choristers who attend church regularly with those that do not could provide some insight here.
• How reliable are the findings that parents and teachers attribute children’s success to their choral participation? The report doesn’t provide information on the full methodology for the surveys, but a couple of biases seem plausible in this part of the work. One is that people are notoriously bad at attributing causation to other people’s behavior and successes: they tend to emphasize what they can observe and never consider possibilities that they cannot observe (for example, the role of family, church, or peers outside of school). Another potential bias is a predisposition among the parents and teachers to see choral participation as effective. The report states that there is a high level of parental participation / influence in many of the schools studied, and that many teachers recommend choral participation for children. People tend towards a “confirmation bias” that leads them to see confirming evidence for their beliefs, and we may be seeing that effect in the survey responses of these “choral advocates.”
Is the research perfect? I don’t think so, and I would be very interested in seeing further work on these and other questions. Even more, I’d love to see a place where all research published in our field gets discussed critically, additional questions about design and findings get raised, and people then go on to do further research that builds on the literature. This model works for other areas of social science research, and I think it would be great for our field.
I don’t find the same level of offense here that Greg finds. The results of research done in the support of advocacy do tend to get overstated but this report seems a very mild example of that particular problem.
Jim
Greg,
Thanks for your reply to my comments above. I do see your point about the comparison of choral singers to the general public. I took the comparison as simply demonstrating that choral singers are, in general, members of the “exceptional citizen” segment of our society. As you point out though, that might not be saying very much: there may be a lot of equivalently useful identifiers for citizens who fall in this segment.
And you ask me a fair question. Is the research and analysis questionable? I guess I read it as an intriguing first step and see it more as incomplete than anything else. I certainly learned some things about choristers by reading the report that I didn’t know before. But if I were being asked to invest in a program based on this research, I would say I need the answers to some additional questions before I’ll bet the farm on this.
Overall what I read in your posts is a call for a higher standard in the research conducted in our field, and a much more critical reading of the studies that land on our desks and in our email boxes. And to that I can only say, yes please.
Jim
Greg –
Though I realize in the blogosphere that “tone” is not supposed to be a concern, I was less alarmed by your armchair statistical analysis than disappointed by the condescending, patronizing tone you seem to take toward the choral world: “But it’s so eager, so hopeful, and so well-intentioned that it’s almost painful to tell the poor choral people that their work is flawed.” Puuleeze! Go ahead and level your criticisms – I think we’re tough enough to handle it.
More illustrative here is what your tone demonstrates yet again about the all too common and foolish dismissal of the choral world by arts commentators and orchestra administrators and conservatories. After all, the performers are mostly amateurs with sentimental, plebian tastes – with neither the professional skills of “musicians” (= instrumentalists) or the cutting-edge sophistication of the arts cognoscenti.
But the more significant conclusion worth probing is the study’s lead statement that “choral singing continues to be the most popular form of participation in the performing arts.” While those with more statistical training than I could no doubt argue with the professional statisticians who generated this report, a simple comparison with participation in orchestras (by professionals and amateurs combined) shows that the numbers are not even close:
42.6 million Americans participate regularly in choruses (CA – 2007-08)
(of whom 32.5 million are adults)
550,000 Americans participate regularly in orchestras (ASOL 2005-6)
Furthermore, the study claims that these choral singers attend an average of 2.5 orchestra concerts per year.
This should make anyone concerned about the continued vitality of “classical” music in America take notice of this core audience. In my experience, they are incredibly diverse in every way, intelligent, perceptive, open to a wide spectrum of musical style (much wider than what is available on instrumental concerts these days), and yes, they are engaged in their communities, in many ways.
And of course, I would venture (without statistics this time) that there is far more new music being written for choirs on all levels than for any other musical medium. For sure, much of it aspires to the “easy listening” of some of the more widely published popular composers, but there is a great deal of new vocal music out there that pushes the creative boundaries more, and still has a significant audience – mainstream composers like John Adams and Jennifer Higdon; Bang-on-a-can composers like David Lang, Phil Kline, and Toby Twining; take a look/listen to new music at the web site of Donald Nally’s The Crossing http://crossingchoir.com/ which has developed an avid following in conservative Philadelphia in a very short time. And Alan Harler recently led the 130-year-old Mendelssohn Club just premiered Lang’s Battle Hymns, with dancers, drummers, and horses in an armory – six performances sold out a week in advance. And choral repertoire not only stretches centuries earlier than the European orchestra and crosses cultural boundaries much more easily today in countries all around the globe.
But before I get carried away with suspect “boosterism” here, my main point is that it’s well past time for people in the field to take choruses more seriously when looking for keys to the future.
Cheers – Tom Lloyd – tlloyd@haverford.edu
Is it true? Was Professor Harold Hill right after all? Does music guide youngsters out of trouble and steer them toward the path of good citizenship?
Do the experiences of choral participation and other musical activities help create socially responsible adults? Does the Chorus America report provide evidence for the hypothesis? I believe that it does somewhat and that evidence could be gathered that would demonstrate that a large proportion of community leaders, medical professionals and other care givers were very active in musical performance as children whether or not they include avocational music as adults.
The unique mix of personal commitment to high standards of achievement and acceptance of responsibilities to their musical co-culture has often been credited by adults with laying the foundation for self-confidence and ultimate success.
Music is called upon across our culture to attend our passages with celebration or mourning. It goes to church with us, attends our weddings and holidays and shows up at our funerals. How can the providers of this music avoid being ennobled by their social roles?
It’s entirely possible that at least some of the differences between adult choir singers and the general public are caused by a confounding psychographic, namely church attendance, just as it’s possible that some portion of the differences could be caused, to a certain extent, by the fact that choir singers (or at least our sample of them) is more white, higher income, and better educated than average. It just seems unlikely that this can fully explain the differences. For example, 96% of choristers told us that they vote regularly in elections; it seems unlikely to us that 96% of white, well-educated, high income members of the general public, or even 96% of this demographic who also regularly goes to church would say the same thing. And even if they did, a substantial portion of our choir sample was non-white, not well educated, not wealthy and/or not church going, so these demographic numbers would likely have to exceed 96% by a significant degree to be fully explanatory of the choir member results, which, of course, would be physically impossible. To take another example, related specifically to church-going: there actually is a significant difference in charitable giving between choir members who regularly attend church and those who don’t–church-going choir members give an average of $282/year to charities vs. only $237/year for non church-going choristers. But $237/year still far exceeds the $104/year given by the general public, even though our general public sample included both church and non-church going respondents. Similarly, while church-going choir members are significantly more likely to say they volunteer either “frequently” or “fairly often” than choir members who don’t go to church (44.5%-32.1%), non church-going members still say they volunteer with these frequencies significantly more than the general public (32%-24%), a difference that would likely be even greater if we really compared apples to apples (i.e. non-churchgoing choir members vs. non-churchgoing members of the general public–or church-going choristers vs. church-going general members of the general public for that matter).
Of course all of these examples assume that we were trying to prove causation in these areas–if we weren’t (and we weren’t), it’s not clear how important it is to separate out what parts of these results can be explained purely by the activity of singing in a chorus vs. demographic (or other psychographic) qualities that choral singers brought with them to the party. Related to this, Greg, let me try to address your question about what I mean by the importance of connections more concretely. Over the past ten years, both political and corporate marketing has become less and less mass media-based and more retail and targeted. Both politicians and corporate marketers are looking for ways to target groups that have high concentrations of individuals who are potentially particularly important to reach. The data on choristers and civic participation indicates that they are, to use classic Roper research parlance, collections of “influentials,” which both political and corporate types believe are important to gain the loyalty of. Of course there are many other ways to reach influentials individually, and in some cases, collectively, besides choirs, but the political and corporate worlds are both aware of this, are bombarding influentials with messages, and are therefore looking for activities and positions that will enable them to “cut through the noise.” Reading our report and data, perhaps an influential politician decides that taking a position against closing school choir programs (or providing state funding to support them) will help gain the loyalty of influentials via their affiliation with choirs, perhaps a corporation decides that making more grants to choir programs is a way to curry favor with this group. For a choral advocacy group like Chorus America, these are desirable outcomes worth trying to promote.
As far as the children’s data goes, what we know is that choir participation is not determined by the classic determining variables of race or social class. There is some correlation between parents education level and choir participation, but neither the difference in average education level (0.6 years of difference between parents of kids in choirs vs. those whose kids aren’t–both groups in our parent sample averaged better than a high school education) nor any specific education level can explain the wide variations we found between kids in choirs and those that aren’t. And in any case, the reality is that the relationship between parent education level and children’s academic achievement (and its extent) is still uncertain and debated in the literature.
Furthermore, as you know, we went beyond just looking at these differences to find out whether parents and educators ascribed these differerences (or qualities, in the case of educators) to choir participation or not. Could there be “confirmation bias” on the part of parents, educators, or both? Of course there could–frankly any number of historians and philosophers of science have argued the same problem can (and often does) exist even within collections/collectives of “hard-nosed” researchers (seeking, observing, and publishing only what they want to believe). But again, scientific rigor doesn’t require us to assume that every time a group of human beings says something is true it’s only because their saying it confirms some preexisting prejudice on the subject (and in fact, as previously indicated, at least in the case of educators, support for choirs was arguably against the self-interest of most of our sample). The fact that more than half of our educators have at one time recommended choir doesn’t mean by definition that their observations about choir are suspect, particularly not if–as is very likely, given what we know about educators–the decision to recommend singing was itself based on observation and/or other forms of evidence (it’s also worth noting in this vein that much greater majorities of teachers agreed with nearly every possible positive choir effect than said they had ever recommended it directly to students or parents).
In short, we aren’t required to believe that parents don’t actually know what is happening in their children’s lives, that teachers don’t know what is happening with their students, and that only “hard-nosed” scientific researchers doing double-blind peer-reviewed studies really know what’s really going on (if only they could come to agreement about it, a particularly notorious problem with academic educational research). What’s required, as is the case of all research, “hard-nosed” or not, is that additional research be done, either repeating the same metholodology or trying different approaches–to confirm or rebut the initial findings. As I’ve indicated before, we think it’s pretty tough to look at the totality of our results (especially those about children) and conclude they could be explained or produced by any other extracurricular activity, but it would be terrific if other researchers took up that challenge.
Tom de Boor
Grunwald Associates LLC
PS The percentage of educators reporting high parental influence in this study was actually generally lower than we’ve found in other educator surveys we’ve done.
I’m a chorister and on the board of my community choir. My choir is a member of Chorus America.
This study is a recent version of a study they’ve done before, and honestly, this type of study frustrates me. These questions and their answers may help educators promote choral singing for youth, but they do not help adult choirs.
Telling community leaders and businesses that your choir is full of civic-minded persons does not get you grants or sponsorships or donations. It doesn’t get you more audience members, either.
What choirs could really use is some solid research on increasing audiences.
Out of all the arts marketing research I’ve tried to track, there’s only one survey that includes as a specific question “Do you like choral/glee club music?” (Survey of Arts Participation in the U.S., 2002) And it doesn’t even ask “Have you attended a choral concert in the past 12 months?”
NEA’s 2002 Survey of Public Participation in the Arts
http://www.cpanda.org/cpanda/getDDIsummary.xq?studyID=a00080
The survey shows that about 9% of the general public says they like choral and glee club music. That’s way below classical music and even below opera. So it’s not easy to find people to come to concerts. Why Chorus America isn’t doing marketing research that would help choirs grow their audiences beats me. Bigger audiences help with grants, too.
Tom,
As the person who raised the questions about church attendance and the potential for biases in parental and teacher responses, I just wanted to thank you for sharing the additional detail on your findings. While you were not interested in understanding any causal relationships, I can’t help but be curious about those questions. The additional data you’ve shared only increases the value of the report — serving not only as the foundation for target marketing that you intended it to be, but also as a source of insights into the world in which we live. Thanks for taking the time.
Jim
Some specialists argue that business loans aid a lot of people to live their own way, just because they can feel free to buy needed stuff. Furthermore, a lot of banks present collateral loan for different persons.