NAEP Arts 2008, Part Three: The Chrome Standard?

NAEP is universally considered to be the “gold standard” for K-12 educational assessment.

My good friend and colleague Lynne Munson, Executive Director of Common Core, has posted a an entry on her blog that compares the NAEP Arts Assessment 2008 with the NAEP math and reading tests. Lynne, a former deputy director of the National Endowment for the Humanities, and author of Exhibitionism: Art in the Era of Intolerance, concludes that the arts assessment is a far cry from what we all know as the gold standard in reading and math.

It would be fair to say that it’s more like the chrome standard.

Here’s an excerpt:

How would you describe NAEP?  Here’s what comes to mind for me:  a
rigorous achievement test given regularly to a statistically
significant number of students in a large number of schools in every
state.  That description certainly fits NAEP’s reading and math
assessments.  In 2007 NAEP’s reading and math tests each were given to
approximately 350,000 4th and 8th graders at more than 14,000 schools.

Now let’s look at the NAEP arts assessment. 
The 2008 test was given to just 7,900 students in 520 schools.  Now,
for analysis purposes, cut that number in half because the test was
actually two tests-one each in music and visual arts-and half the
sample took each. So fewer than 4,000 students in 260 schools took each
test.  That’s about 80 kids in five schools in each state.  Also, it
was given just to 8th graders-no 4th or 12th graders need apply.  And,
this is only the second time the arts assessment has been given in over
25 years. Reading has been given 13 times during that same period.

NAEP’s arts assessment is a different class of test than the NAEP
tests we talk about most often.  Let’s not be confused about the
quality of data the arts test represents.

A radio producer called me yesterday and asked what I thought about some of the questions on the test. He thought that some of what the 8th graders were being asked seemed to be a bit abstract, and stated that few adults, even those who appreciated art would not know the answers. He wondered whether or not the tests made any sense.

What does NAEP tell us, really? Based upon the sample size of eighth graders, in art it tells us what they know and to some extent are able to do (based upon the performance component). In music, it tells us what an eighth grade student knows about music, but not what they are able to do as there is no performance component. And in writing this, I have great misgivings, as I am not sure if what we are asking of these students on the tests is what we want them to know. That’s what the radio producer was asking me, basically.

What a student knows, is clearly in relation to the best that NAEP could do on an age and grade appropriate spectrum. What makes even this difficult, and in particular, places it wildly out of context, is that unlike math and reading, there are eighth grade students who may have had practically no music or arts instruction prior to eighth grade, while at the same time there are students who have been engaged in high quality sequential instruction for years leading up to the eighth grade.

I believe this creates a question of context that does not appear in math and reading. It made me think that perhaps we needed to look at something I would call quality of access. Lots of people would like to think the NAEP results really say something about teacher quality, or quality of instruction, and to some degree that true. However, the quality of access may be a more important factor, for without the sustained access which creates a continuum of instruction that is relatively common among all eighth grade students, the NAEP tests are somewhat meaningless.

By the way, you know that dance and theater were not included in either 1997 nor 2008 because the NAEP couldn’t find enough programs to create a statistically significant sample size?

I firmly believe that any statement NAEP makes about its assessment indicating that there hasn’t been a narrowing of the curriculum to be a terrible misstep, since they really don’t have that data on any reliable level. As I mentioned in my first post, they know what administrators report on what is offered. Even if we take that at face value, for the sake of argument, with all of the expanded efforts in the past 11 years totaling to a holding of the line, not very good line as equity and access goes, is in fact, a terribly disappointing measure, one that should cause many in this field to rethink approach.

One or two last thoughts.

NAEP doesn’t touch why we believe the arts are special, and to some degree different than other subject areas. It doesn’t address things like executive function, attitudes, or for that matter, the magic and beauty of the arts as a form of expression, builder of community, channel to the imagination, and all the other things that distinguish the arts from some of the other subject areas.

I think that this NAEP assessment is a terrific challenge for us to ask for more, to ask for better, to demand that the arts assessment receive the gold standard, not the chrome standard, and to do some of our own research taking this to the next important step.

For example, we are talking with Jennifer Jennings about trying to disentangle some of these long run trends in arts participation and access using the three repeat samples of high school students that we have from the 1980s, 1990s, and 2000s (High School and Beyond, the National Educational Longitudinal Study, and the Educational Longitudinal Study 2002). Each has high school transcript data as well as data about extracurricular participation, so it seems possible to identify long run trends in who participates within schools and how much (not just which schools have access, as NAEP arts did above), and also to see if the super accountability states saw declines in participation for low-performing kids over this period.

That’s just one interesting issue out there that represents how you can leapfrog from NAEP.

And, to wrap up for the moment, NAEP appearing at the same time as an NEA report that is rather startling in the declines it shows in arts participation, is quite the cocktail, don’t you think? In the very same week. How do ya like them apples?

But let’s not forget that the NEA study did not really look at the more popular forms of art. The NAEP test suffers from some of the same issue. I would love to see a study look at the complete picture of arts participation, as well as arts learning…