Does Sistema work? Does it matter? Reflections on the Longy/WolfBrown evaluation

To begin, let me share some thoughts on the recent report, “Building Strengths, Buffering Risk: Evaluating the Effects of El Sistema-Inspired Music Programs in the United States.”

1. One of the two partners behind the evaluation is Longy School of Music, which by its own account has:

a deep interest in and engagement with the leaders of FundaMusical, the umbrella organization led by Maestro Jose Antonio Abreu in Caracas, Venezuela.

Longy has actively supported the growth of Sistema nucleos around the country.  In 2012, the school formed a three-way partnership with the Los Angeles Philharmonic and Bard College called Take a Stand.  The first outcome of this partnership was the launching, in January, 2013, of a Masters of Arts in Teaching in Music (MAT), the only degree program in the world to offer graduate-level training in the principles of Sistema to musicians with the desire to respond to today’s unique educational needs and participate in the growing Sistema movement in the United States.

In other words, one of the evaluation partners has several dogs in this race.

2. The section on context begins with a quote by Abreu (i.e. El Sistema’s propaganda about itself), and follows this up by citing Tunstall (2012) and Booth & Tunstall (2016), both of which are unreliable advocacy texts by unqualified authors.

These are hardly promising or indeed appropriate foundations for a serious evaluation. Where is the peer-reviewed research? Why draw on “personal reports of successes in its home country and throughout the world” collected by the amateur Tunstall when there is professional research available?

The report’s authors cite the following study much further down (though their use of it suggests they haven’t examined it carefully):

Alemán, X., Duryea, S., Gurerra, N. G., McEwan, P. J., Muñoz, R., Stampini, M., Williamson, A. A. (2016). The effects of musical training on child development: A randomized trial of El Sistema in Venezuela. Prevention Science, 18, 865-878.

So why do they not use it in the context section, and why do they fail to mention its most striking finding: that El Sistema appears to have failed to reach the poor in Venezuela? The Venezuela study, in its own words, highlighted “the challenges of targeting interventions towards vulnerable groups of children in the context of a voluntary social program.” Why do the US evaluators do not mention this discovery, in the largest El Sistema study to date, when they bring up the question of poverty?

3. The authors write:

When José Antonio Abreu gathered a handful of instrumentalists to play together in a parking garage in Caracas, Venezuela in 1975 – a vision and a social movement were simultaneously born.

I have spent the last three years producing a variety of evidence that this statement is a myth. I have yet to see a single counter-argument. All the evidence – including Abreu’s own words and El Sistema’s original constitution (cited in my book) – suggest strongly that the musical program was created in 1975 and the “vision” or “social movement” was created some 20 years later. Why do the evaluators ignore the evidence and instead begin their study by reproducing the myth? This hardly inspires confidence.

4. The report only discusses positive findings.

It’s interesting to compare this document to the previous iteration a year ago. In the 2016 version, one of the three headline findings was Rising Academic Achievement. This has now disappeared without trace. Why is there no discussion? Did the authors decide that academic achievement was no longer interesting or important? Or did they search for signs of rising academic achievement and fail to find them?

Furthermore, the evaluation examined:

multiple areas of socioemotional development, including cooperation, empathy, academic behaviors, peer relations, academic self-concept, growth mindset, school engagement, self-efficacy, and perseverance.

Of these nine areas that were tested, six are not mentioned in the findings. The implication is that there were no significant impacts in six out of nine areas. This is surely important to know. Why does it need to be deduced, rather than being stated? The authors describe the genesis of the study as “an effort to prevent the claims for El Sistema-inspired programs from outpacing the evidence for their benefits,” yet nowhere do they specify what these claims are or which ones need to be reined in on the basis of their study.

5. Like most evaluations of Sistema programs I have read, it also ignores the political, ethical, and cultural complexities around this kind of music education. There is now a body of peer-reviewed research on these topics, which deals with El Sistema through lenses such as class, race, neoliberalism, neocolonialism, and so on – and which is overlooked by the authors. Ignoring a major question or body of literature would be a professional howler in peer-reviewed research, but evaluations are not peer-reviewed so their authors can do what they want (like advocacy writers).

Let’s take the example of race.

Nearly two-thirds of the students in the Longy/WolfBrown study were African American or Hispanic. Yet 86% of music teacher licensure candidates in the US are white. Is there a problem here? Not necessarily. But is there a question here? You bet, considering the long history in the Americas of white people attempting to “civilize” or “improve” poor and/or darker-skinned children through education in European-style music. It’s a question that came up, if mainly at the back of the hall, at Reframing El Sistema, where some delegates were troubled by the lack of people of colour in a conference about a program that caters mainly to people of colour in the US. But it’s not one that the authors either ask or answer.

How about gender?

The report actually includes a section on gender, which manages to reference the 2016 IDB study without mentioning that it found few effects for girls overall and “unexpectedly negative effects on empathy (among girls exposed to violence) and on working memory and prosocial behavior (among girls not exposed to violence).” The US evaluation also makes no mention of the strongly and obviously patriarchal character of El Sistema in Venezuela (e.g. the domination of leadership roles, conducting positions, and the top orchestra by men).

Conclusion

The evaluation shows all the signs of Sistema-friendly research, i.e. research that is designed to show El Sistema in the best possible light rather than to explore the pros and cons of this kind of music education. Their use of Booth and Tunstall’s non-academic work as a foundation stone and their omission of all the critical scholarship on El Sistema illustrates clearly the evaluators’ alignment with Sistema advocacy. The sense that their primary aim was to provide data to back up advocacy efforts is underlined by their answer to the question: “Why Research the Impact of El Sistema-inspired Programs? To keep the work alive, the field needs strong evidence that the promised musical growth and other forms of development actually occur.”

For anyone other than the Sistema faithful (and perhaps even for them), this report is something of a damp squib. It fails to shed useful light on music education. The message seems to be that music education is a little bit better than nothing, but in fact the lack of comparison to other activities means that we don’t even know whether the modest effects were a consequence of music education or simply of partaking in organized activities. Since no information about how the control and treatment groups were constituted (random or self-selection?), the perennial correlation/causation question hangs over the report. We don’t have enough information about either group to conclude that the differences in outcomes are solely due to music instruction. The researchers found dramatic variation in outcomes across programs, but while they note that “having a positive impact is not automatic, but requires careful program design and implementation,” they provide no information about what this might entail in practice. So we’re no closer to understanding which aspects of programs are beneficial, which are not, or why, nor – crucially, for those of us interested in music education more broadly – whether Sistema is better or worse than other forms of music education.

***

So, does Sistema work?

Well, we don’t really know much more than before. The evidence is mixed. In some ways, yes; in other ways, no.

But is this the wrong question?

Does private schooling work? Does corporal punishment work?

Does it matter?

Debates about these issues do not revolve to any significant degree around whether they “work” or not for “beneficiaries.” In the case of the former, the big question is the effect on those who do not participate, not on those who do. In both cases, the real issue is not whether they work but rather whether they are right.

The same is true of Sistema. The evidence from Venezuela and around the world provides little solid support for all the stories of “miracles” and so on, but more positive findings wouldn’t necessarily change much, because key problems lie elsewhere. Indeed, Owen Logan argues that the risks posed by El Sistema lie not in its failures but rather in its successes.

So while I’m interested in looking critically at the evaluation literature, I’m rather uninterested in the kinds of conclusions it draws, which miss many of the key issues. If Sistema is shorthand for social projects that aim to impact on poverty through music, then we need to think deeply about society, politics, economics, and culture, and not just look to numbers or brain scans to provide nice, simple answers. And I believe we need to rethink what “working” means. To recycle an example I have used many times before, the Third Wave “worked” – yet it was deeply disturbing in political and ethical terms, which is why its creator brought it swiftly to a close.

I recall Marshall Marcus’s public response to the characterisation of El Sistema in my book: “It may be an autocracy but it’s one that has allowed thousands of people to flourish.” This confronts us with questions about means and ends, about the compatibility or contradiction between autocracy and flourishing – philosophical and political questions, in other words, which cannot be resolved just by measuring the ends.

To consider the question of whether Sistema works (or can work), I wouldn’t look to evaluations (and nor would Sistema teachers in Scotland, England, and North America to whom I have spoken, who are just as sceptical about them as I am). I would look at how programs are set up, at their political and philosophical underpinnings, at how they’re addressing the big questions, at what they’re trying to do and how they’re trying to do it. This is why I’m interested in Brad Barrett’s project at Conservatory Lab Charter School, which I’ve mentioned before. Maybe this project contributed some good numbers to the Longy/WolfBrown report; maybe it didn’t. But I honestly don’t care.

It talks about citizenship – but in a meaningful way, not just as a buzzword. It presents a vision of music and social justice that aligns with research in these areas, not with ungrounded and at times absurd advocacy. It talks about politics, about inequality, about inequity, about voice – topics missing in the original Sistema model. It incorporates creation and other art forms (ditto). None of these elements can be measured quantitatively, yet they are fundamental to social action through music as a meaningful practice and not just a label to slap on a building in return for millions of dollars.

If I have a reservation about this project, it stems from this:

Artists must think critically, constantly hone their reflective processes, and at best, seek to hold up a mirror to society so it can see itself clearly. If we truly seek to provide a social justice education, I believe we must collectively go much further than assuming teaching music will empower students and improve society. In and of itself, it won’t.

This belief bears no relation to El Sistema, and it’s a shame that this project is operating under (and thereby, to most people, promoting) that brand name. It makes a mockery of the idea that there is a unifying thread to Sistema, since it (rightly) rejects the Venezuelan program’s central idea: as Gustavo Dudamel put it recently at the Take A Stand Festival in LA, “with these instruments and this music, we can change the world, and we are doing it.”