IDB study sheds doubt on El Sistema’s claims of social inclusion and transformation (full version)

[13/01/2017] The results of the first officially sanctioned Inter-American Development Bank experimental study of El Sistema indicate that:

1. El Sistema children are three times less likely to be poor than all 6 to 14 year-olds

2. Significant positive effects were noted in only 2 out of 26 variables measured

3. No positive effects, and some negative ones, were noted among girls

4. Nearly half of the children admitted to the program failed to complete a full year

There have been several attempts at quantitative studies of El Sistema since the early 2000s, but none has been convincing: they have suffered from methodological flaws, or the methods or the studies themselves have not been available for scrutiny (Baker 2014). The Inter-American Development Bank (IDB), El Sistema’s major non-governmental funder, distanced itself from all previous studies – including one that it had previously used – when it commissioned the biggest and most thorough one to date in 2011, at a cost of $1 million. The first report has now come in – and it’s not good news for the IDB, El Sistema, or its supporters. What it does do, however, is confirm some of the principal findings of my research.

The article, entitled “The Effects of Musical Training on Child Development: a Randomized Trial of El Sistema in Venezuela,” was published in the journal Prevention Science in November 2016.

1. Poverty

If one argument in my book caused more furore than any other, it was the suggestion that El Sistema was not dominated by or targeted at poor children, as had been claimed by press releases, advocacy statements, and newspaper articles for the previous 7 years. I questioned the linkage of El Sistema with poverty in the following terms:

A key to the program’s fame is its claim to focus on deprived children. FESNOJIV’s mission statement refers to the “rescue of the most vulnerable groups in the country,” while in his TED prize speech, Abreu claimed that “the large majority of our children belong . . . to the most vulnerable strata of the Venezuelan population.” Borzacchini (2011, 7) describes El Sistema as “saving our children from the horror of violence, drugs and material and spiritual poverty.” Lennar Acosta, a former juvenile delinquent rescued by El Sistema, is often held up as representative of the program.

Overseas, Booth (2008, 1–2) claims that the project “teaches 300,000 of Venezuela’s poorest children” and “turns around the lives of hundreds of thousands of at-risk kids.” According to Deutsche Grammophon’s publicity, 90 percent of El Sistema’s students come from poor socioeconomic backgrounds, and this figure appears in a range of widely dispersed sources. Such accounts suggest that extreme poverty is the norm, but how accurate are they?

My research with musicians of all ages in different parts of Venezuela suggested that all kinds of children can be found in El Sistema, from reformed delinquents to the children of the rich; but the majority, at least in the provinces (where most live), are middle-class, with a significant number of lower-middle-class children, but few who could be described as deeply deprived, at-risk, or Venezuela’s poorest children.

This conclusion, drawing on qualitative evidence, has now been bolstered by the quantitative evidence from the IDB’s study, which reveals that the estimated poverty rate among the El Sistema children is 16.7%, while the rate for the states in which they live is 46.5%. In other words, the El Sistema children in the experiment were three times less likely to be poor than all 6 to 14 year-olds residing in the same states.

Further research is needed to confirm whether the experimental sample is representative of all program applicants, since the intention of the study was not to determine the poverty rate in El Sistema. Nevertheless, this is the largest and most rigorous study to date, and nearly 3000 children across 16 núcleos in 5 states constitute a broad and significant sample of El Sistema participants. This is the best data yet gathered; the poverty figures are therefore highly suggestive.

That children from households in poverty are significantly under-represented in the study is hardly a surprising finding, in fact, given that the program draws its participants via an application system rather than more proactive, inclusive methods. Requiring guardians to seek out a program and fill in an application form is likely to skew participation towards more advantaged families.

Nevertheless, it is hard to overstate the importance of this finding, which is presented almost in passing in the depths of the article. It raises significant questions about El Sistema’s central claim: that it is a social inclusion program aimed at the most vulnerable in society. It is worth emphasizing that during my research in Venezuela, I repeatedly encountered skepticism about this claim, which appears to have emerged in the 1990s, primarily for strategic reasons, rather at the project’s outset in the 1970s. The new figures justify that skepticism.

The admission that the study “highlights the challenges of targeting interventions towards vulnerable groups of children in the context of a voluntary social program” is a major one from the IDB, and a very awkward one for all parties concerned. It suggests that El Sistema’s social program is undermined by a basic design flaw, one that it shouldn’t have taken the bank 18 years and $160 million in loans to discover. The evidence that El Sistema may actually be catering to a relatively advantaged population also throws 10 years of international advocacy and journalism about El Sistema “saving the poor” into doubt.

2. Effects

The study measured 26 primary outcome variables within four domains: self-regulatory skills, behaviours, prosocial skills and connections, and cognitive skills. Only 2 significant outcomes were found: in child-reported self-control, and child-reported behavioural difficulties. No significant effects were found for outcomes in the other 24 areas; there were no full-sample effects on cognitive skills or prosocial skills and connections. There have been many and sweeping claims over the last decade that El Sistema develops a wide range of social skills and values; such claims find no support in the study, further questioning the program’s credentials as a social program.

The most significant positive effects were found among children with less-educated mothers and boys exposed to violence. This suggests that El Sistema may be a positive intervention for more disadvantaged children. However, the statistics discussed in the previous section reveal that El Sistema is failing to target such children effectively. Furthermore, the discovery of negative effects on prosocial skills and connections for children with more educated mothers raises questions about the costs and benefits of El Sistema for children from less disadvantaged circumstances.

Additionally, there is a potential generalisation problem given the application process. Boys with less educated mothers benefited from the program, but these were boys with less educated mothers who applied for the program. In other words, this is a particular subset of less educated mothers. So it cannot be assumed that the program is effective for all disadvantaged children. Boys with less educated mothers whose guardians think music is a waste of time, or are too busy or burdened or disorganized to apply, might not be helped significantly by the program. It may be that parental support is a key ingredient in generating positive effects – something that I concluded in my book, and an idea that I have encountered repeatedly in subsequent conversations with researchers and practitioners in the field of social action through music. This would limit the wider applicability of the program, particularly to the most disadvantaged sectors of society.

So many claims have been made about El Sistema’s effects in so many areas that it was likely that a couple would turn out to be justified. The real story here, though, is quite how many did not, casting doubt on El Sistema’s transformative power.

3. Gender

The report notes: “Unexpectedly, we found few effects for girls overall, with some unexpected decreases in different skill domains.” The study found “unexpectedly negative effects on empathy (among girls exposed to violence) and on working memory and prosocial behavior (among girls not exposed to violence).”

No explanation for these findings is offered, and I cannot provide one. But these are significant results in the light of the gender issues that have been raised previously with regard to the higher levels of El Sistema (the domination of leadership roles, conducting positions, and the top orchestra by men) and the institution as a whole (allegations of sexual harassment and abuse, and allegations that sexual relationships between teachers and students are common (Scripp 2015)). The IDB report reveals that differing experiences for males and females begin in the first year of study, further underlining that gender issues in El Sistema should be investigated with urgency.

4. Dropout rate

No official figures for the dropout rate in El Sistema have ever been collected, but I suggested in my book that the qualitative evidence pointed to a high level of desertion. I quoted one teacher’s ballpark estimate that half the students in his núcleo dropped out within a year of inscription. According to the IDB study, 44% of students who were offered a place failed to complete two semesters. This information may also be relevant to the question marks over the official figures for El Sistema’s size.

Summary

El Sistema’s mission statement describes “the pedagogical, occupational, and ethical salvation of children and young people, via the instruction and collective practice of music, [and] dedicated to the training, protection, and inclusion of the most vulnerable groups in the country.” According to its Vision statement, the program focuses on the “comprehensive development of human beings,” and cultivates “transcendental values that influence the transformation of children, youths, and the family environment,” offering “an opportunity for personal development on intellectual, spiritual, social, and professional levels, rescuing children and young people from an empty, disorientated, and deviant youth.”

The IDB’s study raises important questions about these claims. The statistics contradict the alleged focus on “the most vulnerable groups in the country.” The failure to find significant effects in most domains undermines the notion of “transformation” or “comprehensive development of human beings.” And with significant outcomes noted only in the realm of discipline (to which the two variables that showed significant positive results – self-control and behavioural difficulties – are closely related), there is little sign of “transcendental values” or “personal development on intellectual, spiritual, social, and professional levels.”

In contrast, the report confirms a number of arguments in my book El Sistema: Orchestrating Venezuela’s Youth (OUP, 2014): that discipline is El Sistema’s main value; that there are good reasons to be sceptical about most other social claims; that the program provides males and females with unequal experiences; that the dropout rate is high; and that the program is not targeted at or dominated by the poor. (In fact, the report goes further than my book, which simply suggested the program was not pro-poor; the study indicates that Sistema children may be well above average in terms of socio-economic status.) The significant gap between the researchers’ theory of change and their actual findings echoes the gap between beliefs and realities, theory and practice, media images and lived experiences that I uncovered in my book.

Further discussion

There are certain limitations to the study and further issues to consider:

1. There is no mention of the considerable existing academic literature on El Sistema – including that which anticipated this report’s findings. In scholarly terms, the program has been criticized much more than praised, something that the authors omit entirely. (They do, however, cite unreliable non-academic sources in the introduction.)

2. A limitation that the authors recognize is that “findings are limited to self- and guardian-reported outcomes, which may introduce bias associated with scale measures.” In other words, the study depended to a significant degree on asking children and guardians what they thought about music education in El Sistema. It didn’t take account of the possibility that both children and guardians might be influenced by messages and beliefs about what music education is supposed to do.

Let us take the example of discipline and empathy. According to the study, El Sistema has effects in the realms of child-reported self-control and child-reported behavioural difficulties, but not that of empathy. One perfectly plausible explanation for this finding is that El Sistema focuses on discipline but not empathy, and therefore has effects in the realm of discipline but not empathy. But it’s also the case that El Sistema’s figures of authority constantly articulate discipline as the program’s primary value. In which case, it may be that children spent a year in an environment where they heard the word “discipline” regularly from teachers and administrators, but didn’t hear the word “empathy.” And when asked about the effects of the program, they reported effects in the realm that was discursively familiar to them. So whether the program actually inculcates discipline (or doesn’t inculcate empathy) remains a moot point.

3. This study measures certain things and not others. This truism applies to all research. However, some of the things that this study does not measure are rather important. For example, what was the profile of the 44% of accepted applicants who did not make it through two semesters? Are the most disadvantaged children dropping out disproportionately? What are the implications of these statistics with regard to social inclusion? Why did these children leave? It may be that there are barriers or disincentives that affect certain kinds of children disproportionately, which – if true – could have huge implications for the design and effectiveness of the program. Studying El Sistema’s dropouts is thus just as important as studying those who remain in the program, and an important opportunity was missed here.

The lack of a second treatment group undertaking an alternative activity is also disappointing. For example, it would have been fascinating to compare El Sistema to a music education program based on music therapy (Crooke, Smyth, and McFerran 2016) rather than orchestral training, or one focused on empathy (Rabinowitch 2012) or creativity (Koutsoupidou and Hargreaves 2009). After all, a program with a more therapeutic focus or an emphasis on empathy might prove particularly beneficial for a country as violent and polarized as Venezuela.

Few people question that music education is a good thing. The real question is: what sort of music education? Which practices are more or less effective with regard to desired goals? This study tells us little about the specific value of Sistema-style education as compared to other kinds. (That said, it does suggest that music educators who are more interested in social and cognitive skills than discipline would be well advised to consider other models.)

4. A further issue, again mentioned in the report, is the question of duration. It may be that greater effects would be noticed in children who spent more than one year in the program (though fadeout of impacts is also possible, as the authors note). However, without more detailed information about the children who drop out, it would be hard to draw larger conclusions from such data. The findings already suggest that the program does not effectively target the poor; if it were also the case that the most disadvantaged children were more likely to drop out, then evidence of greater social and cognitive effects would not plug the holes in El Sistema’s claims to be a social inclusion program aimed at the most vulnerable.

5. Then there is the issue of representativeness. Of the 24 music centres approached by the researchers, 8 did not participate: “Two sites were excluded because their directors declined to follow the experimental protocol, and six were excluded because of insufficient demand.” Consequently, the study “was limited to over-subscribed music centers which may have implicitly favored better known and/or higher-quality sites.” My own research suggests that quality varies significantly across the program, so even the limited findings of this study may – as the authors admit – reflect the top end of El Sistema rather than the average.

6. Finally, the study reveals the risks of seeking justifications for the arts on utilitarian grounds. As Belfiore (2002) notes, the arts play a dangerous game by arguing for their value on instrumental grounds without robust evidence of their efficacy; they are then vulnerable to being displaced by social programs with more demonstrable results. There is little if anything in the IDB report to suggest that El Sistema’s music education brings social value for either girls or children with more educated mothers – a major proportion of the program’s participants. Yet arts-based justifications for their involvement are also absent: for example, the word “creativity” is not mentioned once in the study.

If we boil the article down to its essence, it suggests that El Sistema “works” only as a kind of National Service for poor boys, while admitting that this group only represents a small minority of the program. Through the lens of the article, at least, most participants appear to get nothing significant out of their involvement. The positing of utilitarian arguments and the search for quantitative evidence to support them leads to an uninspiring and even self-defeating conclusion, one that could – in a perfectly rational world – see El Sistema displaced by a more effective social program that had nothing to do with music.

Conclusion

The big question is what happens next. If the IDB and El Sistema use the report as the basis for a serious reassessment of multiple aspects of the program, ranging from targeting and retention of students to curriculum and pedagogy, then it will turn out to be very valuable and a wise investment of $1 million on the part of the bank. If, however, the report is to be spun by the IDB and ignored by El Sistema, then it will be a case of throwing good money after bad.

Initial signs are not particularly promising, it has to be said. El Sistema is not renowned for its adaptability or openness to change, and the IDB’s blog post on the report was indistinguishable in tone from El Sistema’s institutional propaganda, giving an extremely optimistic take on the study. The study itself is exactly what you would expect from an academic article whose lead authors are employees of the funder of the program under the spotlight: it scrupulously reports all the findings, positive and negative, while giving more emphasis to the former and less to the latter. This gentle slant is amplified considerably in the blog post on the report, which mentions none of the negative or equivocal findings, only the positives. The blog post even makes positive claims about gender equality, whereas the report itself actually suggests the opposite. The overall impression given by the blog post is that the study provides a stamp of approval for El Sistema, which is far from the more complex truth.

It is hardly surprising that the IDB should project the report in such a positive light: after all, it has poured $160 million into the program over 18 years. But a failure even to acknowledge the problematic or disappointing findings is hardly an encouraging sign that the bank is focusing on the lessons to be learnt. Given the extent of El Sistema’s global PR success – with the IDB’s support – it will take more than a congratulatory pat on the back to spur much needed change. Nevertheless, it may be that behind the optimistic front, the IDB is looking at its $160 million in loans, the limited outcomes that have been demonstrated, and the non-existent regional centres that it has paid for, and thinking: it’s time for action.

Finally, the report raises questions for international Sistema advocacy and groups such as Sistema Global. After the publication of my book, there was much ill-informed discussion of the relative merits of quantitative and qualitative research. There are various points that can be drawn from the IDB study: that quantitative studies will not provide all the answers; that such studies may not provide the holy grail that Sistema programs are seeking; and that the quantitative evidence provides considerable support for the qualitative studies. This then leads to the question of the uses to which such research may (or may not) be put: will the Sistema-inspired field ignore the IDB report in favour of more positive studies? Or will it take both seriously, in order to reassess the model on which it is based and rethink its own future?

References

Baker, G. 2014. El Sistema: Orchestrating Venezuela’s Youth. New York: Oxford University Press.

Belfiore, E. 2002. “Art as a Means of Alleviating Social Exclusion: Does It Really Work? A Critique of Instrumental Cultural Policies and Social Impact Studies in the UK.” International Journal of Cultural Policy 8 (1): 91–106.

Booth, E. 2008. “Thoughts on Seeing El Sistema.” http://www.americanorchestras.org/images/stories/lld_pdf/elsistema_Booth.pdf.

Borzacchini, C. 2010. Venezuela en el Cielo de Los Escenarios. Caracas: Fundación Bancaribe.

Crooke, A. H. D., P. Smyth, and K. S. McFerran. 2016. “The Psychosocial Benefits of School Music: Reviewing Policy Claims.” Journal of Music Research Online 7.

Koutsoupidou, T., and D. J. Hargreaves. 2009. “An Experimental Study of the Effects of Improvisation on the Development of Children’s Creative Thinking in Music.” Psychology of Music 37 (3): 251–78.

Rabinowitch, T. C. 2012. “Musical Games and Empathy.” Education and Health 30 (3): 80-84.

Scripp, L. 2015. “The Need to Testify: A Venezuelan Musician’s Critique of El Sistema and his Call for Reform.” http://www.researchgate.net/publication/285598399