[13/01/2017] (A longer version of this article, with more discussion and references, can be found in the next post)
The results of the first officially sanctioned Inter-American Development Bank experimental study of El Sistema indicate that:
1. El Sistema children are three times less likely to be poor than all 6 to 14 year-olds
2. Significant positive effects were noted in only 2 out of 26 variables measured
3. No positive effects, and some negative ones, were noted among girls
4. Nearly half of the children admitted to the program failed to complete a full year
There have been several attempts at quantitative studies of El Sistema, but none has been convincing. The Inter-American Development Bank (IDB), El Sistema’s major non-governmental funder, distanced itself from all previous studies when it commissioned the biggest and most thorough one to date in 2011, at a cost of $1 million. The first report has now come in, published in the journal Prevention Science – and it’s not good news.
The estimated poverty rate among the El Sistema children is 16.7%, while the rate for the states in which they live is 46.5%. In other words, the El Sistema children in the experiment were three times less likely to be poor than all 6 to 14 year-olds residing in the same states.
Further research is needed to confirm whether the experimental sample is representative of all program applicants, since the intention of the study was not to determine the poverty rate in El Sistema. Nevertheless, this is the largest and most rigorous study to date, and nearly 3000 children across 16 núcleos in 5 states constitute a broad and significant sample of El Sistema participants. This is the best data yet gathered; the poverty figures are therefore highly suggestive.
That children from households in poverty are significantly under-represented in the study is hardly a surprising discovery, given that the program draws its participants via an application system. Nevertheless, it is hard to overstate the importance of this finding, which is presented almost in passing in the depths of the article. It raises significant questions about El Sistema’s central claim: that it is a social inclusion program aimed at the most vulnerable in society.
The admission that the study “highlights the challenges of targeting interventions towards vulnerable groups of children in the context of a voluntary social program” is a major and very awkward one from the IDB. It points to a basic design flaw in El Sistema, one that it shouldn’t have taken the bank 18 years and $160 million in loans to discover. It also throws 10 years of international advocacy and journalism about El Sistema “saving the poor” into doubt.
The study measured 26 primary outcome variables within four domains: self-regulatory skills, behaviours, prosocial skills and connections, and cognitive skills. Only 2 significant outcomes were found: in child-reported self-control, and child-reported behavioural difficulties. No significant effects were found for outcomes in the other 24 areas; there were no full-sample effects on cognitive skills or prosocial skills and connections. There have been many and sweeping claims over the last decade that El Sistema develops a wide range of social skills and values; such claims find no support in the study.
The most significant positive effects were found among children with less-educated mothers and boys exposed to violence. This suggests that El Sistema may be a positive intervention for more disadvantaged children. However, the study also reveals that El Sistema is failing to target such children effectively, and that costs may outweigh benefits for less disadvantaged children. Additionally, there is a potential generalisation problem given the application process. Boys with less educated mothers whose guardians are too busy, burdened, uninterested or disorganized to apply might not be helped significantly by the program. It may be that parental support is a key ingredient in generating positive effects.
The research found few effects for girls overall and “unexpectedly negative effects on empathy (among girls exposed to violence) and on working memory and prosocial behavior (among girls not exposed to violence).” These are significant results in the light of the gender issues that have been raised previously with regard to the higher levels of El Sistema (the domination of leadership roles, conducting positions, and the top orchestra by men) and the institution as a whole (allegations of sexual harassment and relationships between teachers and students). The finding that differing experiences for males and females begin in the first year of study underlines that gender issues in El Sistema should be investigated with urgency.
4. Dropout rate
No official figures for the dropout rate in El Sistema have ever been collected, but the qualitative evidence points to a high level of desertion. According to the IDB study, 44% of students who were offered a place failed to complete two semesters.
El Sistema’s mission statement describes “the pedagogical, occupational, and ethical salvation of children and young people” via a program of “training, protection, and inclusion of the most vulnerable groups in the country.” According to its Vision statement, the program focuses on the “comprehensive development of human beings,” and cultivates “transcendental values that influence the transformation of children, youths, and the family environment,” offering “an opportunity for personal development on intellectual, spiritual, social, and professional levels, rescuing children and young people from an empty, disorientated, and deviant youth.”
The IDB’s study raises important questions about these claims. The statistics contradict the alleged focus on “the most vulnerable groups in the country.” The failure to find significant effects in most domains undermines the notion of “transformation” or “comprehensive development of human beings.” And with significant outcomes noted only in the realm of discipline, there is little sign of “transcendental values” or “personal development on intellectual, spiritual, social, and professional levels.”
In contrast, the report confirms a number of arguments in my book El Sistema: Orchestrating Venezuela’s Youth (OUP, 2014): that discipline is El Sistema’s main value; that there are good reasons to be sceptical about most other social claims; that the program provides males and females with unequal experiences; that the dropout rate is high; and that the program is not targeted at or dominated by the poor. (In fact, the report goes further than my book, which simply suggested the program was not pro-poor; the study indicates that Sistema children may be well above average in terms of socio-economic status.) The significant gap between the researchers’ theory of change and their actual findings echoes the gap between beliefs and realities, theory and practice that I uncovered in my book.
The big question is what happens next. If the IDB and El Sistema use the report as the basis for a serious reassessment of multiple aspects of the program, ranging from targeting and retention of students to curriculum and pedagogy, then it will turn out to be very valuable and a wise investment of $1 million. If, however, the report is to be spun by the IDB and ignored by El Sistema, then it will be a case of throwing good money after bad.
Initial signs are not particularly promising. El Sistema is not renowned for its adaptability or openness to change, and the IDB’s blog post on the report mentions none of the negative or equivocal findings, only the positives, and it misrepresents the results relating to gender. It gives the impression that the study provides a stamp of approval for El Sistema, which is far from the more complex truth.
It is hardly surprising that the IDB should project the report in such a positive light, given its long-term investment. But a failure even to acknowledge the problematic or disappointing findings is hardly an encouraging sign that the bank is focusing on the lessons to be learnt. Nevertheless, it may be that behind the optimistic front, the IDB is looking at its $160 million in loans, the limited outcomes that have been demonstrated, and the non-existent regional centres that it has paid for, and thinking: it’s time for action.