The Inter-American Development Bank has helpfully boiled down its 2016 study of El Sistema to one simple graphic. I haven’t got the time or skills to produce something so elegant, but inspired by the bank’s move, I’ve produced a list of key points to remember as you look at the graphic:
- El Sistema is supposed to be a poverty-reduction program, but the IDB’s evidence suggested that it had an anti-poor demographic profile at the time of the study.
- The researchers noted that the study “highlights the challenges of targeting interventions towards vulnerable groups of children in the context of a voluntary social program.”
- The study showed that nearly half the children admitted to the program failed to complete a full year.
- The study found no significant effects in 24 out of 26 categories measured.
- If you squint, you will see that the numbers on the left-hand side of the light green part have asterisks after them. These take you down to the bottom of the page, where the words “Significance level: *10 percent” have been added – helpfully, in light grey text on a white background. Using this significance level is highly unusual. The norm is 5%. According to that norm, the study found no effects in any of the 26 categories measured.
- A single study with results significant at the 5% level would not ordinarily be sufficient to justify a policy intervention, let alone at the 10% level. In the light of this and the previous points, the grounds for the final policymaking recommendations are weak.
- The finding that effects were concentrated in boys needs to be contrasted with an evaluation of a very similar Colombian program, the Red de Escuelas de Música de Medellín, in 2005, which found that the impact was five times higher in girls. More research needs to be done before firm claims relating to gender can be made.
To understand the graphic properly, we need to remember that the IDB’s research study produced a mixture of weak and negative findings after nearly two decades of loans. Little wonder, then, that all the caveats of the report have been removed. It is very likely that this graphic will provide another (shaky) prop to Sistema advocacy efforts; however, this time, unlike in the case of the 2007 loan proposal and its cost-benefit ratio of 1:1.68, we don’t need to wait several years before cottoning on to its flaws.
You can find the IDB study here, and a few thoughts on it here. More detailed consideration will come in my forthcoming article with Anna Bull and Mark Taylor in the British Journal of Music Education: “Who watches the watchmen? Evaluating evaluations of El Sistema.”