Who watches the watchmen? Evaluating evaluations of El Sistema

I first became interested in evaluations of El Sistema when I was writing my book. Like many people, I had simply assumed that evaluations – particularly quantitative ones, full of dense statistics – were reliable and unquestionably so. I have to acknowledge my naivety. Also, it didn’t cross my mind that a scholar of music such myself might have the skills to read such an evaluation critically.

But when I got hold of the report that underpinned the Inter-American Development Bank’s $150-million loan to El Sistema in 2007, I was amazed to find that even I could see what looked like holes in its argument. So I got a top-level development economist to examine it thoroughly, and they backed up my first impression: it was a very questionable study.

Despite this experience, I then made the same mistake with the first major evaluation of Sistema Scotland. It wasn’t until I read Owen Logan’s scathing critique of this and the subsequent study that I went back for another look. My own take was a little different from Logan’s, though complementary rather than contradictory: I was interested to find that the 2015 evaluators did in fact point up problems with the program – problems that were described to me independently by other researchers and former teachers – but then played them down again in their summaries and overviews (and of course, the program and the media then removed the caveats altogether in their public presentation of the findings).

Mark Rimmer’s study of In Harmony Sistema England in a recent issue of the British Journal of Music Education (BJME) is primarily a critical look at children’s responses to the program, and in this respect it puts another dent in the hyperbolic narrative of El Sistema. However, it also includes some commentary about evaluation of the English program:

England’s Sistema-inspired initiative, In Harmony, has been subject to ongoing independent evaluation by the National Foundation for Educational Research (NFER), with several reports issuing as a result (Lord et al., 2013; Lord et al., 2015). The aim of the NFER research has been ‘to track and measure the impacts of the current six In Harmony programmes’ (Lord et al., 2013, p. 2), with ‘impact’ primarily interpreted in terms of ‘the social, emotional and educational development of children’ (Lord et al., 2013, p. 2) and assessed through a combination of key stage attainment measures in literacy and numeracy, pupil attendance figures, a pupil questionnaire and case study visits. Illuminating as these reports may be in certain regards however, they ultimately provide very little insight into the ways participating children have responded to the initiative.

Consider the pupil questionnaire employed by NFER. This appears to suffer from a number of weaknesses, especially in relation to its effort to explore children’s ‘Musical enjoyment and achievement’ (Lord et al., 2013, p. 45). For example, questions about In Harmony participation are framed – far too broadly – in terms of ‘music’ (such that responses about enjoying, learning or playing music may not necessarily relate to In Harmony activities at all). Secondly, the statements against which participating children are able to register their responses in respect of ‘musical enjoyment and achievement’ appear to have been framed in an exclusively positive way (e.g., ‘I am learning a lot in music’, ‘I like playing my instrument in class’, ‘I am able to play more difficult pieces now’), while the response options to such statements – ‘Yes a lot’, ‘Yes a bit’, ‘No’, ‘Not sure’ and ‘No response’ – are also somewhat skewed towards the elicitation of agreement. The pupil questionnaire thus appears to have provided only minimal scope for children to register their disengagement/dislike and no scope at all for the elaboration of any criticisms.

It is also worth noting that children’s own words are conspicuous by their relative absence across the two NFER reports. When verbatim quotations are offered to help illuminate and substantiate key claims, the overwhelming majority issue from school heads and teachers (15 quotes), parents and grandparents (seven quotes) or else In Harmony managers and tutors (five quotes). Children’s reflections, meanwhile, are cited on just two occasions across both reports and in each case they take the form of endorsements (one in relation to music learning, another to a child’s membership of a project steering group). Such an under-representation of children’s reflections is quite surprising given that more interviews were undertaken with them (38 in total) than any other stakeholder group (22 parents, 20 heads/teachers and 12 In Harmony managers/tutors; see Lord et al., 2013, p. 47).

Building on these critical appraisals of Sistema evaluations, Anna Bull, Mark Taylor and I have just published a new article, also in the BJME, entitled “Who watches the watchmen? Evaluating evaluations of El Sistema.” We bring together my knowledge of the Sistema field with Mark and Anna’s expertise in quantitative and qualitative methods respectively, and look closely at evaluations of El Sistema in Venezuela and Aotearoa. We conclude that these studies display an alignment with advocacy rather than explorative research, and that evaluations should therefore be examined through a critical lens rather than their conclusions taken as given.

Something that emerges from observing a range of evaluations of El Sistema and spinoff programs is a consistent pattern of over-claiming. It is not a problem with one or two studies, an isolated or random flaw, but rather a feature of the field that has been present for over two decades, since the first evaluations of El Sistema in Venezuela in 1996 (see my recent article with Ana Lucía Frega). This suggests that there is something in the nature of the process that generates overly optimistic findings. The Sistema field has always been distinguished by idealization and hyperbolic claims, and with only a couple of exceptions, evaluations have done little to temper idealistic visions with a dose of realism. For this reason, independent critical studies are essential in order to provide some balance in the field.

Anyone interested in reading our article may contact me directly at geoff.baker@rhul.ac.uk. Alternatively you can wait until it is published officially in the BJME later this year.