Scoring Artistic Quality: the Meaning of Numbers
I recently received an email asking if I might like to write about the experience of being an adjudicator - which of itself is quite a broad topic, and one on which I have wandered across occasionally in previous posts. The email then homed in on a much more specific kind of question, as follows:
What I'm particularly interested in is the fact that in my experiences, it seems to be really difficult for a chorus at music festivals get less than 75%, no matter how bad they are, and the difference between results for barbershop choruses in a festival may be just 5%, but when that is transferred to a barbershop competition, the spread is 30% +, which means that you then have to try to explain to the chorus why they can do so well in a general festival compared to a barbershop comp.
Now, this is one of those questions that has what at first looks like a simple answer, but when you live with it for a while, you see that the simple answer contains within it a rather more interesting point.
To start with the scoring system used at festivals affiliated to the British and International Federation of Festivals (and indeed, some others that are not affiliated, but use the same system to make things easier for competitors and adjudicators who are used to it). Here is a summary from the bottom of a typical adjudicator's sheet:
So, you can see why the worst performance imaginable isn't going to score lower than 75, can't you? 75 is the effective Zero. In fact, it is quite misguided to use the concept of percent in this context, as the score from 75-90+ (and note, it implies rather than states a notional maximum of 100) are the totally of scores available. A score of 80 is not four-fifths of a maximum when all the numbers below 75 are deemed not to exist.
So, before we start wondering how this system maps onto any other, let us just take a moment to consider why you might develop a system that start at 75. Why not use the numbers 0-25 instead? I don't know anything of the background here - who decided this at what stage - but I can think of reasons why somebody might make this decision.
Well, first because even people who get the lowest possible score don't actually deserve a zero. They have done better than nothing: they have turned up, and produced a performance that fulfils the requirements of the syllabus. Even if the performance is not very competent, that is way more than most people ever manage. The difference between expert and inexpert may be huge, but it's not as big as the difference between doing and not doing.
The simple answer to this question is therefore just that there are different scoring systems in different contexts. The marking systems for graded instrumental exams or educational qualifications are different again, and you just get used to each in its own context.
The more interesting point, though, is the nature of the scores themselves. It is the illusion of percentages that leads people to think that an 85 in a festival should have any simple relationship with an 85 in any other form of scoring. It makes people think quantitatively, as if the numbers are actually involved in counting something.
Whereas, in artistic contexts, the numbers are actually adjectives. An ex-colleague once told a story about doing some instrumental examining in a college in France, in which the marking categories were 'bon', 'très bon' and 'pas bon'. This is actually a rather beautiful system: easy to use and clear in its messages to the students without inviting all that unseemly comparing of marks beyond the basic 'did you do all right then?'
Of course, it is less useful if, say, you want to combine the grades from a number of assessments to produce an overall qualification, or indeed if you want to rank a large number of similarly-skilled performances, or to give someone some idea of how much they have progressed since the last time they received a score. Numerical systems are much more useful for each these purposes.
But whatever operations we may carry out on the numbers, the scores themselves remain essentially adjectives. You can see this in the way that they are usually derived from grade-related assessment descriptors based on bandings. The scoring process isn't one of adding up numbers to reach a total, but of identifying a general level of accomplishment, and differentiating marginally higher or lower levels of achievement within that band using the numbers in that range.
(Even the more analytical systems that require each scorer to rate a number of different elements which are then added up still work on this principle within each category. Indeed, I used to like the A-level Music dissertation marking scheme that required the marker to go through both an analytical process, and produce a holistic mark, and then compare them as an initial check on the validity of each.)
So, the notion of getting a '75' only has any meaning in the context in which it was awarded. In a music festival, that's not very skilled, but thanks for being here. In a barbershop contest, that's really rather good. In an Associated Board of the Royal Schools of Music instrumental exam, it's a spectacular fail. We can use our knowledge of how numbers work to make comparisons and draw conclusions from numbers produced by the same system, but unless we look at the system's key to decode the numbers back into adjectives, we won't actually know what they mean in artistic terms.