Being on a judging panel is difficult. As soon as the judge has to deal with qualities such as style, appearance, taste etc it gets more difficult. So how does Choice deal with subjectivity and personal preference when choosing and operating a judging panel? One obvious part of the solution is to select professionals who are used to the process. This aims to ensure that the judges know the standards that are expected, are capable of distinguishing examples and have the discipline to stick to the procedure. Looking at the panels selected by Choice it is apparent that some effort goes into the choice of judges.
Regardless of the selection of judges it would be good to have some objective measure of whether the process is delivering the expected quality. One examination of the process is to look for consistency. If they are consistent then at least we think the judges are rating the same thing the same way - even if we are not 100% sure what they are rating.
I can think of two tests of consistency. One is between the judges. If the raw scores given by each judge are highly correlated that is good. If not, we need to know why not. There are cases when anomalies are apparent and the reason is obvious. For example, in gymnastics or diving, Russian judges judging American performers and vice versa. In other cases the reasons may not be at all clear.
The second test is over time. That is you repeat the blind judging two weeks later with the same panel and exhibits but with a different random ordering. This allows us to assess if judges are consistent with themselves. If the results vary very much you have a problem.
A good judging is reproducible, the outcome should not be dependent on unknown external variables.
So does Choice perform either of the above tests or any other kind of quality control over judging panels other than their selection?
And, given the number of volunteers for the position, are ice cream judges like wine judges? Do they cleanse their mouth with soda water between tastes and most importantly do they swallow or have to spit in a bucket?