Consistency of Choice testing results

Being on a judging panel is difficult. As soon as the judge has to deal with qualities such as style, appearance, taste etc it gets more difficult. So how does Choice deal with subjectivity and personal preference when choosing and operating a judging panel? One obvious part of the solution is to select professionals who are used to the process. This aims to ensure that the judges know the standards that are expected, are capable of distinguishing examples and have the discipline to stick to the procedure. Looking at the panels selected by Choice it is apparent that some effort goes into the choice of judges.

Regardless of the selection of judges it would be good to have some objective measure of whether the process is delivering the expected quality. One examination of the process is to look for consistency. If they are consistent then at least we think the judges are rating the same thing the same way - even if we are not 100% sure what they are rating.

I can think of two tests of consistency. One is between the judges. If the raw scores given by each judge are highly correlated that is good. If not, we need to know why not. There are cases when anomalies are apparent and the reason is obvious. For example, in gymnastics or diving, Russian judges judging American performers and vice versa. In other cases the reasons may not be at all clear.

The second test is over time. That is you repeat the blind judging two weeks later with the same panel and exhibits but with a different random ordering. This allows us to assess if judges are consistent with themselves. If the results vary very much you have a problem.

A good judging is reproducible, the outcome should not be dependent on unknown external variables.

So does Choice perform either of the above tests or any other kind of quality control over judging panels other than their selection?

And, given the number of volunteers for the position, are ice cream judges like wine judges? Do they cleanse their mouth with soda water between tastes and most importantly do they swallow or have to spit in a bucket?


… something like this I imagine …


Good questions @syncretic.

As you have noted, we chose a panel of experts (their details here for reference). With something as subjective as a taste test, I’m not sure you can ever rule out all personal aspects of delivering a verdict, but knowing that these are food professionals is comforting in this regard.

We’ve discussed your questions in the office this morning, and we’re going to explore ways we can allow people to delve deeper into the process. I think you have a good point on looking at correlation, and in other taste tests (like peanut butter) we’ve collected a higher volume of tests from the general public (rather than expert panel) and then the effect of correlation delivers the results.

In some cases, we do multiple tests over time to ensure our results are correct. This can happen especially if there is an unusually low results, so we do multiple tests to ensure that we didn’t get a faulty batch or that there wasn’t some other problem. For the ice cream test, I believe it was conducted in a single session and we were confident in the expert verdicts.

We made this video of the tasting panel in action so you check it out yourself. You’ll notice each expert has a bottle of water close by along with palette-cleansing crackers, and a cup for spitting if needed - so indeed very similar to wine taster.


I wonder if it is worth adding a weblink to future reviews/tests where a video is taken, so those interested can view. Would be a one liner on the review webpage with the link?

Video doesn’t need to be fancy or have any commentary.


And there in lies a further challenge!

If you mix nuts with herbs and a little olive oil to grind them in a pestle it is a pesto!
But if you grind just the nuts what is it? The same as any other ground ingredient. You get a paste!
If you put whole nuts into a vessel and churn for a long time, what do you get? Mostly a sore arm, and possibly a broken paddle! :roll_eyes:

Fabio had that one right. It’s really not butter! So why do we call it peanut butter? And what are the taste testers assessing? How closely it tastes like butter? Or how closely it tastes like peanuts? :thinking:


Peanut Butter was always the term I heard US based people using, when I was growing up it was Peanut Paste here in Australia and was sold as such but just like patty cakes have now become cup cakes the non Australian terms have filtered into use in our language…so now we have nut butters rather than nut pastes…


Except for Nutella which refuses to be labelled.
It makes no claims to be anything, neither as a butter or paste.

Noted one of the prime ingredients is palm oil, and in the USA it is referred to as a spread by Nutella.

It opens another challenge. Is it possible to objectively compare similar products such as peanut paste/butter and Nutella or similar, between brands and styles? Or does personal preference become a key factor?

1 Like