At the moment I’m considering a new dishwasher. The Choice comparison shows some models as ‘Recommended’ and others that are not - which is exactly what Im looking for - except the criteria for a recommendation is:
We recommend dishwashers which score at least an 80% CHOICE Expert Rating, at least 75% washing performance score, at least 80% drying score and no other scores below 60%.
As an example. the BOSCH SMS66Mi02A has a XX% ‘Expert’ rating, but meets or exceeds the other criteria for a ‘Recommended’ score. Given that the ‘Expert’ rating is a combination of the washing, drying and energy efficiency scores - which the machine seems to pass - how come it can pass the requirements yet fail them at the same time?
I’ll hope @airedale does not need to correct me but, 'and’s added for clarity
As member content I have changed the model and rating to XX%.
The XX% Expert Rating is below the 80% threshold and that would explain it. The expert rating is comprised of washing, drying, and energy so in aggregate it misses the threshold.
No that doesn’t explain anything - except you can alter others posts.
What I’m asking is why the particular dishwasher (which you have ‘helpfully’ de-identified), and numerous other appliances that seem to meet the criteria for doing ‘what is advertised on the box’ fail another criteria which seems to be some sort of “Expert Opinion” rating without any context why that would be the case. If other appliances do it better, or faster or more economically, wouldn’t it be an idea to tell those that subscribe? Who knows, something that might be considered to be of upmost importance according the Expert could be the last think I would ever need/want.
There needs to be more information and less tables. Years ago, Choice used to actually discuss why certain tested items were preferred over others. Sadly I really haven’t seen a discussion other than a few dot points for some time. Choice is poorer for it and as such there is nowhere near as much reason to subscribe to Choice over other product review sites.
There does seem to be some discrepancies within the dishwasher review, where some dishwasher which don’t meet the criteria are recommended, while others which similar scores aren’t recommended.
It might be one for @airedale to answer as he is compiled the information in the review.
I wonder if it is a mash of old reviews and newer reviews, where the recommendation criteria has changed over time. This might explain the discrepancy, and it is the case, possibly a notation on the review page would be useful to provide explanation to the reviews. I know that often Choice choses not to recommend products which may meet the recommendation criteria due to a major issue with the product - a major safe issue is a good example. But such doesn’t apply for your case as it is about mixed recommendations just outside the nominated recommendation criteria.
I understand why Choice doesn’t want results of member only material given out for free but in this case anonymising the data makes it impossible for me to understand the problem much less suggest a solution. Is there no way to have this conversation?
Apologies but I might be having another conversation from that which you intended. The ‘Expert Opinion’ is a weighted composite of 3 metrics. Hence a product with great drying capability could come out ahead of one with very good washing and great energy efficiency, affecting ‘the cut’.
It is a weighted ‘basket of metrics’ and one can reasonably ask how the weighting factors were derived, and I now suspect that was your underlying question. If so, definitely one for @airedale.
In previous explanations there has been a trend to include the ‘scores’ for various things so a consumer can do their own weighting on what is most important to them personally, and in their own comparative manner since individuals do view things differently, and have differing value systems. eg Choice recommendations are not binary quality or usability or performance conclusions, but they are guidelines to start short listing/reviewing.
Hi @philT, thanks for being so active in this one. You’re all partially right, but to clear it up there’s two different things going on here - the Choice Expert Rating, and Recommended models - they’re closely related but not the same thing.
Yes, our Expert Rating is an aggregate of the important scores, weighted in accordance with their importance to the consumer experience. In the case of dishwashers the Expert Rating is made up of washing performance (65%), drying performance (25%) and energy efficiency (10%). We do, as you suggest, include the composite scores so you can conduct your own assessment based on what’s important to you.
Recommended is assessed on other factors, and often contains a clause of no score (not just the ones that make up the Expert Rating) below a certain threshold. In the case of dishwashers, that means we recommend dishwashers which score at least an 80% CHOICE Expert Rating, at least 75% washing performance score, at least 80% drying score and no other scores below 60%.
We do tweak these cutoffs from time to time because as performance across the category improves we see a greater ratio of models making the cut, and it’s not much use to you, our members, if all but one model is recommended. We also want to keep raising the bar on manufacturers so they can’t rest on their laurels and see they need to continuously improve.
In the case of dishwashers, it’s the ‘no score below 60%’ that’s causing the confusion - there’s a couple of dishwashers which appear to meet all the requirements to be recommended, but they have one score below that 60% threshold (I know what you’re talking about - it’s an Asko model and it scored under 60% for energy efficiency, making it ineligible). That’s not to say it’s a bad machine, and it’s a close call but we do have to draw a line somewhere. And of course if energy efficiency isn’t a deal breaker for you then you can look at the other scores more closely, because it does perform very well. And regardless of energy efficiency scores, it’s still way more energy efficient than hand washing but I digress.
Anyways, when we do adjust the calculations for the Choice Expert Rating or for Recommended we go back and re-calculate the scores for previously tested models (because it’s a formula not an absolute score we can do that without needing to retest) and update accordingly. There’s some isolated products where old and new test results aren’t comparable - for example lawnmowers - because differing test sites and conditions between years, but that does not apply for lab tested appliances because lab conditions are controlled to the extent that we’re confident a test conducted 20 years ago is directly comparable to a test conducted today.
Sorry for the long winded reply, but I hope this explains the confusion and makes things clearer, not less clear. Let me know if you need me to go into more detail on anything.
As the person who asked the question (only to be virtually beaten around the head by those that can edit my posts for daring to question the system), your ‘long winded’ reply goes some of the way towards answering the question and is appreciated.
(I know what you’re talking about - it’s an Asko model and it scored under 60% for energy efficiency, making it ineligible). - No I’m not talking about the Asko, I’m talking about a machine that he who edits others posts deleted the model number and ratings from the original post (hint, it’s 10th on the listing) and the ratings for all categories would place it as a recommended model. While it does seem to have a larger number of negatives than others, it would be good to actually spell it out when writing reviews, such as “while meeting the criteria for a 'Recommended” model, we noted that the machine design didn’t include pink unicorns and glittery rainbows that are common across the other dishwashers we tested. We believe/Our surveys show these features are useful because . . ." rather than assume that the reader ‘knows the system’, which I didn’t until you posted (and I’ve been a subscriber for 20+ years).
The criticism isn’t directed solely at this article, its pretty well across the board and was noticed when I recently bought a microwave and researched buying a vacuum cleaner. If the negatives that tip the balance are disclosed for all to see, the consumer can determine if the negative outweighs other criteria and the importer can determine if the ‘failure’ can be addressed immediately or at the next model change - assisting their continual improvement program.
You are being a little unfair about that. Having watched the moderators deal with posts that question or criticise Choice reports and deal with copy/paste problems that break the paywall for several years I do not see there is any connection.
The reason that copied details of paywalled reviews are edited have nothing to do with the content of the post or who is posting. As I mentioned upthread in this case that has produced the undesired side effect of making your material very hard to follow, so the rule has been relaxed.
I am neither a moderator nor a Choice employee but my perspective is that Choice does pretty well listening to and accepting criticism. I have dished some out myself from time to time and it has been taken in good faith and I have never been beaten about the head. I suggest that you consider giving @PhilT an apology for attributing bad motives to his action that are not the case.
Hi @mcziel2, I’m sorry about the frustrating thread, and I’m sorry I jumped to the wrong conclusion by mentioning Asko - we get a lot of questions about why that one’s not recommended so I leapt before I looked. Thankyou also for being with us for such a long time.
Going back to your original post, the Bosch SMS66MI02A is not recommended because it’s Choice Expert Rating is below the 80% threshold. As for why it’s expert rating is lower than you might expect, it’s because the scores that go into it are weighted, and weighted heavily towards washing performance as washing performance is the most important factor for most dishwasher users. This dishwasher performed very well for drying performance, but that’s only 25% of the expert rating, and it had a less than stellar energy efficiency score (as an aside, drying performance is often inversely proportionate to energy efficiency as the more heat the better the drying, but the more energy used).
It’s also worth noting that our scoring is completely objective - we only factor in measurable performance, which is what sets Choice apart from the other review sites - we actually test the products in a laboratory. Other factors such as loading patterns, special programs or glittery unicorns don’t factor in to the scores as their value is subjective and may or may not be important to different consumers. We do list features, and subjective good and bad points about each appliance so you can make your own judgement as to whether they’re a must have or not, but they don’t count towards the scores - that’s strictly on performance.
Hi @airedale, I believe that @mcziel2 may also be asking why this particular model, which was under the 80% threshold was not recommended, but there are four other models with expert ratings of 79%, 78%, 78% and 76% which are on the recommended list and the test results indicate they may be similar (no better than) in some ways to the model in question.
This creates some discrepancies within the review table which can lead to confusion/concerns like that experienced by @mcziel2 .
This dishwasher performed very well for drying performance, but that’s only 25% of the expert rating, and it had a less than stellar energy efficiency score (as an aside, drying performance is often inversely proportionate to energy efficiency as the more heat the better the drying, but the more energy used).
Thank you - but as noted in the post below yours, the system does seem to be somewhat counterintuitive. It would have been cleared up if there was text in the report to discuss.
Effectively to get value from Choice you need a magazine and digital membership - which I have noted previously is an interesting decision by an organisation that is supposed to be upholding consumer rights. Choice also has no problem with information going outside the paywall for a fee and (last time I was silly enough to click on the ‘buy now’ link) recommend that I pay around $400 extra for an appliance - see here Choice 'buy online' "service" - #20 by grahroll. Is it any wonder I’m skeptical of ‘the system’?
I have only the digital version and it suits me fine. If you think you need both that’s up to you but I don’t see why you would complain about having to pay for both. Both cost money to produce and Choice needs income to support itself.
What decision is it that is interesting? If you are suggesting that Choice is charging you twice for the same thing I fail to see it as I am only charged once. The content may be shared but there are separate costs in presenting it in two formats; hence two charges.
If they make back something in another way why is that wrong? Are you suggesting the paywall ought to be abolished or allowed to be permeable without charge?
You are entitled to your scepticism but I still don’t see how you have been wronged in either the way the paywall works or over your subscription. No doubt Choice can do with improvement here and there but I don’t see that you are providing evidence of any wrongdoings in those matters and so you are not assisting in any improvement.
We are getting way from the assessment of product recommendations into all things wrong with Choice. Maybe that needs another thread.
Hi @phb and @mcziel2 , that’s a good pickup and I think I’ve gotten to the source of the confusion - it’s me - I’ve inadvertently flagged a couple of dishwashers from our latest test as recommended when they shouldn’t have been as they scored under the 80% expert rating threshold - I’m very sorry for the confusion and angst I’ve caused, and I’ve now updated our website with the correct data.
Going forward I’ll review our processes and put steps in place to ensure this doesn’t happen again.
Once again, I’m very sorry for the mistake and the confusion I’ve obviously caused. I do hope that this makes things a little clearer though.
For the future, it might also be worth providing a statement in the review header on how a product comes onto a recommended list…and that some which may have a high expert score may not necessarily be recommended for a particular reason. It would also be useful if the reason why a particular product with a high expert score wasn’t recommended in the detailed product page. Currently such products only get a red cross and one as to try and work out why. This would add great value to those perusing the reviews and using them as part of their purchase decision making process.
If one goes to the comparison tables there is a ‘?’ that explains it (this from dishwashers)
@airedale, For consistency perhaps adding a ‘?’ on the list entries to reflect the descriptions of other metrics would be appropriate? Also less likely to get missed by those not reading everything.
