CHOICE membership

Choice high end camera testing

I’ve been a Choice member for decades, and rarely have I felt the need to criticise the organisation. However, on the subject of camera tests, I’ve the written to Choice a number of times with no effect - so I’ll try this forum.

Choice’s tests of high-end cameras are basically worthless, and they should stop.

Firstly, the test as a survey of cameras is far from complete. Where are the high-end Canon cameras? If the top of the line Sony and Olympus cameras and the Nikon D850 are tested, so should the Canon 5D.

Secondly, testing semi-pro cameras like the Nikon D850 on criteria appropriate to a point and shoot wastes everyone’s time. It’s like comparing a Ferrari with a Kluger and marking it down because the Ferrari doesn’t tow a caravan as well as the Toyota.

Actual test results are questionable. For example, the test’s Picture Quality Score has the Nikon Z6 rated at 90% while the D850 is rated at 67%. How that result was arrived at, I can’t imagine. Part of the problem might lie in the fact that the lenses used to compare the Nikons aren’t the same or even equivalent, so little can be assumed from image scores.

Choice also needs to be more careful in its basic facts. Comparing three Nikon models (D850, Z6 and Z7), I note that:

  • Lens mount for the Z series is noted as “not applicable”. This would imply that they are not interchangeable lens cameras. In fact, the Nikon Z cameras use (surprise!) Z mount lenses.
  • The Minimum Shutter Speed for all three Nikons is shown as “31/12/7999 1:00:00 PM”???
  • Image ratios supported for the D850 is given as “29/12/1899 4:02:00 PM”. What??

By all means, test cameras. But test consumer cameras with appropriate, consumer criteria and leave testing of pro and enthusiast cameras to those with the resources and the interest to test them appropriately.

9 Likes

I agree with several of your points. I cannot imagine why you would compare such a range of equipment that has such huge scope in abilities and purpose. There is a reason that car reviews are divided into categories such as large luxury, economy compact and off road, the same applies to cameras.

Comparing high end models with interchangeable lenses with single lens models is made more pointless when the choice of lens for the first category has no explanation that I can find. Even if there is some reason for the choice the assessed outcome will be of that particular body/lens combination, so the score does not represent the ‘camera’ ie the body.

Comparing products with many characteristics and features by creating a single score is difficult enough when the group being assessed is fairly homogeneous. We can assume that washing machines will probably be used to wash clothes and the scope of what that means is narrow enough to make comparisons useful. The scope of what these cameras can do and the way that they could be used makes the single score meaningless. Yes I see that there are many details and qualifications provided but the fact that there are so many just emphasises how silly the headline score is.

I think your final para sums it up well.

4 Likes

In the past there were several Australian magazines that specialised in electronics, technology and computer related products. The options have gradually become fewer.

I remember reading across all of them many digital camera reviews. As time progressed the quality of the products and reviews improved. There were long series of test images taken under equivalent settings for direct comparison. Zoom, wide angle, low light, flash, colour accuracy, portrait faces, scenery, etc.

Other aspects of the reviews included user options, ease of operation, through to shutter lag and flash options. Noise under low light, digital artefacts, colour saturation, grey scale or black and white contrast, the list goes on.

I miss that level of hands on testing and comprehensive assessment of performance. There is more to a good digital camera than long feature lists.

If Choice were to reconsider what cameras to thoroughly test, perhaps staying with the higher volume brands and models that most can afford might be wise.
IE Cameras with a price tag of less than $1,000. Models that offer a basic step up from the average mobile phone camera.

For the higher end cameras with interchangeable lenses and semi pro bodies? Perhaps these should be left to the enthusiast photography publications.

Totally agree.

5 Likes

One has to remember that Choice provides information for the layperson, rather than experts. While one may use a camera professionally, semi-professionallyor as an expert/advanced user, the features and usability of a high end camera would be different for those which are used by the casual punter/photographer.

I imagine that Choice does have a dilemma of whether to cater for both the limited number of members which are more professional type photographer and who knows a lot about photography or cameras…, or most of members which are laypeople.

It is a bit like would a mechanic (home or professional) buy a car magazine to find out detailed information on every car (from a cheapy to the most expensive models)…possibly no as they would have access to better information elsewhere.

2 Likes

Which is why, if you are going to reduce the scope, you would cut off the top end and make it easier and more meaningful to compare the rest.

6 Likes

Exactly!

4 Likes

I agree with everything @ShaneB has said. which is why I don’t read choice reviews of cameras. I prefer to get my camera information from other sources, like cameralabs.com, dpreview.com and so on. These places do excellent reviews which are generally unbiased.

TBH I cant even think why Choice would do camera reviews… are there photography experts on the team?

3 Likes

There are a number of “paid for” services that offer market data. Perhaps Choice subscribes or would pay for a basic report that provides data on the volume and value of this market. Enough to know how many Choice readers might find this of interest.

A more general discussion would be how Choice chooses which items or classes of items are chosen for review.

My simple take is that items that are widely adopted products in every day use should always be the priority. The decision to target products that have safety aspects or the value of essential products and services appears to be the current theme.

  • High end cameras are interesting, but a diversion.
  • Hidden speed cameras an impact on a select few.
  • Residential drinking water (sterilisation) and waste water treatment (backyard grey water and black water) might be of interest to many more than those who buy high end cameras and much more of a health safety concern.
  • Not all of us own lawn mowers, while many use a mowing service.
  • Green Energy and carbon impacts of interest and concern for all but a small number of consumers.
  • Food, housing, medical, banking, transport and … related topics are essential for all.

Yes, it is a surprise high end cameras get a run when one of my preferred topics, for those of us not on reticulated services, is yet to get a review.

I do support Choice and that 90+% of the time it is hitting the right targets. For the other 10%, hopefully the feedback from the community helps with selection? :+1:

P.S.
Residential water treatment (roof-tank or bore) has many options. The investment excluding tanks or bores is similar to a bottom of the range high end camera kit. There are a number of alternate solutions.

Residential waste water treatment is a much greater investment, however the consequences of a poor decision are also significant,

Get either wrong. The impacts extend beyond household finances. Delivery, installation and maintenance is tied to our wonderful professional plumbing profession. Something those outside the big cities often have little choice in.

2 Likes

Some info on how Choice selects products, cross linked

1 Like

Thanks TheBBG. I think I kind of got that. The problem is that comparison testing must compare like with like. To continue my auto analogy from my original post: “Wheels” or whoever doesn’t go out and buy and test half a dozen “representative” cars - a super car, an SUV, a people mover, etc. They decide on a category and test representative cars from that group.

I personally think that Choice could continue to test cameras, but they must avoid interchangeable lens cameras (unless they are prepared to use comparable lenses across the entire test group) and maybe place a ceiling on price. For example, test superzoom bridge cameras under AUD1,200 street price.

That said, and as others have commented, unlike almost all the products Choice tests, there are several reputable web sites which produce credible and detailed tests of cameras and lenses. Maybe Choice should put its resources into some other products not recently tested?

2 Likes

One of the problems I have always had with Choice is that sometimes the selection of ‘tested’ products seems almost like a random shotgun. In any test it does beg the question why eg. a $500 ‘consumer’ product would be compared to a $1500 ‘pro’ product, excepting if they used the latter as a data point for comparative excellence. Then the question becomes would a ‘consumer’ ever buy a ‘pro’ product in the category, and vice versa when each group have disparate needs and probably disparate value systems.

You wrote it well in the close of your OP.

2 Likes

I think this is Choice’s dillema as well. There are many novice type consumers which purchase more professional type products, thinking that doing so will produce a better outcome (better photos in the case of cameras).

I suspect Choice tests higher price bracket products with lower price bracket ones to cater for such consumers. Comparisons are possibly based on what is relevant to the layperson type consumer as a result.

2 Likes

I think I get it, but in the case of cameras at least, it’s not working - IMHO.

Compare the results for the three Nikons: D850, Z7 and Z6 with these scores from the highly respected DxO organisation.


On DxO’s Overall Score, the D850 (score of 100) outperforms the Z7 (99)which outperforms the Z6 (95) on all factors except high ISO noise.

Choice rated them 80%, 87% and 88% respectively.

I acknowledge Choice’s good intentions, but I repeat that testing similar cameras with different lenses, and testing high-end cameras on their performance as a point-and-shoot results in results that don’t assist members to make a good selection.

BTW, most images posted to the internet are made with smart phones. These days, a consumer camera test should include results from the cameras on the iPhone and the Samsung at least. Were that to be done, many members might just stick with their phone!

2 Likes