Choice high end camera testing

I’ve been a Choice member for decades, and rarely have I felt the need to criticise the organisation. However, on the subject of camera tests, I’ve the written to Choice a number of times with no effect - so I’ll try this forum.

Choice’s tests of high-end cameras are basically worthless, and they should stop.

Firstly, the test as a survey of cameras is far from complete. Where are the high-end Canon cameras? If the top of the line Sony and Olympus cameras and the Nikon D850 are tested, so should the Canon 5D.

Secondly, testing semi-pro cameras like the Nikon D850 on criteria appropriate to a point and shoot wastes everyone’s time. It’s like comparing a Ferrari with a Kluger and marking it down because the Ferrari doesn’t tow a caravan as well as the Toyota.

Actual test results are questionable. For example, the test’s Picture Quality Score has the Nikon Z6 rated at 90% while the D850 is rated at 67%. How that result was arrived at, I can’t imagine. Part of the problem might lie in the fact that the lenses used to compare the Nikons aren’t the same or even equivalent, so little can be assumed from image scores.

Choice also needs to be more careful in its basic facts. Comparing three Nikon models (D850, Z6 and Z7), I note that:

  • Lens mount for the Z series is noted as “not applicable”. This would imply that they are not interchangeable lens cameras. In fact, the Nikon Z cameras use (surprise!) Z mount lenses.
  • The Minimum Shutter Speed for all three Nikons is shown as “31/12/7999 1:00:00 PM”???
  • Image ratios supported for the D850 is given as “29/12/1899 4:02:00 PM”. What??

By all means, test cameras. But test consumer cameras with appropriate, consumer criteria and leave testing of pro and enthusiast cameras to those with the resources and the interest to test them appropriately.

9 Likes

I agree with several of your points. I cannot imagine why you would compare such a range of equipment that has such huge scope in abilities and purpose. There is a reason that car reviews are divided into categories such as large luxury, economy compact and off road, the same applies to cameras.

Comparing high end models with interchangeable lenses with single lens models is made more pointless when the choice of lens for the first category has no explanation that I can find. Even if there is some reason for the choice the assessed outcome will be of that particular body/lens combination, so the score does not represent the ‘camera’ ie the body.

Comparing products with many characteristics and features by creating a single score is difficult enough when the group being assessed is fairly homogeneous. We can assume that washing machines will probably be used to wash clothes and the scope of what that means is narrow enough to make comparisons useful. The scope of what these cameras can do and the way that they could be used makes the single score meaningless. Yes I see that there are many details and qualifications provided but the fact that there are so many just emphasises how silly the headline score is.

I think your final para sums it up well.

4 Likes

In the past there were several Australian magazines that specialised in electronics, technology and computer related products. The options have gradually become fewer.

I remember reading across all of them many digital camera reviews. As time progressed the quality of the products and reviews improved. There were long series of test images taken under equivalent settings for direct comparison. Zoom, wide angle, low light, flash, colour accuracy, portrait faces, scenery, etc.

Other aspects of the reviews included user options, ease of operation, through to shutter lag and flash options. Noise under low light, digital artefacts, colour saturation, grey scale or black and white contrast, the list goes on.

I miss that level of hands on testing and comprehensive assessment of performance. There is more to a good digital camera than long feature lists.

If Choice were to reconsider what cameras to thoroughly test, perhaps staying with the higher volume brands and models that most can afford might be wise.
IE Cameras with a price tag of less than $1,000. Models that offer a basic step up from the average mobile phone camera.

For the higher end cameras with interchangeable lenses and semi pro bodies? Perhaps these should be left to the enthusiast photography publications.

Totally agree.

5 Likes

One has to remember that Choice provides information for the layperson, rather than experts. While one may use a camera professionally, semi-professionallyor as an expert/advanced user, the features and usability of a high end camera would be different for those which are used by the casual punter/photographer.

I imagine that Choice does have a dilemma of whether to cater for both the limited number of members which are more professional type photographer and who knows a lot about photography or cameras…, or most of members which are laypeople.

It is a bit like would a mechanic (home or professional) buy a car magazine to find out detailed information on every car (from a cheapy to the most expensive models)…possibly no as they would have access to better information elsewhere.

2 Likes

Which is why, if you are going to reduce the scope, you would cut off the top end and make it easier and more meaningful to compare the rest.

6 Likes

Exactly!

4 Likes

I agree with everything @ShaneB has said. which is why I don’t read choice reviews of cameras. I prefer to get my camera information from other sources, like cameralabs.com, dpreview.com and so on. These places do excellent reviews which are generally unbiased.

TBH I cant even think why Choice would do camera reviews… are there photography experts on the team?

3 Likes

There are a number of “paid for” services that offer market data. Perhaps Choice subscribes or would pay for a basic report that provides data on the volume and value of this market. Enough to know how many Choice readers might find this of interest.

A more general discussion would be how Choice chooses which items or classes of items are chosen for review.

My simple take is that items that are widely adopted products in every day use should always be the priority. The decision to target products that have safety aspects or the value of essential products and services appears to be the current theme.

  • High end cameras are interesting, but a diversion.
  • Hidden speed cameras an impact on a select few.
  • Residential drinking water (sterilisation) and waste water treatment (backyard grey water and black water) might be of interest to many more than those who buy high end cameras and much more of a health safety concern.
  • Not all of us own lawn mowers, while many use a mowing service.
  • Green Energy and carbon impacts of interest and concern for all but a small number of consumers.
  • Food, housing, medical, banking, transport and … related topics are essential for all.

Yes, it is a surprise high end cameras get a run when one of my preferred topics, for those of us not on reticulated services, is yet to get a review.

I do support Choice and that 90+% of the time it is hitting the right targets. For the other 10%, hopefully the feedback from the community helps with selection? :+1:

P.S.
Residential water treatment (roof-tank or bore) has many options. The investment excluding tanks or bores is similar to a bottom of the range high end camera kit. There are a number of alternate solutions.

Residential waste water treatment is a much greater investment, however the consequences of a poor decision are also significant,

Get either wrong. The impacts extend beyond household finances. Delivery, installation and maintenance is tied to our wonderful professional plumbing profession. Something those outside the big cities often have little choice in.

2 Likes

Some info on how Choice selects products, cross linked

1 Like

Thanks TheBBG. I think I kind of got that. The problem is that comparison testing must compare like with like. To continue my auto analogy from my original post: “Wheels” or whoever doesn’t go out and buy and test half a dozen “representative” cars - a super car, an SUV, a people mover, etc. They decide on a category and test representative cars from that group.

I personally think that Choice could continue to test cameras, but they must avoid interchangeable lens cameras (unless they are prepared to use comparable lenses across the entire test group) and maybe place a ceiling on price. For example, test superzoom bridge cameras under AUD1,200 street price.

That said, and as others have commented, unlike almost all the products Choice tests, there are several reputable web sites which produce credible and detailed tests of cameras and lenses. Maybe Choice should put its resources into some other products not recently tested?

2 Likes

One of the problems I have always had with Choice is that sometimes the selection of ‘tested’ products seems almost like a random shotgun. In any test it does beg the question why eg. a $500 ‘consumer’ product would be compared to a $1500 ‘pro’ product, excepting if they used the latter as a data point for comparative excellence. Then the question becomes would a ‘consumer’ ever buy a ‘pro’ product in the category, and vice versa when each group have disparate needs and probably disparate value systems.

You wrote it well in the close of your OP.

2 Likes

I think this is Choice’s dillema as well. There are many novice type consumers which purchase more professional type products, thinking that doing so will produce a better outcome (better photos in the case of cameras).

I suspect Choice tests higher price bracket products with lower price bracket ones to cater for such consumers. Comparisons are possibly based on what is relevant to the layperson type consumer as a result.

2 Likes

I think I get it, but in the case of cameras at least, it’s not working - IMHO.

Compare the results for the three Nikons: D850, Z7 and Z6 with these scores from the highly respected DxO organisation.


On DxO’s Overall Score, the D850 (score of 100) outperforms the Z7 (99)which outperforms the Z6 (95) on all factors except high ISO noise.

Choice rated them 80%, 87% and 88% respectively.

I acknowledge Choice’s good intentions, but I repeat that testing similar cameras with different lenses, and testing high-end cameras on their performance as a point-and-shoot results in results that don’t assist members to make a good selection.

BTW, most images posted to the internet are made with smart phones. These days, a consumer camera test should include results from the cameras on the iPhone and the Samsung at least. Were that to be done, many members might just stick with their phone!

2 Likes

I think this is good point. I’m not an expert and I’m way late to the conversation, but it seems to me that different cameras have different use cases. Much depends on factors like sensor size (apc-s vs. full frame) auto-focus capability, buffer size, ISO performance and the like. As an example the full-frame Canon RP will give a better background blur, low light performance and picture quality than the Panasonic G9 with its 4/3 sensor. However, I wouldn’t take the Canon over the G9 for shooting birds in flight or a sports match for instance - it’s just too slow. But if I wanted to shoot portraits, landscapes and just general photography, then I would go for the Canon RP.

Other more peripheral issues have to be taken into consideration as well. For instance Olympus produces great cameras but the company was taken over by JIP, so there’s no guarantee there will be ongoing support or innovation. Canon probably has the greatest selection of affordable lenses i.e the RF 600mm f/11 IS STM as just one example, and that often wins the day for budding photographers. But hold on there’s even more to take into consideration. According to Tony Northrup in his photgraphy buying guide:

Elsewhere in this book, I’ve done my best to help with the difficult decision every DSLR buyer faces: Canon or Nikon. While I have found a handful of distinct benefits of the Canon infrastructure, in most ways, Nikon meets or exceeds the standard set by the #1 camera manufacturer. You almost can’t go wrong by buying a Nikon.

He goes on:

Nikon DSLRs such as the D500 and D850 have always been our favorite cameras to shoot with; they are reliable and feel great in the hands. Nikon spent the last 40 years in the #2 spot behind Canon, but they always seemed to work a little harder, staying one step ahead of Canon in both image quality and usability. However, the decline of camera sales in the mid-2010’ s has impacted Nikon harder than Canon and Sony, resulting in Nikon falling behind in important technology such as mirrorless autofocus. Nikon sales have fallen to a distant third behind Canon and Sony. In the last several years, Nikon has continued to cut their research and development budget, which makes us worry that Nikon might no longer be a technology leader in the years to come. Still, for those looking for the best DSLR, we recommend Nikon.

You get the impression that the camera industry is a bit of a basket case.

1 Like

Nikon sales have fallen to a distant third behind Canon and Sony. In the last several years, Nikon has continued to cut their research and development budget, which makes us worry that Nikon might no longer be a technology leader in the years to come. Still, for those looking for the best DSLR, we recommend Nikon.

Yes, well … Tony Northrup went off an an anti-Nikon bender in 2021-22 and lost a great deal of credibility.
Even a casual glance at the specs for the Nikon Z9 (a shutterless top of the line camera) and the extraordinary quality of the Z-mount lenses released in the last year would indicated TN lost the plot!

Yeah for sure! Might be why they’ve nicknamed him Sony Tony.

1 Like

Are you reiterating that one can really only compare cameras in a particular class with like?

My first 4 were film cameras starting with a B&W 620 film format. The next 5 digitals with the first 4 chosen simply because they were a long way ahead of any mobile phone camera. Even my current now aging Nikon B700 offers advantages only a more expensive mobile phone might aspire to.

Am I looking for that absolute perfect pic? Not at all. What I do want is a straight forward all in one camera that can point and shoot to record the wildlife on our property. The greatest benefit the high optical zoom. Compromised can be the exposure and sometimes not quite perfect auto focus. Take enough shots and eventually there are those worth sharing. The first objective is simply a reliably record of what wildlife we’ve encouraged to move in or visit. I can’t ever see me choosing a more expensive full body camera and set of interchangeable lenses, then sitting patiently for that one perfect photo. Hence I can see the value of Choice reviewing the more everyday choices.

Is it worth upgrading to a pro camera may be of interest to some? It’s a useful comparison to make.
Whether Sony, Nikon or Canon or another offer a better choice, I can attest across the extended family we have all made different choices.

P.S. added comment
Does the old adage a poor craftsman always blames his tools still apply? Commentary that a camera feels great in the hand is a common exhalation in reviews. Of course that has nothing to do with the quality of the scene or digital capture. Pure puffery given none of us have hands the same.

No, but it is part of the experience in two ways.

A well designed camera that is logical and intuitive makes it easier to learn to control and allows you to forget about the machine and concentrate (I almost said focus) on using it to best effect to grab the image that you want. If you are still fiddling away with the settings the 'roo has hopped away.

Some people revel in the experience of the process, for some the act itself has value as much or more than the resulting image. It is a way to see the world, quite possibly in a different way.

Something like that as I’d think comparing apples with apples would be more useful. I’m unclear on how Choice rates their cameras. But if for example you grouped all cameras together and used a sliding scale to rate them, then I’m unsure as to whether that would be all that informative.

Many share your reasoning and you often see people on the inaturalist app doing just that. Quality is not always the objective, simply documenting wildlife however, is.

They don’t have to be expensive. Lots of people buy second-hand, even the pros, and they save themselves hundreds of dollars. Good bargains are to be had on the eBay and Amazon market place.

Depends what you mean by that. In some cases it probably does. For that reason your best money might be spent on educating yourself to attain better results in photography. Several YouTubers have given an iphone to a professional and a very expensive RED camera to an amateur. Guess who won that competiton. But if you were photographing someone’s wedding and you were using a camera with a single card slot as opposed to having two, and upon attempting to upload the 1000 pictures you took of that particular event to your PC for post processing find that your single card has failed, who is to blame? Right tool for the right job.

2 Likes

It’s an informative diversion from the original topic. In post Christmas lunch recovery mode I can share briefly two observations. One of the visiting family with more than a passing interest in photography has been happy snapping away with a Canon AE1, Ilford AS400 B&W loaded. The holiday snaps of the great outdoors in contrast are being captured with a high end smart phone camera.

A willingness to learn is a pathway to better results, in most things.

IE make the most of what you have.

1 Like