Firstly, it has to work well as a phone. It can be the greatest multimedia player under the sun, but if it's lousy as a phone, then I walk away.
Next, I detest bloatware. I don't care whether it is put there by a manufacturer or a network provider, I hate it. For this reason, my last two purchases have been Google Nexus handsets. I also like the fact that they get rapid updates of android - I'm not left at the whim of the handset manufacturer or network provider.
It has to feel well-made and solid. This doesn't mean that the case has to be made of metal. Some plastic cases are very sturdy. Others, however, feel flimsy and cheap, and when I'm spending hundreds of dollars on a phone, I expect better.
I like to have some control over the look and feel of the user interface. iOS is very limited in the changes you can make, so for this reason, I prefer android. Windows Phone OS strikes me as trying too hard to look like a tablet or desktop (perhaps it is vice versa). I'm disappointed that Pixel seems to spell the end of the Nexus line. Difficult to know which way to jump when my handset next needs replacing.
I'm very happy with my current Nexus 5X, and was happy with my earlier Nexus. Prior to that, I had a couple of HTC phones - I really liked their build-quality.