Should just call them XX or XY because no matter what surgery or identity they think they have you can't change biology.
But for general usage, look to the chromosomes. I believe the same as you but people will bring up people with genetic abnormalities. Women can also have just an X and no second chromosome.
It's a complex thing, but letting a man in a dress or having women "bind" their chests to pass as male is not the best solution, I feel.