It actually is really simple. If you're born with XY chromosomes, you're a man. If you have XY chromosomes and ambiguous genitalia that can look like a vagina, then you're a man with a Disorder of Sex Development (DSD).
Men with DSDs will often have elevated testosterone levels because they also have testicles, sometimes underdeveloped and/or undescended. That's not what makes them a man though, it's their XY chromosomes.
That gender is at least partially a social construct and even if you have XY chromosomes and you've been brought up as a woman you shouldn't be forced to live as a man in case you do a chromosomal test and it turns out you have XY chromosomes.
Imagine the opposite - you're an XX man with a penis and you've lived as a man your whole life and then you do a random chromosomal test and somebody tells you "you see I know you have a penis and I know you're fine being a man but your chromosomes says you're "female" so we have to chop off your dick and you have to grow titties so you can fit into the binaries we've established because it's too hard for others to believe you're a man"
The idea that the use of the term "Man" is at all related to chromosomes, as opposed to how you interact with society. If you were talking about biological sex that'd be a cogent view, but even there there's a ton of nuance.
More important (but still not all-deciding) than the XY phenotype is the presence or absence of SRY, which depending on mutations and faulty crossovers can be present on either X or Y
To oversimplify, every human has an X chromosome and it is necessary to survive. The Y chromosome is optional and codes for some additional traits that make a person male. Chromosomes aren't foolproof and genes and in some cases entire chromosomes can essentially be turned off. However, since we have two of each chromosome this is often not a problem. So, if parts of the Y chromosome turn off, you potentially turn off the mechanisms that make a person male. In which case the person would remain the default of female.
Here's a series of Tweets from 5 years ago by @RebeccaRHelm.
Friendly neighborhood biologist here. I see a lot of people are talking about biological sexes and gender right now. Lots of folks make biological sex sex seem really simple. Well, since it’s so simple, let’s find the biological roots, shall we? Let’s talk about sex...[a thread]
If you know a bit about biology you will probably say that biological sex is caused by chromosomes, XX and you’re female, XY and you’re male. This is “chromosomal sex” but is it “biological sex”? Well...
Turns out there is only ONE GENE on the Y chromosome that really matters to sex. It’s called the SRY gene. During human embryonic development the SRY protein turns on male-associated genes. Having an SRY gene makes you “genetically male”. But is this “biological sex”?
Sometimes that SRY gene pops off the Y chromosome and over to an X chromosome. Surprise! So now you’ve got an X with an SRY and a Y without an SRY. What does this mean?
A Y with no SRY means physically you’re female, chromosomally you’re male (XY) and genetically you’re female (no SRY). An X with an SRY means you’re physically male, chromsomally female (XX) and genetically male (SRY). But biological sex is simple! There must be another answer...
Sex-related genes ultimately turn on hormones in specifics areas on the body, and reception of those hormones by cells throughout the body. Is this the root of “biological sex”??
“Hormonal male” means you produce ‘normal’ levels of male-associated hormones. Except some percentage of females will have higher levels of ‘male’ hormones than some percentage of males. Ditto ditto ‘female’ hormones. And...
...if you’re developing, your body may not produce enough hormones for your genetic sex. Leading you to be genetically male or female, chromosomally male or female, hormonally non-binary, and physically non-binary. Well, except cells have something to say about this...
Maybe cells are the answer to “biological sex”?? Right?? Cells have receptors that “hear” the signal from sex hormones. But sometimes those receptors don’t work. Like a mobile phone that’s on “do not disturb’. Call and cell, they will not answer.
What does this all mean?
It means you may be genetically male or female, chromosomally male or female, hormonally male/female/non-binary, with cells that may or may not hear the male/female/non-binary call, and all this leading to a body that can be male/non-binary/female.
Try out some combinations for yourself. Notice how confusing it gets? Can you point to what the absolute cause of biological sex is? Is it fair to judge people by it?
Of course you could try appealing to the numbers. “Most people are either male or female” you say. Except that as a biologist professor I will tell you...
The reason I don’t have my students look at their own chromosome in class is because people could learn that their chromosomal sex doesn’t match their physical sex, and learning that in the middle of a 10-point assignment is JUST NOT THE TIME.
Biological sex is complicated. Before you discriminate against someone on the basis of “biological sex” & identity, ask yourself: have you seen YOUR chromosomes? Do you know the genes of the people you love? The hormones of the people you work with? The state of their cells?
Since the answer will obviously be no, please be kind, respect people’s right to tell you who they are, and remember that you don’t have all the answers. Again: biology is complicated. Kindness and respect don’t have to be.'
Note: Biological classifications exist. XX, XY, XXY XXYY and all manner of variation which is why sex isn't classified as binary. You can't have a binary classification system with more than two configurations even if two of those configurations are more common than others.