Humans have sexual dimorphism, but it's a cultural thing that women wear skirts and men drink themselves to death instead of talking about their problems (both of these are jokes btw. I have a friend who wore kilts quite often and my mother drank herself to death)
Also, genetics is tricky, there a plenty of examples of people who do not fall into one category or another for these sexually dimorphic traits. There are people who have genetics from both sex, as well and differences in hormones distributions will causes these traits to appear or not appear.
People generally expect you to have an excuse, though. Otherwise, you get your one baby shower (only the first baby!) and anything else is tacky. I think that's why gender reveals caught on so fast. The parents and, maybe, grandparents are grasping at excuses to celebrate. And why people find them so cringe in general. Everyone else is a bit annoyed you're getting more than your one party.
The anthropologist might see skeletal differences but they'd also pay attention to the manner in which the subject was buried or what possessions survived with them that could also serve as clues of the subject's identity in life.