As long as you buy the right size it shouldn't be an issue, most clothing in standardized sizes aren't much different between the genders besides the number being different (a men's size S will be a women's size M) so as long as you get the size right it'll be fine.
Edit: Size comparison flipped the wrong way, fixed it.
Yeah I just realized my mistake of getting them backwards, I just fixed it. Glad I was still able to make the point even though I made an embarrassing mistake 😅
I have a nice Pikachu hoodie which said it's for women on the site I ordered it from but I didn't care because it's cute and I liked it. The only thing that's mildly annoying is having to think about the sizing difference (which I guess is kind of the point of the separate sizing numbers, just to be that extra bit of annoying for people who want it anyway).