To see why, think not about trans people, but about intersex people.
Imagine that you're a kid with complete androgen insensitivity. You are, as far as anyone can tell by looking, a girl.
Really?
(None.)
(None.)
(Side note: have any of the people who worry about bathrooms actually been in a women's bathroom? News flash: THERE ARE STALLS. We do not see one another naked.)