Abuse of trans women is a rampant problem in this country and, as you say, it is often perpetrated by the very same groups who hypocritically claim trans women need to be stripped of public accommodation for the safety of cis women.
That is what we must change.