I made this mistake. I was asked why, and I responded.
I explained in a straight up but not rude manner that the unfollow was simply due to uncommon interest, lack of interaction (likes/rt’s) between us and the general (but not always) negativity in their posts.