Tolerance is not a moral precept, it is a peace treaty. When peace treaties are broken, no party has an obligation to honor them until peace is restored.
If tolerance of Idea X does not further that goal, we have no moral obligation to tolerate Idea X being spoken in public.
It's fine to kick someone who says that out of restaurants, because letting them stay does worse harm to black people or anyone with black friends or family.