It's because Brexit is not the highest priority for many people.
Go back to the referendum. It was about 50-50, leave and remain.
But neither side of that represented a political unity.
25 percent Labour Brexit
25 percent Tory Brexit
25 percent Labour remain
25 percent Tory remain
While at the same time uniting all the Tory voters on the grounds of being Tory.