For months now the predictions have been that the disaster of Brexit will destroy the Conservative party.