If the bankers leave London, it won’t just be the city that will suffer – it’ll be Brexit voters who pushed them out