The numbers show EU workers are deserting us already — after Brexit, we’ll be in deep trouble