British PM ‘irritated’ by leadership speculation