No direct link between prime minister and London flats: SC