Elizabeth didn’t want to be linked to Mary-Kate and Ashley