Elizabeth Olsen recalls getting panic attacks every hour on the streets of New York