Dwindling relationship kept Obama from travelling to Pakistan: White House