Amid opposition, Japan PM says has ‘never put Olympics first’