Defiant Abe hits back over scandal as support plunges