Not part of govt as country ‘not heading in right direction’: Saad Rafique