Fazl not happy as PM ignored his men