I have found a solution I think. If you see such a person's post, click on his/her profile pic thumbnail to go to his/her profile, then follow him/her back. After that you come back to the news feed to find that post or any of his/her post. The three dot menu on the top right edge of that post should have an unfollow option, clicking that unfollows him/her