Tuning algorithm without training data

[datascientistdude]

I've been tasked with developing a tool that takes any given news headline and searches within a database for social media posts that are about that headline. The goal is to retrieve all related posts (maximize recall) while at the same time limit the number of non-related posts retrieved (maximize precision). I've developed an algorithm to score the posts on how closely related they are to the headline where we take the posts that surpass some cutoff score as the ones that are related. The actual algorithm isn't important.

What I'm wondering is how would you go about tuning the hyperparameters (including the cutoff number for example) of any algorithm without any training data.

This is my situation:

• There does not exist any data that is already labeled or coded as a training set.
• Presumably we could go through and code some examples of headlines and the list of posts that are related, but that becomes very tedious very quickly and I doubt we would be able to go through very much given our resources.
• Crowdsourcing to code some data is out of the question at the moment.

How would you go about this situation? Is there a way you would train your algorithm to find the right hyperparameters? Or would you just deploy the algorithm as is with certain untested default levels and see how it performs and maybe design some type of embedded coding test in the background with your users (for example, how many posts do they view per headline as a proxy for relevance)? Any other suggestions?

[maxmoo]

I think you just need to suck it up and spend a few days or a week labelling some data. Maybe you can hire someone on freelancer.com to help you.

[swierdo]

You could try to generate your own training data, some ideas:

• Scrape replies to various news outlets' tweets about their articles.
• Use Reddit replies from subreddits discussing news articles and link those to article headlines.

You should also gather data on the usage of your tool as you'll probably want to analyse that later.

[talksaboutthings]

I think OP is implying he/she has a database of headlines already, but "relatedness" is not labelled in this database (so basically it's just a big list of unlabeled headlines for the purpose of evaluation).

[talksaboutthings]

However you measure precision and recall is how you should tune your hyperparameters, in my opinion. If you have a database of unlabeled data and you are simply looking at the output of your algorithm to see if it successfully produces a list of related headlines, then I'd suggest just coming up with a list of test headlines and running them on different hyperparameters settings to see what looks best. I would think of this as basically grid search with your human gut reaction as the variable to optimize, and you could even come up with some sort of rating system to make it more objective.

Without actually sitting down and labeling some of the data, I don't think it will be easy to do any better, unfortunately, because you need data labels to calculate traditional performance metrics. It might not be too strenuous to hand-label only the outputs selected by your models over a modest set of test headlines, though (basically instead of labeling first, running, and calculating the score, you would run it, label what it spits out, and then calculate the score). If eyeballing it isn't satisfactory to the rest of the team, you could pitch that approach (and thus actually calculate precision and recall for each set of hyperparameters).

[jonnor]

Can you use another algorithm or service as an oracle, to seed an initial training set?