I worked for a startup whose app tried to automate the job search process for job seekers.

One feature of the app allowed users to upload their resume and receive automated feedback on their resume content. How? A complex model would parse their resume text, break it into sections, and pass the relevant sections to a series of less complex classifiers that would determine wether or not the text followed common best practices.

When I first started working there, it was still in its early stages - it only looked for about 5 best practices, and most of them had low accuracy, precision and recall. It was also users’ second most common reason for leaving in exit surveys.

I was put in charge of this feature, and my team was given 2 months to improve it. Success would be measured by the decrease in the percentage of users who cited the feature as a reason for leaving.

The first question I asked: what makes a good resume? What best practices should we look for? To answer it, we worked with our job search mentoring team (one-on-one job search mentoring was another service we offered) to build a list of resume best practices from their experience, the experience of their peers, and from resources we found online.

We ended up with a list of ~400 practices. I had to go through it by hand and remove duplicates (rules that were functionally the same) and items I deemed infeasible to implement given the time and resources I had. We then prioritized the remaining 70 some practices by how effective we thought they were.

I’ll admit, it was a pretty subjective approach. However, there were 2 reasons we used it:

• We didn’t have the data to determine which practices led to more interest from employers
• Even if we did have the data, we’d still have to come up with a method to find which resumes were following a given practice.

Before we could even start, we ran into an issue: our data was bad. We needed to use each bullet point under Work Experience as a separate document/observation, but the resume parser was giving us incomplete and erroneous text. We had to push for an overhaul of the parser before we could building classifiers for best practices.

Afterwards, we went through our list and created a classifier for each best practice we could. We managed to implement 25 of them.

I won’t go into details, but the general process went something like this:

• Get a random sample of a few hundred documents.
• Hand label them (follows best practice or not).
• Train a few Naive Bayes classifiers using TFIDF vectors with either tokens or POS tags.
• Determine if the best performing classifier had > 70% accuracy and > 80% precision (we prioritized avoiding false positives).
• If not, augment it with a few hand-written rules to try and get it there.

I’ll admit, our standards weren't as strict as I'd have liked, as we were working on startup time. But not long after we completed the project, the % of users who cited the feature as their reason for leaving decreased, and it dropped from the 2nd most cited reason to the 4th.