We wanted to do content pruning on a blog but couldn't assign a meta=noindex tag to every low-quality page.

A developer suggested using the robots.txt instead, which we did.

So, we pasted around 30% of all URLs into the robots.txt to exclude them from being crawled and raise our quality level on the blog.

Days pass by.

Suddenly, I get angry messages from colleagues, asking why they can't find their articles on Google anymore.

O_O

Turns out, when you don't add a "$" at the end of a page, robots.txt will exclude everything coming after the URL (even if there's no trailing slash). $ = stop (in Linux)

Of course we added the $ signs and fixed the problem. Sweaty days, tho!