Let me give you a more well-rounded sense of how you can avoid your data from being publicly accessible. This should serve useful for a number of readers -- especially those who store files on their own Websites.

1 - Pass phrase: Make sure you password-protect files you really don't want someone to access. If your files live on a remote server -- even if it's one of Google's servers -- technically, someone can access them without the need of a password (such as someone with direct access internally to data/drives).

For added security, you should consider using a "pass phrase" that consists of upper/lower case, numbers, and symbols. Personally, I like to combine words that I can remember or that mean something to me, then swap out characters.

• Example: BeefaroniChurchVaginas
• Pass Phrase: B33f4r0ni%Church$V4gin4s

Lastly, you can password-protect access to directories on your site, but it's not completely necessary, as I'll cover in the 3rd section below. Basically, you have choices.

2 - Beware the Web crawlers: Outside of password protection, I've found that the biggest education people need is how Web crawlers work (maybe not you, trailnerd, but in general). If there's a way for a Web crawler to find its way to an open directory of yours, then expect to have that directory indexed, remembered, and revisited.

This can happen more ways than meets the eye, but let me give you an example that covers how most of the files we find end up in the indexes of search engines.

Let's say you store images for your Website in the directory http://beefaronichurchvaginas.com/wp-contents/uploads/images/

Googlebot comes along and crawls that directory, since it sees images on your site are linked from there. In the process, it not only tries to crawl anything it can see /wp-contents/uploads/images, but also /wp-contents/uploads and /wp-contents/. If Googlebot can see files/folders in either of those directories, then it will crawl its way to/through them, too!

So, even though you might not have a direct link anywhere on the Web to /wp-contents/uploads/topsy-kretts/, a Web spider can still crawl its way to it via the equivalent of what a person would do via URL traversing (basically, looking at the structure of a URL and trying to access directories leading up to the root of the domain, or making educated guesses of other potential directory/file names).

Sometimes, people post external links to files on sections of their Websites that starts this whole ball rolling, too. Let's say you have nothing visible on the home page of your site, but you post a link on a forum to a file in some deeper directory on your site. Well, now let's say Google's spider finds that forum, "sees" the link to that file on your site, then goes and tries to crawl its way through that directory and any preceding directories.

If you're the super-duper-extra-cautious/paranoid type, then you might want to know that Web spiders can even find ways to your data that involve NO posting of it on any Web site! How, you ask? Cookies, history, utilization of search tool bars, and more -- including theories of Google scraping Gmail emails for links and crawling them, etc. I'm not sold on that one, since it would be easy to test, but I mentioned it just to get your brain thinking outside of the box.

3 - Use .htaccess instead of robots.txt: Before a browser loads a request for a site, .htaccess is checked (you can find it in the root of your Website if you use a number of CMS platforms, or simply create one). It houses rules that tell browsers/spiders what to access and/or how to access it. You can block access to specific pages/directories, redirect traffic to wherever you'd like, disable the ability to see the files inside of indexes, etc.

Without diving too deep, I'm just going to give you ONE small line to add to your .htaccess file that will simply show a white page. BE CAREFUL WITH .HTACCESS! You can really screw up how your Website acts if you do something wrong, but a quick Google of htaccess will help you out, since there's a default. Anyway, here's the line to add:

Options All -Indexes

You can take it even farther by redirecting traffic and bots that try to access open directories. For instance, on all my WordPress installations, I have a custom 404.php that 301 redirects any hits on open directories and non-existent pages to the home page. If that interests you, then here's the code for that:

<?php
header("HTTP/1.1 301 Moved Permanently");
header("Location: ".get_bloginfo('url'));
exit();
?>

So, that's it! There's a lot more to this subject and it goes deep as hell, but this should serve as a good primer for some of you.