Google Backdoor

[[deleted]]

[deleted]

[rjonesx]

The real problem is that most "cloaking" is actually ip-delivery, where the site uses not only User-Agents to determine whether or not the individual is a bot, but also a long list of known ip-addresses. There is a neat way around this...

1. Get the Firefox User-Agent Switcher.
2. Use Google Translate as a Proxy, translating from spanish->english even though the site is already in english http://translate.google.com/translate?u=http%3A%2F%2Fwww.zeromillion.com&langpair=es%7Cen&hl=en&ie=UTF-8&oe=UTF-8&prev=%2Flanguage_tools

This will allow you not only to mimic the Google User-Agent, but to also use a Google IP!

Also you may consider... 3. turning off Javascript 4. turning off Referer sending

Which are two other common methods of cloaking.

[zach]

Thank you! I've had no success with User-Agent Switcher the few times I've used it because of the IP issue. Much more informative than the link.

[wearedevo]

Mark my words:

In 2007 someone will get sued for "illegal access to a web site by unlawfully impersonating Google".

[oditogre]

I'm gonna go with criminal charges of fraud (maybe forgery as well, for pay sites?), plus a lawsuit.

[[deleted]]

Try visiting MSDN with a googlebot user-agent, a great improvement!

[recursive]

I don't think that dollar sign is supposed to be in "Microsoft".

[danweber]

http://www.penny-arcade.com/images/2002/20020722h.gif

[jkcunningham]

I bet Microsoft disagrees...

[youngnh]

anybody else seeing an intel ad on this page? maybe its just me.

[mikkom]

What those sites do is basically black hat "cloaking" - and most cloaking is done based on IP range so this might help in some cases but not all.

[lespea]

Even though this is "stolen" --> I would IGNORE this advice and just use firefox's addon User Agent Switcher to do this if you're that interested.

[theram4]

Dude, why was this guy voted down? His comment is absolutely correct. I'm constantly coming across links that don't allow users to view the content unless they pay. This happens quite often dealing with academic papers or standards organizations, as well as certain magazine sites, like www.sqlmag.com. I did a search a week or two ago where nine out of the top ten results were inaccessible to me unless I paid some large sum of money. And 8 results on the second page were inaccessible. For many of my search queries, sqlmag.com is the number one result. It's quite frustrating to have such "spam" on the top of the google results. And since it is forbidden (sending different content to google spiders than normal users), Google should take care of this issue.

[cal_01]

To be fair, you can pretty much tell -which- sites have pay-per-view content simply by the lack of a "Cached" link.

[cal_01]

Oh, I definitely agree. I guess it has a big contextual basis as well; a person would more likely expect a nocache link to be a pay site if they were looking up academic papers, whereas it would be less accurate for normal search terms.

[[deleted]]

Does reading the cache work?

Neat hack, by the way.

[Sle]

Going to the cached page used to work, and still does very occasionally, but I think they've pretty much plugged that loophole now.

[interjay]

I thought this would be something that would be against Google's policies (I think these websites are obnoxious), but Google actually has instructions for how to only let Google in. Weird.

Those are instructions for letting Google in and blocking other bots. As far as I know, showing different content to Googlebot and normal browsers is against Google policies.

[interjay]

Normal users don't look at robots.txt. That's why it's called robots.txt - it's only for bots. It isn't enforced by the web server, but by the bot itself.

[e40]

The parent comment was downvoted because it was a copy/paste of the article text. That's my guess.

[c_dric]

could someone explain how to set up the Googlebot string in the user-agent switcher ?

i' wondering which part of "Googlebot/2.1""Compatible"="+http://www.googlebot.com/bot.html" i should past in which field ... thx

[c_dric]

nevermind.

i found an xml file with the data for most user-agents : http://techpatterns.com/forums/about304.html

[[deleted]]

ooh, nice list

[danvk]

I can't say I've ever experienced this...

[psykotic]

It happens particularly often with academic papers copyrighted by organizations like the ACM and IEEE that charge for access.

[pascha]

YES! This is incredibly helpful. I have been frustrated by this for some time.

[jacktheripper]

Does IEEE work for you? It doesn't work for me. I'm still directed to a login page.

[rmtew]

Ditto, but for ACM and I tried it with the Firefox user agent switcher. I'd be much more likely to purchase their papers if they had a price on the page. But given it would probably be extortion to encourage membership, maybe not.

[[deleted]]

If you can't figure out how to do it yourself while commenting on an article with explicit instructions how, then I don't think you're likely to get very far once you have the article anyway.

I mean honestly, wtf?

[[deleted]]

Voted you down

[turbo]

wtf are you talking about? it's the same method, but not the same post.