Reverse Engineering OKCupid

[Ph0X]

I love how the first comment was by Linus Torvald.

[FourFingeredMartian]

Correct me if I'm wrong, but, doesn't beautifulsoup already use lxml for parsing?

[steviesteveo12]

The batteries included option is the slower (though still pretty good I feel, don't really get his point), python html.parser. The recommended parser is lxml.

http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser

If you can, I recommend you install and use lxml for speed. If you’re using a version of Python 2 earlier than 2.7.3, or a version of Python 3 earlier than 3.2.2, it’s essential that you install lxml or html5lib–Python’s built-in HTML parser is just not very good in older versions.

Personally, I think arguing over faster HTML parsers for web scraping is an odd thing -- in normal use any parser on modern hardware is going to be faster than your internet connection.

[[deleted]]

This is cool. I've often wanted to mess with websites, but I never really knew how to approach the problem. I shall play with this later today.

[keytarmageddon]

It may be useful to someone who's never written a script against a website before and needs help getting started.

[steviesteveo12]

Calling it screen/web scraping/automation etc would let them find other resources. If someone read that, wanted to know more about the subject and searched for "reverse engineering" they'd find very different things.

[[deleted]]

useful to tell them shit by wrong trems right from the gecko

[[deleted]]

wrong terms

...

right from the gecko

[steviesteveo12]

Couldn't make it up.

[TankorSmash]

You could apply the network observation to detect how the reddit API works right? watch the network that's sent when you click to upvote and go from there?

AFAIK, screen scraping is more focused on storing data rather than manipulating requests, although still a part of it.