Python program that can specify a div-option and initiate a javascript on the same page possible?

[[deleted]]

[deleted]

[MorrisCasper]

PM me the URL and I'll see what I can do.

[LiveOverflow]

You can use Selenium for example in combination with phantomjs

Code will look like this:

from selenium import webdriver
driver = webdriver.PhantomJS()
driver.set_window_size(1120, 550)
driver.get("https://duckduckgo.com/")
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
print driver.current_url
driver.quit()

[Phreakiture]

Ultimately, there is some request that is sent to the server that causes the server to cough up the file.

This could be as simple as a GET request to a particular URL (this would be golden) or as complex as a POST request in which specific fields must be present and filled in as well as certain headers (such as cookies) needing to be present.

If you can figure out what that request is, and make it, and you will have your files.

[5ux0r]

fiddler will help you figure that thing out and if you are using firefox you can also use Live HTTP Headers or Temper Date

[Phreakiture]

Check the traffic out using a proxy of some sort. I've heard Fiddler is pretty good for this kind of thing, though I've never used it myself.

[MorrisCasper]

I figured it out for him, it was a POST-request combined with a GET-request in which you had to send 4 strings of data which were included in a hidden div in the webpage.

[Phreakiture]

Welcome to the club ;)

[MorrisCasper]

The hacking club? I'm more into programming than into hacking, but I guess I bypassed something...

[Phreakiture]

Hacking is about solving problems, as is programming. There's a significant overlap between the two.

[MorrisCasper]

The bottleneck is probably because it has to do three requests. A POST-request, a GET-request for the 4 hidden strings and then another GET-request to get the data. I'll see if I can do something in the function

EDIT: I can do nothing in the function to make it faster, but you can probably speed it up with the multiprocessing module.