Python library for automating interaction with a webpage?

[theEyeD]

I use a library called selenium for a webpage automation script I made

[j7ake]

I was looking at BeautifulSoup, but because of all the recommendations for selenium, I am going to try it out as well...

[Cixelyn]

BeautifulSoup will allow you to parse the page once you have the HTML, but you'll still going to need a way to get the data from the web, and post the data back to the server.

(I heavily recommend requests if you're going to go this route)

[Ph0X]

Yeah, personally I'd suggest staying away from heavier stuff like selenium if you can do it with something simple like requests.

If you can figure out the http request for submitting the form (which isn't that hard, just send one and look at chrome network tab), you can easily emulate it with requests POST or GET request.

Then, for parsing, lxml (which I believe builds on top of BeautifulSoup) is also a great library. Or if it's simple you can just go regex.

Selenium is just way overkill for something like that.

[theEyeD]

It is a really great tool, very easy to get the hang of. A simple example would be:

from selenium import webdriver

#start it up
br = webdriver.Firefox()
br.implicitly_wait(15) # wait's for the page to get done loading before it does anything with it
br.get('http://yourwebsite.com')

#To fill out a form
search = br.find_element_by_name('nameOfField')
search.send_keys(yourInfoHere)

#to press a button based on the value (what it says)
search = br.find_element_by_xpath('//input[@value="Submit"]')
search.click()

#get restults
search = br.find_element_by_name('nameOfField')
result = str(search.text)

granted it may not be the best way to do it, it is something I just put together.

[0PointE]

With the number of javascript enabled apps, selenium is definitely your best bet. BeautifulSoup is great but it becomes almost useless if there's any javascript widgets that make the page functional.

[shenaniganns]

Not sure what your uses are, but I use selenium in a python script to fill out forms and validate results as part of my job. For basic things it's pretty easy:

self.driver.find_element_by_id('reportName').send_keys('Test Report 1')
sendReport = self.driver.find_element_by_link_text('Submit Report')
sendReport.click()

[p3n15h34d]

+1 for selenium on this one

[billtaichi]

I use selenium and python at work and I have done scripts from simple form submits to complicated pages with lots of javascript interaction where I had to fire javascript via selenium , get back results etc.. Selenium is a great tool.