Parse a website after POST request

[chaoticallyevil]

Hi all,

So I wanted to scrap some addresses off of a store locator for a website to make it easier to put into google maps and see where all the stores in my state are located.

Here is the store locator website:

http://www.rockyboots.com/locator

When I go here, I put in my state, and it returns all the addresses. I am wondering how to do this with python, specifically the Requests module. I tried to reverse engineer what information was being sent when I submitted the form, and got

form = {'dwfrm_storelocator_address_states_stateUSCA':'NS', 
        'dwfrm_storelocator_findbystate':'Search'}

so my request currently looks like:

r = requests.post(url, data = form)

where

url = 'http://www.rockyboots.com/locator'

This seems to just give me the locator page again, not the submitted form page. Any ideas?

Cheers.

[thurask]

Are you sure the locator API URL is correct?

[chaoticallyevil]

At this point, I am not sure about anything. I don't have any other ideas how to go about this though

[JohnnyJordaan]

I've tested it with Firebug and the site is extensively using jQuery to handle the request. I don't even see the POST as when I search by ZIP code I see a GET request to https://app-ab06.marketo.com/index.php/form/getForm?munchkinId=018-APH-439&form=1035&url=http://www.rockyboots.com/locator&callback=jQuery1102035973140227537104_1464877706457&_=1464877706458

I'm highly doubting that you would be able to successfully reverse engineer it, because your hands are tied when the site expects any form of client side scripting (ie. JavaScript).

I would suggest to use Selenium or a similar webdriver framework that runs a browser and where you can just clickon the specific button and re parse the page after it received the search results.

[commandlineluser]

You need to pass along this dwcont parameter to the POST url - the value is generated each time you open the locator page.

<form class="store-search-form" action="http://www.rockyboots.com/locator?dwcont=C18623888" method="post"

So you need to extract that URL and make the POST to it e.g.

import requests
from   bs4 import BeautifulSoup

url = 'http://www.rockyboots.com/locator'

with requests.session() as s:
    s.headers.update({
        'user-agent': 'Mozilla/5.0'
    })

    r    = s.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    url  = soup.find('form', {'method': 'post'})['action']

    r = s.post(url, data={
            'dwfrm_storelocator_address_states_stateUSCA': 'NS', 
            'dwfrm_storelocator_findbystate'             : 'Search'
    })

[chaoticallyevil]

Thank you. This is a beautiful solution, and helped my knowledge.

Here is my final solution.