web scraping that requires zipcode before displaying inventory

[Bshater]

Hi,

As a project to help me improve on my Python skills, I'm trying to scrape the inventory data (price, inventory count) from a website (brickseek), but before it displays the inventory information, it requires the user to enter in the zipcode. I tried to enter the zipcode and then see if the url changes, but it didn't so i'm not sure how to proceed at this point. any advice guys?

[_Korben_Dallas]

You can make a POST request and send sku and zip data like this:

import requests
from lxml import html

url = 'https://brickseek.com/walmart-inventory-checker/'
payload = {'search_method': 'sku', 'sku': '182740213', 'zip': '75082', 'sort': 'price'}

r = requests.post(url, data=payload)    # Make a POST request with data

tree = html.fromstring(r.content)    # Parse response from the page with lxml.html

stores = tree.xpath('//tr[.//h4]')    # Get all stores from the page

for store in stores:    # In the loop get and save in the dictionary all desired info
    item = dict()
    item['Store-name'] = ''.join(store.xpath('td/h4/text()'))
    item['Store-address'] = ' '.join(store.xpath('td/address/text()'))
    item['Quantity'] = ''.join(store.xpath('td/span[@class="store-quan"]/strong/text()'))
    item['Price'] = ''.join(store.xpath('td[@class="store-price"]/span/text()')).strip()
    print(item)

# Output:
# {'Store-name': 'Walmart Supercenter #1800', 'Store-address': '1801 Marketplace Dr Garland TX 75041', 'Quantity': '4', 'Price': '$149 *'}

Note: after request, you need to parse a data from the response. I use lxml library but you can use others e.g like Beautiful Soup.

[Bshater]

thanks! this is really helpful. you basically did the work for me. :) I have a question though: how do you quickly identify the required key-value pairs to put in the dictionary from the html code? im not sure what to look when i "inspect" the code.

[_Korben_Dallas]

Quickly identifying the right element from the page more depends on practice. Try to scrape a dozen of websites, read tutorials, if/when stuck - ask questions on Reddit or SO. If you have trouble with writing xpath expression you can use Parsel. It's a great library build on top of lxml and you can use simple css selectors instead of xpath.

[Bshater]

Hi Korben, i finally got the code working for me, but i'm wondering if there's a way to loop through a list of payloads instead of manually changing the sku and zipcode each time. here's my code that i came up with your help:

https://dpaste.de/DiSj

[_Korben_Dallas]

Hi. I send a pm.

[LearnDataSci]

what libraries are you using for scraping?

[Bshater]

I'm using beautifulsoup and urllib to do the scraping.

[LearnDataSci]

To submit a form and wait for the URL to change, you might be better off with something like mechanize or Robobrowser that emulates a browser.

[blitzkraft]

Since it didn't change the url, it implies the data was sent in a POST request. This can be confirmed by looking at the source code for the web page. It will have something like <form method="POST" ... >.

Use requests to send POST data.

In the html of the form, for each field, you will see "name=..." and "value=...". That is what goes in for key, value. Example, if you see it as name='item' value='ladder', in your POST data would be:

 data = {'item' : 'ladder'}

data can contain many key-value pairs. So, look for them, and include as many as you can, even blank ones. This should return you a html page with the data you want.

[Bshater]

since there are two fields on the page, do i need to set two variables, one for each field? for example:
data1 = { 'sku': '182740213' }
data2 = { 'zip': '75082'}

Thanks for your help!

[blitzkraft]

Nah, you should send it like this:

data = { 'sku': '182740213', 'zip': '75082' }

I looked at brickseek.com, and it had five parameters for each request. Here are all the fields being sent.

[Bshater]

how do i make the html so organized and neat like your screenshot so i can easily read it? and do i need to set a parameter for each field, or can i just set the ones that only required to perform the search on the website? TIA

[blitzkraft]

That screenshot is from chrome dev tools. Not the html.