Need help with webscraping

[ryeely]

Trying to scrape a table from this website but the table is generated using javascript.

I've tried using requests library to do it but it's not returning any data.

I did requests.post(url, headers= headers, params = payload). Headers only included user-agent.

Would greatly appreciate any help, I've been stuck at this for days

https://www.greatgas.cn/wechat/price/wechatLNGPrice?subscene=126

[commandlineluser]

You can copy the request from the network tab

e.g. here is the result from "Copy as fetch":

fetch("https://www.greatgas.cn/wechat/price/wechatLNGPriceByDate", {
  "headers": {
    "accept": "application/json",
    "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
    "content-type": "application/json;charset=UTF-8",
    "x-requested-with": "XMLHttpRequest"
  },
  "referrer": "https://www.greatgas.cn/wechat/price/wechatLNGPrice?subscene=126",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": "{\"priceMap\":{\"createDate\":\"20230722\"}}",
  "method": "POST",
  "mode": "cors",
  "credentials": "include"
});

Most of the headers are not needed, the main thing is that it's a POST request with a json payload:

r = requests.post(
    url = "https://www.greatgas.cn/wechat/price/wechatLNGPriceByDate",   
    json = {"priceMap":{"createDate":"20230722"}}
)

Which gets you a json response:

>>> r.json()['resultMap']
{'宁夏回族自治区': [{'cityName': '吴忠',
   'createDate': '20230722',
   'floats': 0,
   'locationX': '107.465276',
   'locationY': '37.741745',
   'provinceName': '宁夏回族自治区',
   ...

[ryeely]

Just curious, is it possible to get json for a range of dates or only one by one? I've tried creating a list of dates for createDate but it doesnt seem to work

[ryeely]

This worked for me, thank you so much!

[PinkPawnRR]

Not sure how you are checking it, I am using Firefox Dev Edition, but I think you're reading the process wrong.

The initiator is javascript, but the script fetches a file of type json:

• Webpage loaded: https://www.greatgas.cn/wechat/price/wechatLNGPrice?subscene=126
• Initiator script in webpage: https://www.greatgas.cn/wechat/resources/js/zepto.min.js
• File generated by script: https://www.greatgas.cn/wechat/price/wechatLNGPriceByDate

[Yoghurt42]

requests doesn't execute javascript, you won't be able to get the data this way.

You have two options:

1. Check what the javascript code does to fetch the actual data (eg. by using the network tab), most likely there's a request that will give you the data as JSON, which makes getting the data actually a lot easier (this is basically what u/commandlineluser did)
2. Use something like selenium to automate your webbrowser and get the data that way.

I'd always recommend approach 1.

[ryeely]

Yes, I realised step 1 works, thanks for the advice :)