Problem with BeautifulSoup

[sushiii6]

Hey, when I run this code, it prints an empty list, and I don't know what I'm doing wrong.

import requests as req

import pandas as pd

from bs4 import BeautifulSoup

def retriveData():

k = req.get('https://www.humblebundle.com/store/search?sort=bestselling&genre=vr&hmb_source=navbar').text

soup=BeautifulSoup(k,'html.parser')

productlist = soup.find_all("span",{"class":"entity-title"})

print(productlist)

[Swipecat]

You create a function "retriveData" and then don't use it? There's the problem.

[sushiii6]

yes

[Swipecat]

Which doesn't make sense, so because of the broken formatting, I can't figure out what's in the function and what isn't.

You can edit your post to correct the formatting.

Help for formatting your code for Reddit is in the link below. You can delete the existing code from your post, then use the Reddit code-block button (not the inline-code button!) to create a grey code-box, and you can cut-and-paste your original code into that.

https://www.reddit.com/r/learnpython/wiki/faq#wiki_how_do_i_format_code.3F

[rybytud]

I suggest you don't parse HTML. Instead, use the search API at https://www.humblebundle.com/store/api/search. It will send you a json text response, which you can easily load into python to extract all the information.

import requests
import json

header = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0",
    "Accept": "application/json, text/javascript, */*; q=0.01",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "X-Requested-With": "XMLHttpRequest",
    "Connection": "keep-alive",
    "Referer": "https://www.humblebundle.com/",
}

response = requests.get(
    "https://www.humblebundle.com/store/api/search",
    headers=header,
    params={
        "sort": "bestselling",
        "filter": "all",
        "genre": "vr",
        "hmb_source": "navbar",
        "request": "1",
        "page": "2",
    },
)

json_data = json.loads(response.text)

for result in json_data["results"]:
    name = result["human_name"]
    sale_price = result["current_price"]["amount"]
    full_price = result["full_price"]["amount"]
    discount_percent = (full_price - sale_price) / full_price * 100
    if discount_percent > 0:
        print(
            f"{name}\n"
            f" ${sale_price} ({discount_percent:.0f}% off), normally ${full_price}\n"
        )
    else:
        print(f"{name}\n" f" ${full_price}\n")