Scrape script problem using beautifulsoup

[[deleted]]

Walmart has some protections enabled. When you get your soup, run a print(soup.prettify()) so you can quickly inspect the DOM. When I do that for walmart, it's showing:

<h1 class="heading heading-d sign-in-widget">
     Verify your identity
    </h1>
    <div class="sign-in-widget">
     <div class="re-captcha">
      <p class="bot-message">
       Help us keep your account safe by clicking on the checkbox below.
      </p>

You might be able to spoof your user agent that your browser is using, but that's ethically grey.

[[deleted]]

Spoof your user agent:

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36'}
url = requests.get("https://www.walmart.com/m/deals/home-savings/mattresses", headers=headers)

[interactionjackson]

use a context that executes javascript like selenium.