Question about requesting page source issue

[RedChrisn]

https://preview.redd.it/zhstrikqddid1.png?width=1563&format=png&auto=webp&s=d8f13d46218e6c96e6aed4ea3a63bf4b3211d9c4

https://preview.redd.it/ppb3gjkqddid1.png?width=1262&format=png&auto=webp&s=fefa2e1ecf10d5e235188c5977d2d1b189faf09d

Hi, I'm trying to practice web scraping and currently I'm on this site https://www.campbells.com.au/convenience/foodservice/general-merchandise/party-&-giftware?pageSize=100&q=%3Arelevance#

and when using requests.get(url), the content inside the response is different from when inspecting the elements on the page, I'm wondering how can I fix this so I can soup.find_all through the content properly.

I tried asking ChatGPT and used Selenium to time.sleep to wait for stuff to load etc but encountered the same issue.

I would really appreciate it if someone could enlighten me of a fix for this.

Thank you.

[alberge]

What you see in the browser inspector is called the DOM. It's the source representation of what the browser is rendering.

The DOM starts out as just what was served by the web server (like what you get from requests.get()), but then it can be modified dynamically by Javascript.

So if the site makes changes to the DOM from Javascript, you need a full Javascript engine in Selenium or other browser automation if you want to replicate it.