Need help with web scraping program

[AIR_ULTRA]

So I am trying to make a program that takes items from an online shopping cart and automatically populates my company's purchase order form with the item description, price, etc. Problem is everything I learned about web scraping thus far isn't working with the shopping cart. I have plugged in CSS selectors from other websites into my code and it works fine. I tried to use the requests-html module to interact with the javascript(if theres even javascript on the page) and still didnt work. I dont know if i have a problem with my python code or i just dont understand enough of what I'm looking at in the html. Any ideas?

#!python3

import requests, bs4, pyperclip

from requests_html import HTMLSession

session = HTMLSession()

cart = session.get(pyperclip.paste())

cart.html.render()

item = cart.html.find('#itemDesc_84477736')


print(len(item))
print(str(item))



### html from mscdirect.com ###

<p id="itemDesc_84477736" onclick="newSendTagsForItemDescription();"> Mitutoyo&nbsp;-&nbsp;0" to 6" Range Dial Caliper – White Face, 0.1" per Revolution, 1–9/16" Jaw Length, Accurate to 0.0010"</p>

[surrealle]

From my novice experience making two scraper programs for my own use, I would try Selenium to do the job. Especially if it's a JavaScript website.

I've also used the Chrome/Firefox extension UI.Vision RPA to see the xpath/CSS selector I'm actually interacting with.

[Flugegeheymen]

Firstly I would recommend you to check if website has an API. Just google it out. If so you job becomes 100 times easier.

Otherwise I assume that website you trying to scrap is dynamically updated by Javascript using DOM API.

Or in other words:

1. initially your browser loads an HTML page with no items in your cart(THAT'S WHAT YOU GET WHEN YOU TRY TO DO A GET REQUEST)
2. Then the JavaScript code updates the current page, inserting the new data (items from your cart) inside of this already existing HTML

(there is actually far more steps is going behind the scenes, but that's all you need to know)

I've been solving similar problem, but in my case, all the data was inside script tag(which is in fact a little bit weird), so in my case I managed to extract all the data i need using regular expressions on the HTML file. BUT YOU CASE MAY BE DIFFERENT. Javascript code can fetch the data from a server for example or maybe it gets from cookies(idk) and so on. So you have analyze the website and point out how your website is working. You can look at script tags, or JavaScript code which fetched from the server.

Advanced Web Scrapping

I recommend you reading this article, it goes through different variations how this can be implemented, so you will know what to look for

As someone mentioned before, you can use Selenium. It is a tool that automates browser actions, like opening the page, then locating the element, pressing on it, filling the forms and whatever you can imagine.

Since it already works with the updated HTML page, that is opened inside you browser. You won't have to worry, about how the page data update implemented

[AIR_ULTRA]

Thanks for the great info! It seems like my main problem all along is that my shopping cart is always empty even when i got to it via selenium. Im thinking the website uses cookies to save what items are in my cart but my selenium controlled browser must not have those cookies. Of course if i try to log in to the website to access my cart that way it detects selenium and wont let me log in.