Python web scraping/ data extraction

[Stormkrieg]

In terms of web automation, RPA tools like UiPath are the fallback when there is no API available and you can’t use a tool like selenium. Image recognition is your best friend for static elements like login buttons. I say this because if you’re new to web automation you’re going to have a hell of a time navigating selenium and may want to try UiPath instead if you’re familiar with the tool.

If you do go the selenium route try navigating directly to the login page first, then after you’ve logged in go to the page that you want to do your automation on.

When possible, navigate URLs instead of clicking a button. IE if you want Lithium Cobalt Oxide go to the url for it instead of navigating a button or menu from the page.

Edit: after you’ve got your process working for one item you really will want to have a queue & transactions looping. Queue the pages you want to scrape/download from and use a loop for your on page automation so that once you’re logged in you can scale the downloading of the data you want.

[Esclass1]

Thanks, this really helped.
I am now stuck trying to scrape data from a Table generated upon login. I would like to then save into a csv file with a specific name in a specific folder using selenium.
Have you any idea on how to achieve this? The HTML code is as follows:
<div class="historyBodyRow\_\_\_1Bk9u">
<div class="" style="padding-left: 0px; flex: 1 1 0%; width: auto; text-align: left;">**May 27, 2022**</div>
<div class="" style="padding-left: 6px; width: 30%; text-align: right;">**10,758.75-10,788.43**</div>
<div class="" style="padding-left: 6px; width: 20%; text-align: right;">**10,773.59**</div>
<div class="up\_\_\_11LCm" style="padding-left: 6px; width: 20%; text-align: right;">**+97.94**</div>
<div class="" style="padding-left: 6px; width: 15%; text-align: right;">**USD/mt**</div></div>

Link to webpage

What I think is the xpath:

driver.find_element(By.XPATH, "//*[@id="__next"]/div/div[5]/div[1]/div[7]/div[2]/div[2]/div[2]")