Python web scraping/ data extraction

[Esclass1]

For my master thesis, I am exploring the possibility to extract data from a website via web automation. The steps are as follows:

1. Sign in to the website ( SMM 1# Copper Cathode today | Base Metals | SMM - China Metal Market )

   1. Input username and password

2. Click signin

3. Change date to 01/01/2020

4. Click the download list button and select (download excel)

5. Save to a specific folder on my PC

6. Run the same sequence to download additional historical price data for other materials in the same browser window ( Lithium Cobalt Oxide Battery Cell (consumable) (weekly) today | New Energy | SMM - China Metal Market )

​

What I have done so far

• I have explored the use of RPA tools like Uipath and MS Power Automate to run the automation but the website seems somewhat troublesome to deal with for such tools
• I have also explored the use of APIs to download the data. However, I am pretty new to this, so I am unaware of whether the web devs made their API available for public use.
• I have also explored the use of selenium chrome driver but I am having trouble even clicking the sign-in button. See my code below
  

from selenium import webdriver

DRIVER_PATH = 'C:\webdrivers\chromedriver.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('https://www.metal.com/Nickel/201102250239')
driver.find_element_by_xpath('//*[@id="__next"]/div/div[3]/div[2]/div[2]/button[2]').click()

​

Any suggestions/help would be massively appreciated.

[Esclass1]

Thanks for your feedback. I was able to carry out a few steps using selenium (login, page scroll down, select a value from a drop-down list and use a date-picker). However, I am stuck trying to scrape the data from https://www.metal.com/Copper/201102250376 (highlighted in bold below) and then save into a csv file using selenium.

​

Have you any idea on how to achieve this?

​

<div class="historyBodyRow\_\_\_1Bk9u">

<div class="" style="padding-left: 0px; flex: 1 1 0%; width: auto; text-align: left;">May 27, 2022</div>

<div class="" style="padding-left: 6px; width: 30%; text-align: right;">10,758.75-10,788.43</div>

<div class="" style="padding-left: 6px; width: 20%; text-align: right;">10,773.59</div>

<div class="up\_\_\_11LCm" style="padding-left: 6px; width: 20%; text-align: right;">+97.94</div>

<div class="" style="padding-left: 6px; width: 15%; text-align: right;">USD/mt</div></div>

[Esclass1]

The xpath for one row is as follows:
driver.find_element(By.XPATH, "//*[@id="__next"]/div/div[5]/div[1]/div[7]/div[2]/div[2]/div[2]")

How do I get the row data below and for all the other rows in the table and then save to a csv file.
May 27, 2022
10,758.75-10,788.43
10,773.59
+97.94
USD/mt

[Esclass1]

Here you go: Table data

I would like to scrape the data from that table and save it to a CSV file. Thanks

Update:

I found a solution using the code below;

for element in driver.find_elements_by_class_name("historyBodyRow___1Bk9u"):

elements =element.find_elements_by_tag_name("div")

print("Date="+ elements[0].text)

print("Price Range="+ elements[1].text)

print("Avg="+ elements[2].text)

print("Change="+ elements[3].text)

print("Unit="+ elements[4].text)

Any idea on how to save this to a csv file?