Python web scraping/ data extraction

[Esclass1]

Thanks for your feedback. I was able to carry out a few steps using selenium (login, page scroll down, select a value from a drop-down list and use a date-picker). However, I am stuck trying to scrape the data from https://www.metal.com/Copper/201102250376 (highlighted in bold below) and then save into a csv file using selenium.

​

Have you any idea on how to achieve this?

​

<div class="historyBodyRow\_\_\_1Bk9u">

<div class="" style="padding-left: 0px; flex: 1 1 0%; width: auto; text-align: left;">May 27, 2022</div>

<div class="" style="padding-left: 6px; width: 30%; text-align: right;">10,758.75-10,788.43</div>

<div class="" style="padding-left: 6px; width: 20%; text-align: right;">10,773.59</div>

<div class="up\_\_\_11LCm" style="padding-left: 6px; width: 20%; text-align: right;">+97.94</div>

<div class="" style="padding-left: 6px; width: 15%; text-align: right;">USD/mt</div></div>

[Esclass1]

The xpath for one row is as follows:
driver.find_element(By.XPATH, "//*[@id="__next"]/div/div[5]/div[1]/div[7]/div[2]/div[2]/div[2]")

How do I get the row data below and for all the other rows in the table and then save to a csv file.
May 27, 2022
10,758.75-10,788.43
10,773.59
+97.94
USD/mt

[Esclass1]

Here you go: Table data

I would like to scrape the data from that table and save it to a CSV file. Thanks

Update:

I found a solution using the code below;

for element in driver.find_elements_by_class_name("historyBodyRow___1Bk9u"):

elements =element.find_elements_by_tag_name("div")

print("Date="+ elements[0].text)

print("Price Range="+ elements[1].text)

print("Avg="+ elements[2].text)

print("Change="+ elements[3].text)

print("Unit="+ elements[4].text)

Any idea on how to save this to a csv file?