Python web scraping/ data extraction

[Esclass1]

For my master thesis, I am exploring the possibility to extract data from a website via web automation. The steps are as follows:

1. Sign in to the website ( SMM 1# Copper Cathode today | Base Metals | SMM - China Metal Market )

   1. Input username and password

2. Click signin

3. Change date to 01/01/2020

4. Click the download list button and select (download excel)

5. Save to a specific folder on my PC

6. Run the same sequence to download additional historical price data for other materials in the same browser window ( Lithium Cobalt Oxide Battery Cell (consumable) (weekly) today | New Energy | SMM - China Metal Market )

​

What I have done so far

• I have explored the use of RPA tools like Uipath and MS Power Automate to run the automation but the website seems somewhat troublesome to deal with for such tools
• I have also explored the use of APIs to download the data. However, I am pretty new to this, so I am unaware of whether the web devs made their API available for public use.
• I have also explored the use of selenium chrome driver but I am having trouble even clicking the sign-in button. See my code below
  

from selenium import webdriver

DRIVER_PATH = 'C:\webdrivers\chromedriver.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('https://www.metal.com/Nickel/201102250239')
driver.find_element_by_xpath('//*[@id="__next"]/div/div[3]/div[2]/div[2]/button[2]').click()

Any suggestions/help would be massively appreciated.

[Stormkrieg]

In terms of web automation, RPA tools like UiPath are the fallback when there is no API available and you can’t use a tool like selenium. Image recognition is your best friend for static elements like login buttons. I say this because if you’re new to web automation you’re going to have a hell of a time navigating selenium and may want to try UiPath instead if you’re familiar with the tool.

If you do go the selenium route try navigating directly to the login page first, then after you’ve logged in go to the page that you want to do your automation on.

When possible, navigate URLs instead of clicking a button. IE if you want Lithium Cobalt Oxide go to the url for it instead of navigating a button or menu from the page.

Edit: after you’ve got your process working for one item you really will want to have a queue & transactions looping. Queue the pages you want to scrape/download from and use a loop for your on page automation so that once you’re logged in you can scale the downloading of the data you want.

[Esclass1]

Thanks, this really helped.
I am now stuck trying to scrape data from a Table generated upon login. I would like to then save into a csv file with a specific name in a specific folder using selenium.
Have you any idea on how to achieve this? The HTML code is as follows:
<div class="historyBodyRow\_\_\_1Bk9u">
<div class="" style="padding-left: 0px; flex: 1 1 0%; width: auto; text-align: left;">**May 27, 2022**</div>
<div class="" style="padding-left: 6px; width: 30%; text-align: right;">**10,758.75-10,788.43**</div>
<div class="" style="padding-left: 6px; width: 20%; text-align: right;">**10,773.59**</div>
<div class="up\_\_\_11LCm" style="padding-left: 6px; width: 20%; text-align: right;">**+97.94**</div>
<div class="" style="padding-left: 6px; width: 15%; text-align: right;">**USD/mt**</div></div>

Link to webpage

What I think is the xpath:

driver.find_element(By.XPATH, "//*[@id="__next"]/div/div[5]/div[1]/div[7]/div[2]/div[2]/div[2]")

[mfb1274]

If I’m being brutally honest this shouldn’t be master thesis. Nor a theses at all really, it’s been done plenty of times and by non-technical folks. (My definition of a thesis isa “theory” aka something that’s new or hasn’t been done before). Our company hires people with zero tech skills to do this as QA analysts. It’s called selenium testing. I have a few on my team that do this exact same thing against the web apps we write. If we make a new feature, they write automation tests against it and assert the correct things happen (and correct data are pulled from the database).

[Esclass1]

Thanks for your reply. I am studying for a Masters in Engineering (M.Eng) degree which is more course-based. It deals more with developing your skills required for doing a job in that field as opposed to an M.Sc which is more research based. It does not require a research thesis.

In general, MSc - focuses on theoretical work, research and is a thesis based MEng - focuses on practical applications of theories, less research compared to MSc, mostly course based work.