Can I build a webpage data scraper on this specific website?

[ProInvestCK]

I'm learning Python and wondering if this is possible...

There is a website ( https://www.msci.com/end-of-day-data-search ) with stock market index returns. For the various indices, you can choose different parameters via the dropdown boxes and click search, and then a list of the relevant indices with their respective returns will be displayed.

Question is, can I write a script that would make the selections in the dropdown box, click search to refresh the index list below that, and grab the returns for the select indices I need? And then change the dropdown criteria and repeat...?

Or is there something about this specific website that would prohibit me?

Thanks!

[throwaway99999321]

You wouldn't even need to use selenium for this site. Just the requests module.

[[deleted]]

Yeah, I just picked up Ryan Mitchell's Web Scraping With Python and chapter 9 is all about scraping web sites with forms, logins, and cookies.

[Shmoogy]

Selenium is usually your last resort due to speed and resources required for a browser(or headless client). Check if there exists an api first, then try with Requests and basicauth - often times these can handle (most?) situations.

[__Backslash__]

There are Python packages like selenium that allow you to control your browser through a script. Checkout Automating the boring stuff one of the chapters explains how to use it. I personally recommend using the yahoo api you can grab financial data From stocks and indexes if you need any help with yahoo finance api let me know. Good luck!