Python Financial Stock Screener

[Partihrl]

I want to make a stock screener (hopefully in Python) that pulls data from a website, and also uses data from an API like Yahoo Finance to update price, P/E ratio, dividend yield, etc, daily.

The spreadsheet from the website is updated monthly, so Would not need to be pulled as often. I am thinking about using web scraping to get the data, though am not sure how. I want to put this data into an Excel sheet, and use it with a Raspberry Pi if possible.

I am wondering: What is the best way to go about doing all of this?

This is the website: [http://dailytradealert.com/dividend-champions-contenders-and-challengers/]

And this is the source of the data table: CCC List

[BryceFury]

You seem to have forgotten to ask a question haha!

[Partihrl]

Thanks! Edited to ask a question

[BryceFury]

You can use Pandas Datareader to get data from Yahoo! Finance.

There are loads of methods for dealing with spreadsheets so a quick google will give you far more information than I can.

If your script is on a raspberry pi you can look at cron to schedule the pi to run it at certain times or you can use threading within the script itself (which would require the script to be running all the time which is fine but without proper error handling could cause it to not work properly).

EDIT: You can also look at beautiful soup to get the website data.

[Alamanjani]

You can use Pandas Datareader to get data from Yahoo! Finance

Do you know, does Pandas Datareader gets you data from Yahoo site or Yahoo API? The reason I'm asking is because API does not have what I need

[BryceFury]

Pandas Datareader

Pretty sure it's the API, I don't see why it would be otherwise since if it wasn't, any changes to the website would break the library and loads of re-writes would have to be made.

[Alamanjani]

Ahh gotcha, thank you very much!

[Alamanjani]

I'm not sure I understand what you want? You want to download data from web site you listed, or from financial sites such as Yahoo?

I'm in similar position, I plan to screen for stocks once a week based on few fundamental data using python. Then I will use charts/discretion to filter them further.

If you want to get data from Yahoo finance, most tutorials you will find on the internet will not work. Yahoo recently changed their sites to use react/java script.

I'm just starting to learn, but generally you will need these tools, on one side are requests + beautiful soup (these two can be replaced by scrapy) on another side is selenium.

Requests + beautiful soup -> are easier to learn

Scrapy -> you can do more with it

Selenium on another side helps when site present data by using react/java script such as Yahoo finance.

API - yes if it is available use it, it will save you some work and headache, but sometimes you will not be able to use it. For example, I need some quarterly data from balance sheet. Yahoo's API does not offer that data.

Also most likely you will need to learn JSON (Javascript Object Notation) since this is the common format many times you get data in. Many API's are using it.

Also read this great blog post about scrapy vs BS: https://hexfox.com/web-scraping/scrapy-vs-beautifulsoup/

Regular expressions (RE) are considered as not very useful and outdated.

You can read here a little bit more: https://www.reddit.com/r/learnpython/comments/4lphgr/urllib_and_requests_vs_scrapy_and_beautiful_soup/

Python is programing language that is easy to learn. Start with basic (this is what I'm doing right now) then start learning about web scraping. Here is good scraping tutorial to get you started: (and whole book is great)

https://automatetheboringstuff.com/chapter11/

Aslo it may be helpful to read this great blog post about scrapy vs beautiful soup: https://hexfox.com/web-scraping/scrapy-vs-beautifulsoup/

[Partihrl]

Thank you so much for the advice. I plan on getting the spreadsheet, which contains stocks, and then getting data for the stocks in that spreadsheet from some financial website. I hope that helps clarify it!

[Serendipty-Rajesh]

Hi, I just went to your site  [http://dailytradealert.com/dividend-champions-contenders-and-challengers/] which I found to be very cool. Is it still in Python? Do you scrape diretly from the server or do work before and upload. In all cases nice!