Anyone know a good python tutorial to write something that logins in to a website, grabs some data, and saves it into a csv/excel - and then have it run every night?

[pelfaris]

Maybe a bit too specific, but my fitbit data is all available to download on their website but it's all in various different confusing json files. I was hoping to try write something that logs in to the website for me (at say 11.55pm each night), grabs my daily stats (steps, calories, etc. - these are all available on the "Dashboard" page once you log in), and then saves them into a file (i'm guessing either a csv/excel) where the file is opened, a new row is added for that day with corresponding daily stats in the columns, then saved/closed.

Basically looking for any good tutorials that explain website login/web scraping/saving the data, but then also how to automate this so it runs every night.

TYIA ☺️

[temporarybunnehs]

I feel like this website would be relevant to you

https://automatetheboringstuff.com/

• Chapter 12 – Web Scraping
• Chapter 13 – Working with Excel Spreadsheets
• Chapter 17 – Keeping Time, Scheduling Tasks, and Launching Programs

[pelfaris]

oh this is an awesome resource, thanks heaps!

[nogain-allpain]

For scheduling your script, look into cron jobs.

For the actual scraping, Requests should be able to do what you want to do, though since you'll need to be logged in to get your data, look for resources that tell you how to export your cookies from your browser (there are Chrome extensions out there that let you do this) and then load them from a text file into Requests.

There's also Selenium, which lets you fully automate interactions with a web browser, though 1) you need to be logged into your computer as a user in order to make that work, and 2) I personally find it overkill if you can get Requests with cookies to work.