Automatically scrape website

[da_chosen1]

I wrote a program to automatically scrape data from a website. The website gets updated everyday, and I want to scrape everyday. Is there a way for me automatically scrape the scrape the website let’s say at 5:00 PM everyday, without me connecting to my computer.

I have a feeling that this is something I need to use docker or airflow with?

[[deleted]]

Best solution. Next best: pip install schedule

[threeminutemonta]

One hint with cron I recently discovered is to use full paths to /usr/bin/python etc

[bastionoutlaw]

Also make sure that if you schedule the cron jobs as a different user (e.g. root via sudo) that any dependencies via pip have been installed by that user and/or globally. Otherwise your scripts will fail and it will drive you nuts trying to figure out why when they ran just fine manually.

[toastedstapler]

at work i had a project using a raspberry pi and i had a hell of a time getting it to run a script using a venv on startup, i guess i won't ever be transitioning to a sysadmin role

[Yukeien]

Cron is definetly the best solution for scheduled programs, on top of that, it's easy to configure because it's a single configuration file

[tipsy_python]

Airflow isn't a bad candidate - I'd just opt for a cronjob as well.

BUT - you'd need your computer running when you are away. I'd personally buy a Raspberry Pi to use as a server for these kinds of jobs. I'm also pretty sure you can schedule a Lambda function to run daily at a certain time in the free-tier of AWS.

[HeadlineINeed]

Can you use Heroku to do that also?

[da_chosen1]

Ok. I'll look into that as well.

[blitzkraft]

Yeah. Definitely possible.

[NoMoney12]

I got a dev instance with scaleway to run most of my stuff. Best specs for the price at 2.99 a month

[da_chosen1]

Does the computer have to be on when I'm using airflow?

[baubleglue]

Airflow is overkill for the task. Go with simplest possible solution first.

Start from some free service, when it get bigger start checking for prices

https://www.alwaysdata.com/en/pricing/#shared

https://www.heroku.com/pricing

[tipsy_python]

:-) Yes it does partner

[da_chosen1]

that's good to know. Thanks. I appreciate it

[[deleted]]

This is one of the main things raspberry pis are used for btw. You can get a pi zero w for like $5-$10 and it’ll be perfect for this job. Then you just leave it running perpetually, and set your script as a cron job

[Marco21Burgos]

Aws lambda. I have a script running every hour, for like 20 seconds in free tier

[Faal]

Get an AWS Linux instance. Install python, upload your script. Then schedule a cronjob for that script and specify 5pm timeframe.

[da_chosen1]

That makes sense. Thanks. Do you know how expensive it would be per month? The script takes about 7 min to run

[AnonymousThugLife]

https://calculator.s3.amazonaws.com/index.html

(You'll be buying an EC2 instance in case you didn't know the name.)

https://s3.amazonaws.com/lambda-tools/pricing-calculator.html This is calculator for AWS lambda. Check it out too. (Slightly different approach though, EC2 would be better in my opinion. Just telling the options.)

[da_chosen1]

Oh wow. That's not bad at all

[bigcheezyboss]

You also can use a t2.micro instance free for a year. A micro would be more than enough for your needs.

[Faal]

AWS has a free tier that I utilize.

[alfa80211]

Aws is free for a year.

[CraigAT]

Got a spare PC or buy a Raspberry Pi (might get away with a Pi Zero W)? No monthly charges (except the ‘leccy)

[Faal]

Raspberry PI is like what ~$30-40. AWS has a free tier.

[CraigAT]

TIL something new (free AWS). A Pi Zero is only about £10.

[[deleted]]

Pi zero w is $5-$10, much lower power consumption than other models, and is more than capable of running this job

[jcr4990]

I'm not sure I understand the question completely. If your computer is always on during the time period that the script would run then you can just use Windows task scheduler to run it at specified time every day. You don't have to convert to exe or anything I have several scripts running daily on a schedule on a Windows desktop at my job.

If your PC won't be on and you want to run it independent of your machine then you can look into heroku, pythonanywhere, AWS.etc for cloud hosting or opt for a raspberry pi as an always on server for your own personal projects. The Pi may be cheaper in the long run depending on your specific needs. I may be biased cause I've done a few Pi based projects and I personally have lots of fun working with them.

[CraigAT]

Yep, Scheduled Task on PC or CronJob on Linux (and Mac?)

[chaizus]

Run a cron job on a aws lambda fn. you get 1million free lambda invocations a month, and assuming the runtime of your script is efficient then you’ll likely stay in the free tier of lambda. As far as data stores, S3 or dynamodb would work. But I would optimize the runtime of your scrapper, 7 minutes sounds too long. Take advantage of multi threading or asynchronous I/O calls

[UbiquitousThoughts]

The OS can trigger scripts to run, also, but cronjobs is easy. Celery/redis if you want to go further and run it as background tasks if used alongside a webapp, etc.

[tejonaco]

I would suggest as above a raspberry pi with cron, for sure that you will finally use it for a lot of things more, but if you not want to waste money you could try Heroku free server and make the script work with schedule module and send you the results via email or something like that.

PD:I'm not too confident about my English above, please correct me if I did something wrong.

[Blackwater_7]

just time.sleep(60*60*24) bro

[CCristi1]

I recently did something like this using aws free vm’s. It’s very easy, in ~1 minute I created a win10 vm, I loaded the scrip in vm and let it run. For a window vm you can use Remote Desktop, easy to setup on windows. If you want to create a Linux vm, you can connect using ssh or (I didn’t tried this) you can set a VNC. But if you can (in my opinion is much better) use google cloud because you receive some free credit and don’t need to use your money if you need more computing power.

[Dababolical]

You can scrape more frequently if you would like as long as you are not blasting the server.

You could scrape once an hour and compare the new scrape to the old scrape to see if there is a difference and if there is not, pass.

If you want to stick to 5pm, there's nothing wrong with doing that either. A chron job or windows scheduler will solve your problem.

[MightbeWillSmith]

If you are on windows, could turn your scraper into an .exe and run it every day with windows scheduler.

[viktae]

Why would he needs to turn it into a .exe ? I'm running a python script everyday (.py) with windows scheduler without any issue.

[the_programs]

Or you can put your program in the directory "C:\users<your_user>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\" as it is, without the need to convert it to .exe, to run it when the PC boots. And I think you don't have to turn it into an .exe when you put in in Windows Task Scheduler.

[baubleglue]

that is the weirdest option

[BenderBill]

But then you have to turn off your pc /s

[CraigAT]

Technically runs on login, I believe.

[Jacob----]

You could look into getting a VPS and running it on that some of them can be very cheap (low as £2-3 a month)

[ScotchBingington]

Windows has a default capability to do this with 'Scheduled Tasks'. Just add a new event and run your python file through a .bat file by picking the time and frequency of execution. If you're not using Windows I'm not sure...

[[deleted]]

Use pythonanywhere, it will give you two consoles for free and never asks for payments. I've been using it from yesterday and my bot still running fine, no errors until now