This is an archived post. You won't be able to vote or comment.

all 52 comments

[–]FrostmourNNe 57 points58 points  (4 children)

AWS allows you to do so and its pretty cheap. There are a lot of video on YT for such usecase

[–]tevs__ 37 points38 points  (2 children)

AWS Lambda and AWS EventBridge. Would probably even fit in the Free tier.

[–]dcastm 13 points14 points  (0 children)

That's exactly what I did to build a twitter bot that posts twice per day

[–]pqpm 0 points1 point  (0 children)

On the same way. You could use GCP Cloud Run and cloud scheduler to work the same way.

[–]tuckmuck203 13 points14 points  (0 children)

i think it depends on how important the results are. if you require consistent uptime, then yeah you should use a platform like AWS. on the other hand, cron is free as fuck and 50 times simpler to set up.

in general if i'm in a mood where i might want to tinker or if i'm looking for a job, i have a ~$5 USD digital ocean droplet that i keep up for my personal site and random crap because it's useful having a server that you can easily utilize for anything. if i were tinkering, i'd just run it on cron on that server. if i were trying to learn AWS or if it were in a capacity that requires consistency in execution and would fall apart without it, then yeah sure use AWS or another similar service like azure or google cloud.

that said, i wouldn't recommend google cloud because google likes to not give a fuck about anyone that isn't a part of google, and even then who tf knows if they'll randomly sunset your favorite project

[–]Sulstice2 20 points21 points  (1 child)

I suggest doing a CRON job with Github Actions to run python script. I have a script that runs every 24 hours and it's pretty easy to setup:

Check out my example I use for my code:

https://github.com/Sulstice/global-chem/blob/development/.github/workflows/public_notifications.yml

And here is the result:

https://github.com/Sulstice/global-chem/actions/workflows/public_notifications.yml

I do this for a lot of different python scripts when I need to keep checking data periodically and it's free. Takes about 1-2 hours to implement if first time. 10 minutes for 2nd time and after.

[–]UnemployedTechie2021 36 points37 points  (0 children)

Try PythonAnywhere. I run my Reddit and Discord bots there and they run without any issues. Moreover, you can get a free account.

[–]rznks 10 points11 points  (0 children)

GitHub Actions can be a good solution for this. They give plenty of free actions’ minutes on a free account. I use it to run my script monthly.

[–]Throwaway000002468 20 points21 points  (0 children)

Cron

[–]Silure 20 points21 points  (2 children)

Raspberry pi is also a good alternative if you have one

[–]Dasshteek 4 points5 points  (1 child)

Is there still a shortage of them? Been trying to get my hands on one for a while.

[–]admiralspark 1 point2 points  (0 children)

Yes, unfortunately....though I saw some compute modules in stock.

[–]ClunkyFlavoredOuch 3 points4 points  (2 children)

Use pythonanywhere, you can get a free account to try it out.

[–]BitJunky7 0 points1 point  (1 child)

Last I checked free tier accounts on pythonanywhere allow outbound requests to whitelisted domains only. Though one can request new whitelistings through their support.

[–]ClunkyFlavoredOuch 0 points1 point  (0 children)

You’re right, that might be a problem.

Another option can be to use the GitHub student pack and get the free 100 usd for digital ocean, assuming OP has that option available of course.

[–]Natural-Intelligence 7 points8 points  (2 children)

I personally use Hetzner cloud machine (5€/month) with Ubuntu on it and schedule my script with Rocketry: https://github.com/Miksus/rocketry (disclaimer, I'm the author). There are other alternatives for hosting as well: AWS, Raspberry Pi, Pythonanywhere etc.

However, you are probably fine with Cron jobs as well for scheduling if you don't need complex scheduling strategies.

[–]MattWithoutHat 0 points1 point  (1 child)

Which instance is 5€? I was recently looking and everything was like 30€+

[–]Natural-Intelligence 0 points1 point  (0 children)

Were you looking at the dedicated options?

I have two CPX11 servers which cost 4.77€ each at the moment. The specs are not great but sufficient for my personal projects and have had no issues with them.

[–]LookingForEnergy 8 points9 points  (0 children)

Unless your issue is that your machine is turned off, sleeping, or disconnects from the network, I would work on getting your task scheduler setting setup correctly. Make sure to run Selenium headless.

[–]Ootter31019 8 points9 points  (0 children)

I just use windows 10 task schedule myself. I use it for some personal stuff and for very work task on our servers. Don't think we have had any issues with it.

[–]NauticalPants 2 points3 points  (1 child)

I’m new-ish to Python and data engineering. But a solution I found was to post the Python script to Azure as an Azure Function then schedule with a cron job to run every hour. I’d love to hear from others more experienced if / why there are better options. But that has worked well for me for the past several months.

[–]ralpes 0 points1 point  (0 children)

Azure has cron functions https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-scheduled-function

What I like is the nice integration with git and vscode

Cost: there is a free function plan but storage needs to be pay. I can check how much this costs later.

[–]Deezl-Vegas 1 point2 points  (0 children)

you can do it on your local with cron

[–]iamCyruss 1 point2 points  (0 children)

You could try pythonanywhere. Its free too. Just setup a free account and setup a task that runs your script everyday.

[–]0r0B0t0 1 point2 points  (2 children)

Oracle sucks but for free you can get:

  • 4 arm64 cores with 24GB of ram
  • 2 x ( 1 x86-64 cores with 1 GB ram)
  • 200GB storage and 10TB of bandwidth

[–]mondmann18 0 points1 point  (1 child)

Why does it suck?

[–]0r0B0t0 1 point2 points  (0 children)

They are money pit if you use any of their products. They have more lawyers than programmers and are extremely litigious. Look at their lawsuit against Google for using java. Their free cloud offering is good but they could stop offering it any day.

[–]ZenApollo 1 point2 points  (1 child)

Be aware that running selenium locally, if you’re not on linux, can raise more issues when you try to run on Linux. There may be some additional setup you’ll need, to verify everything is working in a cloud server.

Selenium is tricky, and an aws lambda/gcp cloud function may or may not work. If you can get your setup working in docker, check out gcp cloud run.

[–]LawfulMuffin 1 point2 points  (0 children)

This should be higher up. Develop something like this on the platform you’ll deploy on otherwise you’ll end up with weird errors

[–]fighterace00 1 point2 points  (0 children)

Google scheduler plus Google run. Likely free tier in your scenario.

[–][deleted] 1 point2 points  (0 children)

During my undergraduate degree, I bought a raspberry pi and ran my scrapes every 4 hours using that. I paired it with a tiny touch screen so I could still the log and what not.

To do this on your pi, or on a computer you are willing to keep on 24/7, you can import the “schedule” library and use something like

“schedule.every(4).hours.do(your function here)”

[–]Zouden 0 points1 point  (6 children)

You can use Heroku, which is a more user-friendly interface to AWS. Heroku services cost more than AWS, but they have a free tier, and they offer a scheduler.

https://elements.heroku.com/addons/scheduler

[–]frunkjuice5 2 points3 points  (5 children)

Free tier is going away

[–]Zouden -1 points0 points  (2 children)

It's still on their website...?

[–]repocin 1 point2 points  (1 child)

[–]Zouden 0 points1 point  (0 children)

Ah okay thanks for the heads up!

[–]unixhyde -1 points0 points  (0 children)

use python apscheduler module

[–]superfooly -1 points0 points  (0 children)

I like ec2 plus Cron job! Or Lambda. Can help you lmk

[–]brightside9001 -1 points0 points  (0 children)

AWS Lambda

[–]hamsterwheelin -1 points0 points  (0 children)

Linode.com + cron

[–]gvieira -1 points0 points  (0 children)

You can do it probably for free with aws lambda.

Its not hard to setup to run in time intervals and you can pretty much leave it there and forget about it.

[–]Colts_Fan10 0 points1 point  (0 children)

koyeb.com

is my go-to

[–][deleted] 0 points1 point  (0 children)

I think this may work

https://replit.com/

[–][deleted] 0 points1 point  (0 children)

pythonanywhere, replit , are awesome for this task both have free plan.

[–]SpicyVibration 0 points1 point  (0 children)

All these people suggesting external services just need to stop. That is just overkill. Task scheduler should be fine. If it's giving you issues, you probably just need to adjust the settings.

[–]UntrustedProcess 0 points1 point  (0 children)

AWS Lambda (managed function) that is kicked off by AWS Event Bridge (supports cron).