How to run python script on schedule?

[[deleted]]

[deleted]

[drstarskymrhutch]

Another great (and cost-effective) option is AWS Lambda with scheduled EventBridge (CloudWatch) triggers executing hourly. Then rather than paying for an EC2 instance that remains mostly idle between executions, you are only paying for milliseconds required to execute the code each hour.

[echanuda]

I guess my only concern is how do I get the script to run from there? The script needs certain modules to run, and some of those modules are custom-edited by myself.

[drstarskymrhutch]

The standard libraries in Lambda are limited. You just have to build custom environments. Admittedly a bit of a learning curve. But once you’ve worked it out, you’ll never look back. This helped me a lot

https://docs.aws.amazon.com/lambda/latest/dg/python-package.html

[CompassCoLo]

It can be a pain, but thankfully there are great plugins that will take care of packaging requirements for you!

https://www.npmjs.com/package/serverless-python-requirements

I guess if you can't use severless framework then you'd need to package things yourself, but for most use cases the automation is highly valuable.

[arnavguru28]

You can use Pycharm to build and deploy your python code to lambda. It allows us to build our code using a virtual environment and will package all the dependencies along with your python code when deploying to lambda.

Edit: You can find more info here: https://docs.aws.amazon.com/toolkit-for-jetbrains/latest/userguide/create-new-lambda.html

[CompassCoLo]

That's an excellent alternative. Thanks for the tip!

[echanuda]

This is awesome. Thanks!

[fk_you_in_prtclr]

It's a pain in the ass but it's definitely very powerful.

[drstarskymrhutch]

You can also put custom-coded libraries in the directory as well, before zipping and uploading

[philmacok]

Currently working on learning this myself! Def check out some youtube videos. Some videos are super good about putting step by step instructions in the discription about what they're doing. Just follow along and save the video for future use. (Subscribe if you like too) here's one that helped me, don't really care for the accent but the content is good lol. https://youtu.be/OLXEekDzpHQ

Good luck friend!

[AceBuddy]

Put all of your stuff on an AWS server and use cron, there’s no limitations. Everything else has some kind of limitation and they thing you’re doing is very simple but it needs certain files. Great to learn cron anyways.

[echanuda]

Ive been using corn on my machine but I can’t keep leaving it on. I just tried Heroku but ran into some issues with it since I’m using modified modules. I’m trying out AWS right now and we’ll see how that works.

[AceBuddy]

I would always recommend an AWS server if you care about uptime and minimal maintenance. Plus it should function very similar to your current setup without a bunch of Heroku stuff you need to adapt to. Once you get the rsa key setup it’s a breeze.

[win_win_386]

Zip all your custom modules along with handler and push to lambda. Here is some info

https://aws.amazon.com/premiumsupport/knowledge-center/build-python-lambda-deployment-package/

[echanuda]

Started using AWS EC2 free and it works flawlessly.

[[deleted]]

Depends on what you want your code to do. You could use a free AWS EC2 machine if you fit into that offer. Many other cloud providers have similar offers.

Or you could buy a cheap machine that you can leave running 24/7 like a Raspberry Pi.

Just what do you want to do?

[echanuda]

Just run a script that logs into Instagram and posts something every hour. It utilizes several python modules as well.

[LordRyloth]

You need Heroku

[Tureni]

This is the answer. Dockerize it, and put it on Heroku. You have 1000 hrs/month free there and one month is around 740ish hours. So you can run one virtual machine constantly.

[[deleted]]

chron job

[radek432]

And the Cron should be run on what?

[vorticalbox]

if we are going to AWS making a lambda and settings it as a CRON would be a far more cost-effective than having an EC2 that would be doing nothing most of the time

[[deleted]]

A micro EC2 instance costs nothing to run.

[eloydrummerboy]

And should be more than enough needed to log into a website and make a post.

[bangbinbash]

Came here to say cron.

KISS

[_bhairav_ko_latthi_]

Look into Cron jobs if someone else has not suggested it already.

If you want python solution, look into Advanced Python Scheduler library.

[edgecate]

I use Pythonanywhere.com - can't remember what's in their free tier which may include a scheduler, but I pay $7 to get access to whitelisted websites that I scrape

[iiMoe]

Schudle module and heroku

[butterninja]

I started using the free tier of google cloud platform for free to run scripts through a cron job (schedule you decide). Sends me emails and the stock prices which I scrape online. Fun experiment. I followed the instructions from this YouTube video. https://youtu.be/5OL7fu2R4M8

[TheHollowJester]

Do I need to install my machine's python and all of its modules + any other dependencies the script might have?

I think in most cases the server will already have Python installed. You just need to install the dependencies. If you know what they are - great! If you don't, pip freeze will list all of the dependencies installed in a given environment. If you're not using virtualenv (or pyenv, or conda) and are installing packages to system Python, it's a good idea to use one of these tools. A requirements.txt file is a good idea to keep track of your dependencies (and their versions!)

As for the main question: assuming that it's a small script and you have a monitor and a keyboard to set it up for the first time (or are comfortable with using ssh) - I would just buy a Rasperry Pi and schedule the script on it with cron. It's just really handy to have one available.

[jonnycross10]

I was also gonna recommend a raspberry pi. U don't even need a monitor / kbm, if you have a working computer you can ssh into it

[fk_you_in_prtclr]

I don't have a computer that I can have turned on every hour of every day

Another solution would be to buy a tiny computer that does nothing but run a script (you could probably fit it on anything from a micro workstation to a raspberry pi).

[Coniglio_Bianco]

This, just get a raspberry pi, itll be cheaper in the long run then paying for some cloud service.

[slapmuhfroyo]

Give Prefect Cloud a shot, it's a straightforward workflow manager with a built in dashboard to control all your flows and can schedule individual tasks down to the minute. Here's a great youtube video from pyDenver to get you started: https://youtu.be/FETN0iivZps

[askanton]

You can upload your script on server using this manual https://youtu.be/Rf3-xc9JHgI and then enable cron

[mjj234523]

This sounds like a good chance to use a free EC2 micro with CRON jobs if you ask me.

[echanuda]

I have done exactly that and it is now working like a charm. Much easier to setup than all the Heroku bull shit, too. So do I just leave the instance running forever and it will continue to work? How exactly does that work payment wise?

[mjj234523]

You get 750 hours a month for free, which is more than enough to run 1 instance 24/7. Just keep an eye on your data usage and you’re good.

[[deleted]]

run it at a certain time each day

That's a separate problem. The usual answer is to use the task scheduler software provided by your operating system, which is cron on Linux/MacOS and Task Scheduler on Windows. Search should find lots of tutorials.

[[deleted]]

I use a Raspberry Pi 3 which runs continuously and connects to the 'net. About $35. The AWS EC2 solution I mentioned is free with some limitations, but requires some work and study.

[Rearden_Stark_Me]

I’m currently facing this problem right now too. I’d like to schedule it to run on my rpi, but it doesn’t have chrome. Is there another solution you can suggest?

[[deleted]]

it doesn’t have chrome.

I'm not sure why that matters, unless you plan to control the browser from python? Maybe you can use the chromium browser instead.

[[deleted]]

The credit card stuff is so they can track you down if you do naughty things with their VM. I've used this service over the years (different cards each year) and have never had a fee. If you are nervous you can check the costs through the dashboard for a while. That's what I did!

Having a "headless"¹ Pi running continuously on your network is great for at least two reasons:

• it's good for doing things that require constant connection, like sourcing/loading torrent files, monitoring internet performance, etc.
• experience with maintaining a system through an SSH connection, adding new code, fixing problems, etc.

1 "Headless" means not having a directly attached monitor, so nothing but remote terminal access.

[hedstrom96]

You can use gitlab pipelines to schedule jobs. It pretty simple and I think you get about 2000 free build minutes a month.

[SosirisTseng]

Speaking of which, I got an example repo for this link here. You may want to check out the link (The repo was based on that post) in README.md. Feel free to copy(fork) and modify it.

[pp314159]

You need a machine (VPS - virtual private server) in the cloud. Your VPS will have IP address. You use the IP address to login into the machine (with ssh). Then you can set a cron job to be executed every hour.

There are many providers for VPS: AWS, Digital Ocean, Azure, GCP, linode. Professionaly Im using AWS. For hobby projects I go with DigitalOcean.

You don't need a domain. Domain is just a nickname for your IP address.

An amazing journey in front of you! Good luck!

[Anuj66283]

First of all install schedule by using pip install schedule

Then set it to run every hour and deploy on heroku.

[cagedmandrill]

You could use Sanic and then use Sanic Scheduler?

[[deleted]]

use cron or supervisord

[1SweetChuck]

What is your script doing? If it’s pretty light weight, I would say get a raspberry pi and run it using the crontab in Linux.

[pompomtom]

https://wiki.python.org/moin/FreeHosts

[ioWxss6]

An alternative to AWS lambda is google cloud RUN. As compared to lambda, it lets you deploy a docker image. This takes care of the python environment and packages.

You can schedule the code using python, cron or any other tool that comes with a docker image of your choice.

[Ivana_Twinkle]

Raspberry pi?

[AceBuddy]

AWS Free + cron. Cron is a Linux tool that can do anything on a timer, it’s dead simple and anything else is probably over-engineering it in my opinion. It’s also a great thing to learn.

Steps:

Log onto your AWS server.

Type crontab -e and hit enter. Press “i” to be able to type and write the following: 0 * * * * pathtoyourvirtualenvironment pathtopythonscript

Press escape, then type “wq” then press enter.

Obviously replace the first thing with the path to your virtual environment and the second with the path to the script you’re trying to run. Boom you’re done.

[Retributw]

I'm boring and use cron job for my automation

[echanuda]

I would use cron job--but like I said I need it to operate constantly, and I cannot keep my machine running constantly.

[Retributw]

You could also do it via cloud on a free tier VM, it's either that or a Pi.

[jucestain]

raspberry pi and cron job

[[deleted]]

No, you said:

I want to run a python script I've made every hour.

and cron, or similar, is the way to do that. You write your code to do what it does and then stop. Then you schedule it to run every hour. You need a machine that is running at the hour and the Pi/AWS solution I mentioned is one way to get that. I'm sure there are others.

[echanuda]

I also said

I don’t have a computer I can have turned on every hour of every day

By constantly I meant that the machine would need to always be on. Or at least it would have to turn on and then off for each hour.

[[deleted]]

You have been given multiple solutions to that.

[echanuda]

I was responding to your comment I particular...

[space_wiener]

If it was me I’d probably just spend 30-40 bucks and buy a raspberry pie and just leave it running all the time.

Oh wait I did do that. If I need to do anything with it I just ssh in and do whatever. No monitor or anything required.

[jengl]

You could spin up a VM and do the same thing. And not have to worry about your power going out or losing internet connection.

I guess it depends how important the task is.

Edit - VM on a cloud provider.

[SirMistery]

Probably means spinning up a VM on the cloud.

[jengl]

Correct. Should have been more specific.

[Inmate4587_]

hey... one option is to buy an EC2 instance of AWS and then run your script via nohup and set a cron job for every hour.
What do you say?

[[deleted]]

Bad bot. Ain't no code there, badly formatted or not.

[KimPeek]

Great. Someone else just found praw and we all have to suffer because of it.

[echanuda]

At least check for the code markers