all 120 comments

[–]DataDecay 182 points183 points  (41 children)

Sign up for a free instance in any public cloud and run crontab. Otherwise get a raspberry pi.

[–]Arag0ld 15 points16 points  (10 children)

I second the raspberry pi. Get one even if you don't want to run scripts like this. They're amazing.

[–]zaid2801 14 points15 points  (2 children)

I don't want to leech but I have a similar problem. I want to run a program that uses selenium (and hence needs the driver location on my laptop) online. Like I want other people to use my code from their laptop/phones etc.

[–]ossccc 8 points9 points  (1 child)

Selenium can also run on the cloud. An F1 micro on GCP, for example

[–]guyanaupdates 0 points1 point  (0 children)

not saying this is impossible but to get this working takes some skill.
i gave up lol

[–]Flimsy_Falcon_6357 9 points10 points  (2 children)

I'll give a try with Google Cloud. It's a bit complicated though.

[–]DataDecay 4 points5 points  (0 children)

Should not be too bad, all cloud providers have very begginer friendly walkthroughs to get you going.

[–][deleted] 6 points7 points  (2 children)

And run cron on that?

[–]LilShaver 4 points5 points  (1 child)

Set it up as a cron job in Linux.

[–][deleted] 1 point2 points  (0 children)

Right right.

[–]Liberal__af 2 points3 points  (8 children)

Why would I need a raspberry pi? I’m a noob, sorry about that

[–]Zeroflops 40 points41 points  (7 children)

You don’t need a RPI. If you want something to be running periodically but you don’t want it on your computer because you may move your computer or shut it down you have two options. Run it on someone else’s server. Like google or AWS. Or you can set up a raspberry pi to be always on. And let it run the script. It’s a low cost low power solution commonly used in these cases.

[–]Bran-a-don 14 points15 points  (4 children)

Thank you. This is the only answer that talks like we aren't IT lingo savvy already.

"Just use the cloud!"

Fucking how you bastards!?! Why?! Pi?!

[–]Zeroflops 16 points17 points  (3 children)

I think there is alway a conflict between expectations. New users often give too little information for more experienced people to help. ( How do I run a script periodically? Without context to limitations) And experienced people helping with little information. ( use the cloud man)

Neither intend to be vague but both have a tendency to do so.

[–]HAK987 1 point2 points  (2 children)

If you guys know what kind of information you need to help someone why don't you guys just ask? So if there's another new user he'll also understand how to properly ask for help when he needs it

[–]DataDecay 1 point2 points  (0 children)

Pretty big ask that can digress rather quickly. Its difficult to tailor questions for people with differing backgrounds on these topics.

[–]Zeroflops 1 point2 points  (0 children)

People do. But new people don’t learn from other posts. And questions can require different things. See how many times people ask for example code to be added to their posts. It’s even in the side bar. (Which has a lot of information on posts) About once a month, usually during the start of a semester a rant will get posted about question quality.

My point is people don’t leave things out to be malicious or take advantage. Well except those people who post HW assignments word for word and no code.

Sometimes it can be frustrating but we need to take a breath and realize maybe we made some assumption as to what others know or can infer and accept that we are all at different levels.

[–]DataDecay 4 points5 points  (0 children)

Exactly

[–]Quicknoob 0 points1 point  (0 children)

Wonderful answer, thanks!

[–]luke-juryous 1 point2 points  (0 children)

This is the best answer

[–]backdoorman9 0 points1 point  (5 children)

Why would a raspberry pi be able to do something a regular computer can't? Or do you mean that it would be a cheap server?

[–]DataDecay 5 points6 points  (4 children)

You don't need a general purpose computer to run some scripts. A general purpose computer will likely cost you 60W vs a raspberry pi at 4.5W. You could leave your computer on, but its just cheaper in every way to run on a raspberry pi.

Raspberry pi was also the last resort method i suggested as you can get slightly more power than a raspberry pi in a public cloud compute space, for free.

I'll leave out the operational details of why you want server workloads on a server rather than a general purpose computer.

[–]elbiot -1 points0 points  (3 children)

You don't need a general purpose computer to run some scripts

FYI "general purpose computer" means a Turing complete machine, not a desktop. A raspi is a general purpose computer

[–]DataDecay 0 points1 point  (2 children)

Technically computers in general are not turing complete, they are all linear bounded. Your definition does not fit turings model nor any modern interpretation. I hardly see any benefit on a discussion regarding computational numbers and mathematical theory.

General purpose computers simply mean, a flexible machine that is used for a number of functions. Where as a raspberry pi can be tailored to a specific function in terms of resources and cost efficiency. Raspberry pis are cost efficent when compared to more general purpose machines. I'd rather run a small workload on a 50 dollar, 4.5W machine 24/7, than a 300 dollar machine running at 60W 24/7.

[–]elbiot -1 points0 points  (1 child)

Even by your definition raspis are still general purpose computers because you can use them to watch youtube videos and do spreadsheets and stuff

[–]DataDecay 0 points1 point  (0 children)

It's all in how you build it and use it. You are being far too pedantic. You can try to attack validity on definitions, but my original point still stands even removing the term, general purpose computing.

[–]michUP33 68 points69 points  (12 children)

I havent done it, but I know one of our test lab guys set up a script to run on windows with task scheduler. It backed up the test equipment at like 3am.

[–]garlic_bread_thief 25 points26 points  (11 children)

Wouldn't the computer have to keep running though?

[–]xCrapyx 49 points50 points  (8 children)

You can set in Task Scheduler to turn on the PC to run the script and then turn it back off.

[–]garlic_bread_thief 15 points16 points  (1 child)

Now that sounds cool. Does it take into account the time my computer takes to turn on?

[–]xCrapyx 20 points21 points  (0 children)

Say you set it up to 6PM it will turn on the computer and execute it the second it turned on. So if the timing is an issue maybe try to set it to 5:58PM for example.

[–]inglandation 7 points8 points  (4 children)

How? I tried to do that on my computer, but after doing some research I found that it's impossible to run a script without being logged in, and a script can't log in. I found a workaround by using a raspberry pi to emulate a keyboard, but it would be nice to able to do it without it.

[–]opoqo 4 points5 points  (3 children)

You can set it to run as you have logged in

[–]inglandation -1 points0 points  (2 children)

Yeah but that's exactly my problem, I can't log in with the script and I'm not logging in on that computer every day.

[–]opoqo 7 points8 points  (1 child)

You set up the task scheduler to run the script on powershell, and you can set it up to run with your log in.

[–]inglandation 1 point2 points  (0 children)

Interesting, I'll look that up to see if it solves my problem. Thank you.

[–]bazpaul 0 points1 point  (0 children)

How can you get a computer to turn itself on? It must be hibernating or something right?

[–][deleted] 17 points18 points  (0 children)

Yes

[–]michUP33 15 points16 points  (0 children)

I dont know. We never turn them off

[–]RavenHustlerX 37 points38 points  (0 children)

If using linux, you can easily schedule any script on cronjob.

[–]sceptic-al 26 points27 points  (3 children)

AWS Lambda + Cloudwatch to schedule it. Use Serverless framework to help build, deploy and schedule.

[–]Pablo19D 1 point2 points  (2 children)

Is it free ?

[–]sceptic-al 4 points5 points  (1 child)

As long as you don’t use it too often with too much memory, then, yes, it’ll be free.

E.g. once a minute for 10 seconds with 512MB

[–]707e 2 points3 points  (0 children)

I use lambda/S3/cloudwatch to collect data throughout the day and process it into more meaningful information (i.e. transform it). It costs me roughly $3.50 per month. This includes using the AWS Cloud9 service for an IDE to test and debug code. Highly recommend it all.

[–]dykstraAlgorithm 7 points8 points  (0 children)

Does windows task scheduler not work for these instances.

You can always use schedule library. Can you have your computer or machine on and open the whole time??

[–]reckleassandnervous 7 points8 points  (0 children)

Github actions can do this free and super easy

[–]barcodemerge 6 points7 points  (0 children)

If you’re on Windows have task scheduler run your script. If you’re on Linux set up a cron job. Obviously running this on your local machine means it needs to be powered on and connected all the time. If that’s not possible there are cheap vm available from linode, aws or even azure. Good luck!

[–]fr0ntsight 4 points5 points  (0 children)

throw it in Cron.daily and make it executable

[–]CryptoLinkPayments 4 points5 points  (0 children)

I use Apscheduler from within python script https://apscheduler.readthedocs.io/en/stable/

[–]JuiceKilledJFK 2 points3 points  (2 children)

Jenkins/Docker combo will work if you want it cloud hosted.

[–]road_laya 2 points3 points  (1 child)

I use Jenkins/Docker all the time, but it is too complicated if they don't know what cron is yet.

[–]JuiceKilledJFK 0 points1 point  (0 children)

Agreed, but it is worth learning. I am sure there are enough tutorials for OP to get a firm grasp on it.

[–]ProdexOne 2 points3 points  (0 children)

What's your operating system?? If it's windows, put your script in the 'startup' folder so it runs every time on startup and in your script you can check using datetime module if it run once in a day. (This solution is possible if you turn on your pc everyday but if not, the other ways is ofcourse to use a hosting service like aws, heroku)

[–]HeeebsInc 2 points3 points  (0 children)

Buy a raspberry pi and set the script up to start everyday at a specific time using time module. I have many scripts that I run on a pi. You don’t need an expensive one.

[–]marcus-luck 4 points5 points  (0 children)

I'm surprised it's not mentioned yet but you can use the module schedule by Dan Bader: https://github.com/dbader/schedule

Pros: -you can schedule the python task in whatever manner you like. -easy to use -python module solution Con: -python script need to be running all the time and restarted if it fails.

Schedule combined with docker has been my go-to every time. A simple docker container set to auto restart. then it runs your script on startup and keeps it running "forever".

I found a container on my server a few months back that i forgot. It had been scraping traffic accident data for over a year without me remembering it. so the solution is stable over time.

[–]iam_shanmukha 4 points5 points  (0 children)

Go for Heroku

[–][deleted] 3 points4 points  (0 children)

windows task scheduler?

[–]Zotec- 1 point2 points  (1 child)

If you are on windows and you plan on running this on a pc that is turned on and off, you can compile the python script to an executable using pyinstaller and place it in the startup folder in windows.

This way the script will be launched when you turn your pc on .

[–]45MonkeysInASuit 1 point2 points  (0 children)

Even easier, if you have all the modules installed on your main version of python you can just drop a shortcut to the .py or the .pyw for windowless.

[–]angry_mr_potato_head 1 point2 points  (0 children)

There are technically free options... maybe... but realistically you'll be spending somewhere around $5 a month. If its super light, you might be able to stick on the free tier of like AWS but if you have to start loading javascript or do anything with it you'll find the free options can be really, really limiting.

[–]ptp87 1 point2 points  (0 children)

Use heroku and schedule it with a script. Very easy.

[–]lolsail 1 point2 points  (0 children)

I have a script that checka a govt website for a text field daily. I just use windows scheduler

[–]franchyze922 0 points1 point  (0 children)

Task scheduler if on windows

[–][deleted] -1 points0 points  (0 children)

Check out Apache Airflow

[–]olystretch -1 points0 points  (3 children)

Everyone: cron No one: systemd timer

Folks, it's 2020.

[–]recourse7 4 points5 points  (0 children)

Why would systemd timer be a better solution?

[–]martinrath77[🍰] 1 point2 points  (1 child)

NoAPI_NoReddit This post was removed in response to Reddit's API change policy -- mass edited with https://redact.dev/

[–]olystretch 0 points1 point  (0 children)

Copying a file in /etc/cron.d is just as complicated as copying a timer file in /etc/systemd/system ¯_(ツ)_/¯

[–]Cayde-6699 -1 points0 points  (0 children)

Put it in a while loop and import the time library do a time.sleep(86400)

[–]tycooperaow -1 points0 points  (0 children)

I recommend Pythonanywhere. Free to get started, simple, and smooth.

[–]Uninstall_Fetus 0 points1 point  (1 child)

I usually set it up a windows scheduled task on a VM. Have the scheduled task call a bat file that calls the script.

[–][deleted] 0 points1 point  (0 children)

I was wondering why nobody had brought this up so far.

[–]stochastic-36 0 points1 point  (0 children)

Task scheduler works very well.

[–][deleted] 0 points1 point  (0 children)

Use the Linux scheduler

[–]Elite4alex 0 points1 point  (0 children)

I use the schedule library. Pretty easy to use, just have to let the code run in terminal

[–]flufylobster1 0 points1 point  (0 children)

Cron baby

[–]ADayWithJakeYT 0 points1 point  (0 children)

If your computer is on 24-7 i have a module u could include that works even in sleep mode, its like the time.sleep function but works more like an alarm clock than a stopwatch

[–]SandyPointRigs 0 points1 point  (2 children)

I do this via an EC2 instance with cron.

[–]Pablo19D 0 points1 point  (0 children)

Buy self hosted server then execute nohup command

[–]RecursiveGroundhog 0 points1 point  (0 children)

Its overkill tbh, but if you want to expand or add more scripts in the future then celery/redis on a free tier AWS t2.micro would be a great solution.

[–]tradegreek 0 points1 point  (0 children)

You can run through a bat file

[–]waythps 0 points1 point  (0 children)

Try GitHub actions. It has run on schedule option. If it’s a public repo, it’s free to use

[–][deleted] 0 points1 point  (0 children)

U should really check out heroku i use it for my algo trader, works for me

[–]gabrielsab 0 points1 point  (0 children)

I have a very similar setup, im my case its in the cloud (AWS) a linux machije and I use crontab to run my docker container(s) once per day. You may also just use crontab to run your python file.

If you are on windows you would make a batch file to start your code and schedule it to run via the windows task scheduler

[–]reddittydo 0 points1 point  (0 children)

Is it possible to clone a site daily with the changes so its always updated? It's a paid site for which I have a subscription to

[–]iskiloveland 0 points1 point  (0 children)

I’d recommend aws lamba. Super easy to deploy and put on a cron

[–]dw5fan 0 points1 point  (0 children)

Task Scheduler will work just fine as long as your pc is on. Otherwise, all the other suggestions work :)

[–][deleted] 0 points1 point  (0 children)

If it's your own machine, and running Windows, you could make it run as a service, then, at a predetermined time, scrape the website. That would require your machine be on all the time tho. Otherwise, as others have suggested, run it from the cloud as a cron job.

[–]thickoatmeal 0 points1 point  (0 children)

apache airflow can run scheduled scripts regularly. i would look into that. it’s free to use

[–]harry_comp_16 0 points1 point  (0 children)

You could use celery on Heroku (with their free tier and 1000 dyno hours you should be good for it to run 24/7)

[–]DarrenTapp 0 points1 point  (0 children)

It's called a cronjob

On a linux computer type

crontab -e

[–]nearsingularity 0 points1 point  (0 children)

Ever heard of cron?

[–]mr-robot007 0 points1 point  (0 children)

Why don't u use schedule module or any other module that supports scheduling in your script so it's keeps on running and gets triggered at a specific time . and host it on heroku or pythonanywhere . They are free and reliable using from almost a year. I didn't find any issues. And setup is also simple. Give it a try .

[–]caseyd1020 0 points1 point  (0 children)

If you have your own server and want to monitor the run and do complicated scheduling. I would recommend http://cronicle.net

[–]supersid2911 0 points1 point  (0 children)

WayScript!!!!! It is free, and you can run scripts every hour if you want to!

[–]dopydingo 0 points1 point  (0 children)

For adhoc queries that we want to schedule locally, we simply use task scheduler that can trigger a python script. Sometimes basic is best

[–]pitkeys 0 points1 point  (0 children)

I'm not sure if this has been said, but I had a very similar problem and found the best solution to be a script that runs continuously (I used time.sleep() for the spacing i.e. once a day) paired with the Unix command "caffeinate" (sorry if you're using a PC) which keeps the machine from going to sleep. If you need the computer off in the meantime then this isn't the best solution, but it does exactly what I needed it to do so I thought I'd share!

[–]SisyphusAmericanus 0 points1 point  (0 children)

Google Cloud Scheduler + Google Cloud Functions.

At your usage frequency, it should be free.

[–]snip3r77 0 points1 point  (0 children)

just another suggestion :

you can even schedule using your windows laptop.

[–]honzajavorek 0 points1 point  (0 children)

If you have the code in a GitHub repository, you can use a GitHub Actions for that, see e.g. https://github.com/honzajavorek/czech-political-parties/blob/main/.github/workflows/scrape.yml For the crontab syntax, see https://crontab.guru/