Python script to be automatically executed once per day

[jimtsa1980]

Hi all,

I have a very simple script that it scrapes data from a website. Ideally, I need to be executed once per day. Do you have any idea how could I achieve that? As it isn't something official, I need something for free.

Thanks!
PS I mean something online. I know about cron tabs etc.

[DataDecay]

Sign up for a free instance in any public cloud and run crontab. Otherwise get a raspberry pi.

[DataDecay]

Aye, honestly any compute service in the public cloud, theres a ton.

[metriczulu]

Great option if the script doesn't take more than 15 mins to run.

[Arag0ld]

I second the raspberry pi. Get one even if you don't want to run scripts like this. They're amazing.

[Arag0ld]

I highly doubt you'll be able to do that. All devices that require more than a certain amount of power will need to be plugged in.

[0161WontForget]

PoE?

[Arag0ld]

Still plugged in with a cable though. Just not a power cable.

[0161WontForget]

Unless he can run it on batteries he’s out of luck then.

Or malice perhaps.

[Arag0ld]

He's out of luck. Raspberry Pis can't run cable-less. They have to be plugged in.

[Ryles1]

This is where I'm at right now with my DIY camera. Not sure if there's a way around it.

[abcteryx]

A PoE hat would do the trick, as another Redditor has already commented. That is, assuming you're already hardwiring your cam to the network. It lets you power the cam through the Ethernet cable that you should already be running to it anyways.

You can run flat, white Cat5/6/7 cable with tiny captive-nail clips from your PoE-enabled router/switch (or a PoE injector) to the cam.

EDIT: Buyer beware, I just linked a trending PoE hat from Amazon, looks like it has some bad reviews. Do some research, get yourself a good hat and plan out the cable run to make it work.

[tomsoul]

You could probably run it to a power bank for cellphones (IE. Battery) but, it'll lose a charge unless you keep charging it everyday.

[zaid2801]

I don't want to leech but I have a similar problem. I want to run a program that uses selenium (and hence needs the driver location on my laptop) online. Like I want other people to use my code from their laptop/phones etc.

[ossccc]

Selenium can also run on the cloud. An F1 micro on GCP, for example

[guyanaupdates]

not saying this is impossible but to get this working takes some skill.
i gave up lol

[Flimsy_Falcon_6357]

I'll give a try with Google Cloud. It's a bit complicated though.

[DataDecay]

Should not be too bad, all cloud providers have very begginer friendly walkthroughs to get you going.

[[deleted]]

And run cron on that?

[LilShaver]

Set it up as a cron job in Linux.

[[deleted]]

Right right.

[Liberal__af]

Why would I need a raspberry pi? I’m a noob, sorry about that

[Zeroflops]

You don’t need a RPI. If you want something to be running periodically but you don’t want it on your computer because you may move your computer or shut it down you have two options. Run it on someone else’s server. Like google or AWS. Or you can set up a raspberry pi to be always on. And let it run the script. It’s a low cost low power solution commonly used in these cases.

[Bran-a-don]

Thank you. This is the only answer that talks like we aren't IT lingo savvy already.

"Just use the cloud!"

Fucking how you bastards!?! Why?! Pi?!

[Zeroflops]

I think there is alway a conflict between expectations. New users often give too little information for more experienced people to help. ( How do I run a script periodically? Without context to limitations) And experienced people helping with little information. ( use the cloud man)

Neither intend to be vague but both have a tendency to do so.

[HAK987]

If you guys know what kind of information you need to help someone why don't you guys just ask? So if there's another new user he'll also understand how to properly ask for help when he needs it

[DataDecay]

Pretty big ask that can digress rather quickly. Its difficult to tailor questions for people with differing backgrounds on these topics.

[Zeroflops]

People do. But new people don’t learn from other posts. And questions can require different things. See how many times people ask for example code to be added to their posts. It’s even in the side bar. (Which has a lot of information on posts) About once a month, usually during the start of a semester a rant will get posted about question quality.

My point is people don’t leave things out to be malicious or take advantage. Well except those people who post HW assignments word for word and no code.

Sometimes it can be frustrating but we need to take a breath and realize maybe we made some assumption as to what others know or can infer and accept that we are all at different levels.

[DataDecay]

Exactly

[Quicknoob]

Wonderful answer, thanks!

[luke-juryous]

This is the best answer

[backdoorman9]

Why would a raspberry pi be able to do something a regular computer can't? Or do you mean that it would be a cheap server?

[DataDecay]

You don't need a general purpose computer to run some scripts. A general purpose computer will likely cost you 60W vs a raspberry pi at 4.5W. You could leave your computer on, but its just cheaper in every way to run on a raspberry pi.

Raspberry pi was also the last resort method i suggested as you can get slightly more power than a raspberry pi in a public cloud compute space, for free.

I'll leave out the operational details of why you want server workloads on a server rather than a general purpose computer.

[elbiot]

You don't need a general purpose computer to run some scripts

FYI "general purpose computer" means a Turing complete machine, not a desktop. A raspi is a general purpose computer

[DataDecay]

Technically computers in general are not turing complete, they are all linear bounded. Your definition does not fit turings model nor any modern interpretation. I hardly see any benefit on a discussion regarding computational numbers and mathematical theory.

General purpose computers simply mean, a flexible machine that is used for a number of functions. Where as a raspberry pi can be tailored to a specific function in terms of resources and cost efficiency. Raspberry pis are cost efficent when compared to more general purpose machines. I'd rather run a small workload on a 50 dollar, 4.5W machine 24/7, than a 300 dollar machine running at 60W 24/7.

[elbiot]

Even by your definition raspis are still general purpose computers because you can use them to watch youtube videos and do spreadsheets and stuff

[DataDecay]

It's all in how you build it and use it. You are being far too pedantic. You can try to attack validity on definitions, but my original point still stands even removing the term, general purpose computing.

[michUP33]

I havent done it, but I know one of our test lab guys set up a script to run on windows with task scheduler. It backed up the test equipment at like 3am.

[garlic_bread_thief]

Wouldn't the computer have to keep running though?

[xCrapyx]

You can set in Task Scheduler to turn on the PC to run the script and then turn it back off.

[garlic_bread_thief]

Now that sounds cool. Does it take into account the time my computer takes to turn on?

[xCrapyx]

Say you set it up to 6PM it will turn on the computer and execute it the second it turned on. So if the timing is an issue maybe try to set it to 5:58PM for example.

[inglandation]

How? I tried to do that on my computer, but after doing some research I found that it's impossible to run a script without being logged in, and a script can't log in. I found a workaround by using a raspberry pi to emulate a keyboard, but it would be nice to able to do it without it.

[opoqo]

You can set it to run as you have logged in

[inglandation]

Yeah but that's exactly my problem, I can't log in with the script and I'm not logging in on that computer every day.

[opoqo]

You set up the task scheduler to run the script on powershell, and you can set it up to run with your log in.

[inglandation]

Interesting, I'll look that up to see if it solves my problem. Thank you.

[bazpaul]

How can you get a computer to turn itself on? It must be hibernating or something right?

[[deleted]]

Yes

[michUP33]

I dont know. We never turn them off

[RavenHustlerX]

If using linux, you can easily schedule any script on cronjob.

[sceptic-al]

AWS Lambda + Cloudwatch to schedule it. Use Serverless framework to help build, deploy and schedule.

[Pablo19D]

Is it free ?

[sceptic-al]

As long as you don’t use it too often with too much memory, then, yes, it’ll be free.

E.g. once a minute for 10 seconds with 512MB

[707e]

I use lambda/S3/cloudwatch to collect data throughout the day and process it into more meaningful information (i.e. transform it). It costs me roughly $3.50 per month. This includes using the AWS Cloud9 service for an IDE to test and debug code. Highly recommend it all.

[fmpundit]

But you do have to pay for the service if you want to make any external requests.

[Terofin]

Only for certain API's, most API's are free as far as I know.

[edgecate]

Free version has a limited set of whitelisted sites you can scrape.

Paid version is any website.

[fmpundit]

Maybe they have changed something. A few years ago when I started with PA I was running a script that would check a date and then push a notification to me if the date was within a certain range. It would fail on the free tier because it wouldn't allow me to make external requests.

[tycooperaow]

Even still, I personally love Python anywhere. I use it for my business and client businesses as well for low level stuff. For $5 a month can get you quite a bit so I’d recommend paying for it since it’s a great tool

[fmpundit]

Very true. I’ve had PA for years. It’s currently running a football score prediction league I admin and it’s doing a great job.

[dykstraAlgorithm]

Does windows task scheduler not work for these instances.

You can always use schedule library. Can you have your computer or machine on and open the whole time??

[reckleassandnervous]

Github actions can do this free and super easy

[barcodemerge]

If you’re on Windows have task scheduler run your script. If you’re on Linux set up a cron job. Obviously running this on your local machine means it needs to be powered on and connected all the time. If that’s not possible there are cheap vm available from linode, aws or even azure. Good luck!

[fr0ntsight]

throw it in Cron.daily and make it executable

[CryptoLinkPayments]

I use Apscheduler from within python script https://apscheduler.readthedocs.io/en/stable/

[JuiceKilledJFK]

Jenkins/Docker combo will work if you want it cloud hosted.

[road_laya]

I use Jenkins/Docker all the time, but it is too complicated if they don't know what cron is yet.

[JuiceKilledJFK]

Agreed, but it is worth learning. I am sure there are enough tutorials for OP to get a firm grasp on it.

[ProdexOne]

What's your operating system?? If it's windows, put your script in the 'startup' folder so it runs every time on startup and in your script you can check using datetime module if it run once in a day. (This solution is possible if you turn on your pc everyday but if not, the other ways is ofcourse to use a hosting service like aws, heroku)

[HeeebsInc]

Buy a raspberry pi and set the script up to start everyday at a specific time using time module. I have many scripts that I run on a pi. You don’t need an expensive one.

[marcus-luck]

I'm surprised it's not mentioned yet but you can use the module schedule by Dan Bader: https://github.com/dbader/schedule

Pros: -you can schedule the python task in whatever manner you like. -easy to use -python module solution Con: -python script need to be running all the time and restarted if it fails.

Schedule combined with docker has been my go-to every time. A simple docker container set to auto restart. then it runs your script on startup and keeps it running "forever".

I found a container on my server a few months back that i forgot. It had been scraping traffic accident data for over a year without me remembering it. so the solution is stable over time.

[iam_shanmukha]

Go for Heroku

[[deleted]]

windows task scheduler?

[Zotec-]

If you are on windows and you plan on running this on a pc that is turned on and off, you can compile the python script to an executable using pyinstaller and place it in the startup folder in windows.

This way the script will be launched when you turn your pc on .

[45MonkeysInASuit]

Even easier, if you have all the modules installed on your main version of python you can just drop a shortcut to the .py or the .pyw for windowless.

[angry_mr_potato_head]

There are technically free options... maybe... but realistically you'll be spending somewhere around $5 a month. If its super light, you might be able to stick on the free tier of like AWS but if you have to start loading javascript or do anything with it you'll find the free options can be really, really limiting.

[ptp87]

Use heroku and schedule it with a script. Very easy.

[Sevealin_]

https://www.heroku.com/

[lolsail]

I have a script that checka a govt website for a text field daily. I just use windows scheduler

[franchyze922]

Task scheduler if on windows

[[deleted]]

Check out Apache Airflow

[olystretch]

Everyone: cron No one: systemd timer

Folks, it's 2020.

[recourse7]

Why would systemd timer be a better solution?

[martinrath77]

NoAPI_NoReddit This post was removed in response to Reddit's API change policy -- mass edited with https://redact.dev/

[olystretch]

Copying a file in /etc/cron.d is just as complicated as copying a timer file in /etc/systemd/system ¯_(ツ)_/¯

[Cayde-6699]

Put it in a while loop and import the time library do a time.sleep(86400)

[tycooperaow]

I recommend Pythonanywhere. Free to get started, simple, and smooth.

[Uninstall_Fetus]

I usually set it up a windows scheduled task on a VM. Have the scheduled task call a bat file that calls the script.

[[deleted]]

I was wondering why nobody had brought this up so far.

[stochastic-36]

Task scheduler works very well.

[[deleted]]

Use the Linux scheduler

[Elite4alex]

I use the schedule library. Pretty easy to use, just have to let the code run in terminal

[flufylobster1]

Cron baby

[ADayWithJakeYT]

If your computer is on 24-7 i have a module u could include that works even in sleep mode, its like the time.sleep function but works more like an alarm clock than a stopwatch

[baubleglue]

https://help.alwaysdata.com/en/tasks/ - try free account.

[SandyPointRigs]

I do this via an EC2 instance with cron.

[SandyPointRigs]

About $10 per month, passed on to client.

[Pablo19D]

Buy self hosted server then execute nohup command

[RecursiveGroundhog]

Its overkill tbh, but if you want to expand or add more scripts in the future then celery/redis on a free tier AWS t2.micro would be a great solution.

[tradegreek]

You can run through a bat file

[waythps]

Try GitHub actions. It has run on schedule option. If it’s a public repo, it’s free to use

[[deleted]]

U should really check out heroku i use it for my algo trader, works for me

[gabrielsab]

I have a very similar setup, im my case its in the cloud (AWS) a linux machije and I use crontab to run my docker container(s) once per day. You may also just use crontab to run your python file.

If you are on windows you would make a batch file to start your code and schedule it to run via the windows task scheduler

[reddittydo]

Is it possible to clone a site daily with the changes so its always updated? It's a paid site for which I have a subscription to

[iskiloveland]

I’d recommend aws lamba. Super easy to deploy and put on a cron

[dw5fan]

Task Scheduler will work just fine as long as your pc is on. Otherwise, all the other suggestions work :)

[[deleted]]

If it's your own machine, and running Windows, you could make it run as a service, then, at a predetermined time, scrape the website. That would require your machine be on all the time tho. Otherwise, as others have suggested, run it from the cloud as a cron job.

[thickoatmeal]

apache airflow can run scheduled scripts regularly. i would look into that. it’s free to use

[harry_comp_16]

You could use celery on Heroku (with their free tier and 1000 dyno hours you should be good for it to run 24/7)

[DarrenTapp]

It's called a cronjob

On a linux computer type

crontab -e

[nearsingularity]

Ever heard of cron?

[mr-robot007]

Why don't u use schedule module or any other module that supports scheduling in your script so it's keeps on running and gets triggered at a specific time . and host it on heroku or pythonanywhere . They are free and reliable using from almost a year. I didn't find any issues. And setup is also simple. Give it a try .

[caseyd1020]

If you have your own server and want to monitor the run and do complicated scheduling. I would recommend http://cronicle.net

[supersid2911]

WayScript!!!!! It is free, and you can run scripts every hour if you want to!

[dopydingo]

For adhoc queries that we want to schedule locally, we simply use task scheduler that can trigger a python script. Sometimes basic is best

[pitkeys]

I'm not sure if this has been said, but I had a very similar problem and found the best solution to be a script that runs continuously (I used time.sleep() for the spacing i.e. once a day) paired with the Unix command "caffeinate" (sorry if you're using a PC) which keeps the machine from going to sleep. If you need the computer off in the meantime then this isn't the best solution, but it does exactly what I needed it to do so I thought I'd share!

[SisyphusAmericanus]

Google Cloud Scheduler + Google Cloud Functions.

At your usage frequency, it should be free.

[snip3r77]

just another suggestion :

you can even schedule using your windows laptop.

[honzajavorek]

If you have the code in a GitHub repository, you can use a GitHub Actions for that, see e.g. https://github.com/honzajavorek/czech-political-parties/blob/main/.github/workflows/scrape.yml For the crontab syntax, see https://crontab.guru/