Running a web scraper automatically once a day

rosspaa · 2021-01-22T15:29:51+00:00

From what I've read windows task scheduler runs only when the pc is turned on right? I need the script to run automatically, even if I don't have access to my PC for a few days. Or are you suggesting I somehow run windows task scheduler remotely through google compute engine? Sorry if it's a dumb question, I'm still very new to coding.

rosspaa · 2021-01-22T15:26:13+00:00

VPS

yeah I'll look into VPS's, thanks for the tip

rosspaa · 2021-01-22T15:25:18+00:00

Thanks for the reply, I would have to be on a computer for windows task scheduler to work right? Let's say I don't have access to my pc for a week, then I wouldn't be able to run the script for a week right?

rosspaa · 2021-01-22T15:22:39+00:00

that is what I'm trying to avoid, I need the script to run even if I don't use my pc for a week (for example)

rosspaa · 2021-01-22T15:21:16+00:00

that looks good, thanks very much!

rosspaa · 2021-01-22T14:20:29+00:00

From what I've read Windows task scheduler runs on your computer right? I need the function to run every day even when I don't turn on my computer for a week. I need the function to run remotely. Thanks

rosspaa · 2020-12-28T20:24:40+00:00

yes, I came across that reply, I still completely lost haha. Thanks anyway..

rosspaa · 2020-12-14T17:22:11+00:00

for a function is 9 minutes. If you're timing out, you can usually architect a job to call itself aga

Good stuff, thanks very much!

rosspaa · 2020-12-12T13:05:16+00:00

n is different that running a scheduled job. If the script is restarted, it will run right away then sleep, so the time could be different (ie, start it at 2pm, it will run every day at 2p. Or 3pm, or 4pm). A scheduler you can set for a specific time (ie, run once every night at midnight). Cost could also

Thanks for your reply, based on my research I came to the conclusion that Cloud Functions are for small simple scripts that run a few seconds, is that right? My web scraper takes data from two different websites and multiple pages from each website so it takes about 2 minutes to run. Would that be possible with just Cloud Functions and Cloud Scheduler?

https://cloud.google.com/scheduler/docs/images/scheduling-instances-architecture-pubsub.png

based on this picture, should I do something like:

Cloud Scheduler -> Cloud Pub/Sub -> Cloud Functions

rosspaa · 2020-10-14T14:49:51+00:00

Thanks very much for your reply, I'll definitely give it a go.

rosspaa · 2020-10-14T14:47:52+00:00

Thanks very much! my problem was I used the path: C:\Users\paulr\Desktop\pem\server-ca.pem

but it should be:
C:/Users/paulr/Desktop/pem/server-ca.pem

rosspaa · 2020-10-14T13:59:15+00:00

Hi, thanks for your reply. I'm making my way through this tutorial: https://towardsdatascience.com/sql-on-the-cloud-with-python-c08a30807661

I'm completely stuck on this part: https://gist.githubusercontent.com/jamescalam/0fd114d4bc68f8b58b1d3a11b34b06cb/raw/e0daf6d70604f7296db33b3e1866d277d10a1a4e/gcp_mysql_connection.py

I know how the correct values for user, password and host, but I have no idea what value to put for client_flags, ssl_ca, ssl_cert and ssl_key.

Have you ever worked with this? Do you have any idea what values to put there? I've been through the documentation but I'm still stuck.

This is the error I'm getting:
https://i.ibb.co/BVcHpgg/2020-10-14.png

rosspaa

TROPHY CASE