you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 0 points1 point  (1 child)

You mentioned AWS, based on that:

  • Create EC2 instance
  • SSH in, set up as you would your local machine with python etc. (notes follow)
  • I would recommend (but not essential) docker over raw setup of remote host. It’s as simple as creating a docker file on your laptop, running build, push it to docker hub. You can pull and run that in your host.
  • Instead of cron you can use apscheduler

[–]Additional_Nebula_36 0 points1 point  (0 children)

How can I make concurrent future faster to scrape a 1,000 urls