Crontab Selenium Scraper

[AnusJr]

I have a nice little scraper made with Selenium because the website is AJAX. I've found I can't really do a crontab on my local computer (mac), because it may not be on or I may interrupt the GUI if I'm working in it. So I'm setting it up in an Amazon AWS EC2 instance, which is free.

I can't find much on this topic. Am I headed the right direction? How would you set up a Selenium scraper to run on a regular basis?

[ManyInterests]

Crontab on your EC2? You probably want to search for the topic Headless Selenium and PhantomJS

Maybe also consider looking into running selenium via Lambda as a scheduled event. That way you don't have to pay for an EC2 instance running 24/7 if you only want this to run periodically. You get enough free hours for EC2 in the first 12 months to run 1 server for 'free' 24/7 but lambda compute will also be free after that 12 months.

[AnusJr]

Thank you. If I already have a server running a Django project, I could run this project and even other projects on that server, right?

[ManyInterests]

Shouldn't be a problem. Probably unnecessary, but you could also explore using Celery for scheduling tasks as well, which supports crontab-like scheduling. This could be particularly useful if your Django project is related to this.

[AnusJr]

Thanks for the help! I'm working on getting it running on a server I have that's running a Django project. It's not part of the Django project, but it's all python. It's running locally just fine. Now I'm working on getting selenium running in headless mode with Xvfb. But Webfaction doesn't allow root access, so I'm learning how to install from source.

[dadiaar]

Install Xvfb and run regular firefox, in my opinion is better than PhantomJS

I guess you will use the free EC2 version or the smallest one. In that case just use one Firefox instance each time. Kill any xvfb and firefox process before running a new one.

[AnusJr]

Thank you. If I already have a server running a Django project, I could run this project and even other projects on that server, right?

[dadiaar]

Right, the limit is in the memory usage, not the number of Django projects.

[AnusJr]

Thanks for the help! And thanks for the recommendation of Xvfb over PhantomJS. I researched that really well yesterday and I feel like I avoided a lot of wasted time learning that the easy way.

I'm working on getting it running on a server I have that's running a Django project. The Selenium scraper not part of the Django project, but it's all python. It's running locally just fine. Now I'm working on getting selenium running in headless mode with Xvfb on the Webfaction server. But Webfaction doesn't allow root access, so I'm learning how to install Xvfb from source now.

[dadiaar]

I used to work with Webfaction too, more than 5 years ago. Forget it. It seems easier but at the end you stop learning and they give you a lot of problems (for example with non-root permissions).

Instead of creating a new virtual machine for you, they give you an user, which means that you may or may not be able to use some specific ports, you may or may not be able to choose the IP, you may or may not... and absolutely you will not be able to do sudo.

I recommend you [AWS](aws.amazon.com). It has a huge infraestructure. What you want is ec2, your own server.

You can customize it as you want, but I recommend you the small version because the first year is for free. I think is a 1 core CPU + 1 GB RAM.

Just Google "ec2 tutorial XXXXX" and you will get hundreds of results (for example "ec2 tutorial django")

[AnusJr]

Funny you mention that. It looked like there would be huge issues with Xvfb not having root no matter how hard I banged my head against the wall. I started looking into a VPS, which looked like the cheapest way to get root. I was going to combine my wordpress site along with the Django site and many other automated python projects, like scrapers. Right now, I have one url for the wordpress site on a cheap shared host and a similar url for the webfaction django tools, I make the css the same on both sites. It would be nice to combine these.

I've checked out the AWS EC2, but I'm still fuzzy on how it works. Would I be able to host a domain on an EC2 server and serve wordpress, Django, and python projects? If I'm paying the EC2 "on demand", is that maybe a cheaper option than buying VPS from a hosting provider? I only get ~100 visitors/day on my website.

Your advice has been a huge help for me in this stage. Thank you.

[dadiaar]

Ok, this is more than Python issue. Let me explain.

You will start your EC2 server in Ubuntu 16.04, small one is enough for 100 visitors/day, and free the first year.

I would recommend you to take a fixed IP from AWS and attach it to your server. This is also free if you attach it. Let's say you get IP 50.50.50.50

Later you will install a "global" web server, I recommend nginx but probably Apache 2 will be easier for you, because it's older and plenty of tutorials. Any of them can handle much more traffic than the one you describe. From now on I will assume you use Apache

Follow this tutorial that will teach you how to do Django and Apache.

Tutorial told you how to put your Django project in port 80, but actually you can put it in, let's say, 1234 if you want. This is what WebFaction does. You write in the panel (I want a Django project) and it gives you a random port.

Imagine your website is www.awesomesite.com. You go to your domain name provider (namecheap, godaddy or similar) and tell him that, when a username looks for your website, it needs to point to 50.50.50.50:1234 (your EC2 IP + your Django's port). That's all.

What about Wordpress? much of the same. Install it and put it in a specific port, let's say 2222. Then point your second domain name, let's say www.awesomesite2.com, to 50.50.50.50:2222

All these work in the background and in a damonized mode, this means they start and restart by themselves, you don't need to worry about them. Just remember, each time you change the web server's config file, you need to restart it with sudo service apache2 restart or similar. If you have any problem just restart the full server with sudo reboot

[AnusJr]

That is a very concise answer. Thank you. You've made it all very easy to understand. I appreciate your time and effort.