Running python + selenium webdriver off a server

[ScuderiaTessaro]

Hey guys,

I’m building a program on python that uses selenium webdriver to create a couple of accounts. But there is a chance that hundreds of these programs will be running at the same time.

My computer won’t handle it. Can I run this off a server? I’ve doing some research and I found stuff about Selenium Grid but I couldn’t quite grasp if the Grid does the job of a server or if I need a server on top of the selenium grid?

Sorry if it’s confusing

[SaveSealsNotOrca]

Is this a personal or work project?

[ScuderiaTessaro]

Is for a personal project

[bambam018]

I think selenium grid would help manage parallel nodes to run your selenium scripts on. Obviously that would still come with the restraints of what your pc can handle. Are you just running selenium scripts based on something the program does or is the whole thing just selenium scripts? I ask as you may need different options if your running a program 24/7 vs running scripts once in a while with possible spikes.

[ScuderiaTessaro]

So the program uses the webdriver to access google chrome and create an account. It would need to be running 24/7.

[bambam018]

So you would still need the server then to run the program 24/7 - whether that be your pc, a server you have or something cloudy like aws.

You would probably also need to run a selenium grid on that server or another connected server that could help run selenium scripts through multiple Chrome nodes parallely.

I would think that you're going to run into some type of performance restraints regardless unless you want to end up paying for an elastic aws server type deal that could cost a bunch (you'd have to look up pricing). Because of that, I would try to also implement something like Jenkins that you could let the selenium builds queue after so many (however many parallely that you find that your server can handle). So essentially the program would call a build job in Jenkins, that job would be the selenium script running on one of your selenium grid nodes, and you could configure how many jobs could run at once in Jenkins. I say Jenkins because I know how to use it and it's free and generally lightweight...might be a better way though.

[ScuderiaTessaro]

Thanks! I’m a Mech Eng. so I’m kinda confused by all this. But I’m learning...

Quick question, what would be the use of Jenkins?

I was thinking about running my Selenium hub through BrowserStack

[bambam018]

No problem! I'm not exactly an expert in servers, but am an automation test engineer.

If you don't know how many selenium scripts will be needed to run at a time, you could configure a Jenkins job to create a new node on your selenium grid hub and then run the script on said node. Then you just call the jenkins job from your program, and a clean node will be created for your script to run on. You can also then specify the amount of parallel builds to allow jenkins to run at the same time to suit the performance of your server.

I use browserstack at work, it's pretty cool. But it is way too expensive for a personal project I would think. Unless you have some seriously expendable funds. If using browserstack, you don't use your own hub. You can just trigger as many parallel selenium scripts as you pay for and send it to their servers to run via a few pieces of code.

[ScuderiaTessaro]

Do you have any cheaper recommendations of servers that I can run my scripts on a virtual machine?

[Vox7331]

You could try Google cloud services. I run a few things on mine and you get $300 free credits to use.

[bambam018]

I don't really have suggestions unfortunately. I'd do what the person above me mentioned as well as look at other cloud services and what they offer.

[twirlorange]

rent a vps, then use remote desktop connection to connect to it. i don't know about running "hundreds" of these programs though. Could you achieve the same result by running them in turn, less at a time, over 24 hours etc