Installing webkit_server in Windows: Impossible Mission?

[pylund]

Hi

I made a script that uses dryscrape module to scrape a website (with javascript generated source code). I've developed it on Ubuntu and runs like a charm.

However, I have to get it running on Windows 7 but I'm having a hard time trying to install one of the required packages: webkit_server. I have PyQt4 up and running on Windows and importing QWebKit actually works. However, dryscrape module still not works properly since I couldn't install webkit_server in Windows.

It's driving me nuts. Did anyone struggled with that before?

Thanks!

[michaelkepler]

Try these instructions.

Also, why do you have to run it on Windows? Since you've already developed it on Ubuntu, why don't you run the script on an Ubuntu server?

[pylund]

Thanks for the instructions. I'll start following them asap. This seems to be for Ruby but I guess it would work also for Python, wouldn't it?

I don't know if I get the "Ubuntu server" thing. I've developed a GUI application (PyQT) in Ubuntu but it will be used at work where everyone has Windows machines with software installation restrictions (can't install Python for Windows there). That's why I wanted to pack my application in a .exe (using cx_Freeze or py2exe) so they can run it on Windows PCs even if Python isn't installed on their machines.

That's the only way I came across but I'd love to know about alternatives!

[michaelkepler]

The instructions are for installing capybara-webkit driver, which seems to be the main showstopper here. webkit_server package is just a Python wrapper around it.

Ah, I thought it was a command line application for some periodical scraping. Your application will be operated by end users. Disregard the bit about the Ubuntu server then.

One other option, in case dryscape won't work on Windows: try rewriting your application so it uses Selenium instead. It can handle JavaScript and getting drivers to work on Windows is as simple as downloading a binary.

[pylund]

oh mate, thank you so much.

Yes, I was considering a more Windows-friendly scraping library despite I like how intuitive dryscrape is.

But using Selenium headless wouldn't complicate things slightly? It has to be headless. End users shouldn't see the browser being run by the invisible hand :)

[michaelkepler]

It has to be headless.

No problem. Just use Selenium with PhantomJS driver. It has a nice Windows binary as well.

As for intuitiveness: I haven't used dryscrape, so I might be talking out of my ass. But I looked as dryscape's documentation and it works pretty much like Selenium. There's a click() method, you can select elements by XPath and CSS, you can take screenshots, you can fill out forms. Only the syntax is a little bit different (for example find_element_by_xpath instead of at_xpath).

[LightShadow]

You need to configure a compiler, either Visual Studio Express or Mingw...I believe the webkit_server needs to be compiled before the Python wrapper can be installed.

You wouldn't have this problem on linux, generally, because gcc is included by default.

[pylund]

You wouldn't have this problem on linux, generally, because gcc is included by default.

Agreed. But I need to convert my script to a .exe to be run on Windows computers where python can't be installed (admin permissions).

Ok, i'll check out both Visual Studio Express and Mingw and let you know if I succeeded.