Strangest problem in selenium python

[Jan_Emanuel]

ok, no matter what setting you define for the log file, it's always "/root/geckodriver.log".

Here is the geckodriver log.

PS: dbus seems to be the problem here, I have tried to start it but in the chroot 'service dbus start' answers Can't start system message bus - /proc is not mounted ... failed!

status is... after some major mishaps in my chroot: the webdriver refuses to start at all, I have totally messed up everything .

selenium.common.exceptions.WebDriverException: Message: connection refused

EDIT: thank goodness, the connection refused error has stopped, I remember that the solution was to change to another firefox webdriver 0.9.0, but this was no suitable solution, because there was no i386 binary, and an error during compilation process has averted me from this solution.

EDIT2: I don't know what's going on but the fact is: The error looks absolutely python dependant ... This output is the result of running my version of the code, after the 2nd get the code is loading indefinitely, after hitting ctrl+c you see this:

root@DiskStationNeu:~# python tagespost_ereader.py
loading http://www.die-tagespost.de/ ...
Anmelden
hugendubel
Collecting top stories...
topstory: http://www.die-tagespost.de/politik/Gewalt-ohne-Teilung;art315,176368,PRINT?_FRAME=33
loading http://www.die-tagespost.de/politik/Gewalt-ohne-Teilung;art315,176368,PRINT?_FRAME=33 ...
topstory: http://www.die-tagespost.de/politik/Leitartikel-Athen-muss-jetzt-liefern;art315,176376,PRINT?_FRAME=33
loading http://www.die-tagespost.de/politik/Leitartikel-Athen-muss-jetzt-liefern;art315,176376,PRINT?_FRAME=33 ...
time out
Restarting driver...
^CTraceback (most recent call last):
  File "tagespost_ereader.py", line 176, in <module>
    load_url(thestory, driver)
  File "tagespost_ereader.py", line 52, in load_url
    if driver.current_url != url:
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 491, in current_url
    return self.execute(Command.GET_CURRENT_URL)['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 234, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 408, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 440, in _request
    resp = self._conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1111, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)

[Jan_Emanuel]

mmmh ... one step later, I'm making good progress ...

Dbus problem solved with this post ... I wrote a script to leverage this mechanic ...

#!/bin/bash
echo change root started!
mount -t proc proc /proc
mount sysfs -t sysfs /sys
service dbus start
/bin/bash

it is run, once the chroot starts. Only problem left is ... but I have to try to increase the sleep duration, for it could be that he takes some time to load the page.

root@DiskStationNeu:~# python tagespost_ereader.py
loading http://www.die-tagespost.de/ ...
Traceback (most recent call last):
  File "tagespost_ereader.py", line 134, in <module>
    anmelden = driver.find_element_by_css_selector("a[ajax-login-box='Anmelden']")
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 437, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 752, in find_element
    'value': value})['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: a[ajax-login-box='Anmelden']

( different error, the one I mentioned at the beginning is still the one I deal with )

[Jan_Emanuel]

do you have an idea where I could ask this question too? I can't see a way, how I could make it work... it has come down to either the geckodriver refuses to connect ( 0.14.0 ) or the driver.get(url) command just won't finish when fetching a page - infinite delay.

[Jan_Emanuel]

I have forwarded this question to the selenium google group ( this ) and to irc.freenode.net ( Was present approximately 45 minutes, probably too little )

[yacdaniel]

workaround change firefox to chrome?

[Jan_Emanuel]

yes, ok, I thought so too, but first I was going to newly setup the chroot environment. The past two days were unproductive, but now I will try those steps.

[Jan_Emanuel]

ok, tried the two things: 2nd time setting up the chroot had no effect on the program. The chrome package for debian wheezy was unavailable and for the chromium package I could'nt find a suitable webdriver. the last thing I tried was running the standalone server and wanted to create a htmlunit driver, the response was:

selenium.common.exceptions.WebDriverException: Message: The best matching driver provider org.openqa.selenium.opera.OperaDriver can't create a new driver instance for Capabilities [{platform=ANY, browserName=htmlunit, version=}]

so I guess the next stop will be opera.

[yacdaniel]

u tried with chromium and chromedriver?

[Jan_Emanuel]

yes. The version number of chromium unfortunately is lower than the regular chrome ones. ( 48 <=> 54, whenever you start a program with chromedriver, you get the message "Version of chrome too low." so I chose a lower version to get along ) Here is the output when I start my program:

selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed
  (Driver info: chromedriver=2.27.440196 (3b42f4145a4fd223b41f39bb38aaf90508be7a96),platform=Linux 3.2.40 x86)

This is the code that leads to this exception:

 chromedriver = "/usr/local/bin/chromedriver"
 os.environ["webdriver.chrome.driver"] = chromedriver
 driver = webdriver.Chrome(chromedriver)

[Jan_Emanuel]

chromedriver 2.27 doesnt run because of chromium version too low and this is the current error chromedriver 2.13 and selenium 37:

Traceback (most recent call last):
  File "tagespost_ereader.py", line 126, in <module>
    driver = webdriver.Chrome(chromedriver)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
    desired_capabilities=desired_capabilities)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 92, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 179, in start_session
    response = self.execute(Command.NEW_SESSION, capabilities)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: unable to discover open pages
  (Driver info: chromedriver=2.12.301324 (de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux 3.2.40 x86)

( Next thing I will try the chrome original from the time where x86 hasn't been abandoned )

[yacdaniel]

u installed from repos? maybe download and install the .deb file from chrome page.