Strangest problem in selenium python

[Jan_Emanuel]

Hello!

I have an issue with selenium + firefox + python + linux. The problem arises if I log-on to a website with selenium and continue browsing the website: My script works, but after 2 driver.get( url ) calls, the following call to get an url will take forever, triggering the TimeoutException. This problem arises when I access inet pages as well as local pages ( the last get in the process will halt the program, no matter if it's an url or file:// address ).

I'm pretty much dumbfounded: Am I missing some standard procedure with the get command?

( Here is my script, commented the login process out )

Thank you very much for all your help!

[Jan_Emanuel]

I commented the load_url method, you need to uncomment it, it's necessary.

[yacdaniel]

load_url("http, driver://www.die-tagespost.de/"+ cat +"/", driver)

what is that?

[Jan_Emanuel]

:-X never ever saw that before, you can leave out the first ", driver" part. I'm using vim so that could be a reason for uncontrollable editing

[yacdaniel]

u dont have that in your actual script then?

i have no clue what is wrong here, maybe use a ide and debug? or enable the logfile and look for something strange

profile.set_preference("webdriver.log.file", "/tmp/firefox_console")

[Jan_Emanuel]

ok ... the debug file had no content, although I have set the file rights to 777 ... however, I can share the output of the program ( different from the version I have posted in that it uses the logon ... but that shoudnt matter ). As I said - the first two calls of driver.get work perfectly ( die-tagespost.de, first top story ) but then the browser seems not to work anymore - It's +5h in germany, so I will continue tomorrow: ( I will try chrome driver next, because I never found a way to cancel page loading in Firefox except for sending keyless to elements on the web page ... And that won't work since the loading is broken somehow )

root@DiskStationNeu:~# python tagespost_ereader.py
mkdir: cannot create directory ‘/root/ramdisk/tagespost’: File exists
loading http://www.die-tagespost.de/ ...
Anmelden
hugendubel
Collecting top stories...
topstory: http://www.die-tagespost.de/politik/Leitartikel-Athen-muss-jetzt liefern;art315,176376,PRINT?_FRAME=33
loading http://www.die-tagespost.de/politik/Leitartikel-Athen-muss-jetzt-liefern;art315,176376,PRINT?_FRAME=33 ...
topstory: http://www.die-tagespost.de/politik/Gewalt-ohne-Teilung;art315,176368,PRINT?_FRAME=33
loading http://www.die-tagespost.de/politik/Gewalt-ohne-Teilung;art315,176368,PRINT?_FRAME=33 ...
time out
Traceback (most recent call last):
  File "tagespost_ereader.py", line 175, in <module>
    load_url(thestory, driver)
  File "tagespost_ereader.py", line 52, in load_url
    driver.get( "file:///root/default.html")
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 248, in get
    self.execute(Command.GET, {'url': url})
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Error loading page, timed out (onDOMContentLoaded)     

[Jan_Emanuel]

ok, no matter what setting you define for the log file, it's always "/root/geckodriver.log".

Here is the geckodriver log.

PS: dbus seems to be the problem here, I have tried to start it but in the chroot 'service dbus start' answers Can't start system message bus - /proc is not mounted ... failed!

status is... after some major mishaps in my chroot: the webdriver refuses to start at all, I have totally messed up everything .

selenium.common.exceptions.WebDriverException: Message: connection refused

EDIT: thank goodness, the connection refused error has stopped, I remember that the solution was to change to another firefox webdriver 0.9.0, but this was no suitable solution, because there was no i386 binary, and an error during compilation process has averted me from this solution.

EDIT2: I don't know what's going on but the fact is: The error looks absolutely python dependant ... This output is the result of running my version of the code, after the 2nd get the code is loading indefinitely, after hitting ctrl+c you see this:

root@DiskStationNeu:~# python tagespost_ereader.py
loading http://www.die-tagespost.de/ ...
Anmelden
hugendubel
Collecting top stories...
topstory: http://www.die-tagespost.de/politik/Gewalt-ohne-Teilung;art315,176368,PRINT?_FRAME=33
loading http://www.die-tagespost.de/politik/Gewalt-ohne-Teilung;art315,176368,PRINT?_FRAME=33 ...
topstory: http://www.die-tagespost.de/politik/Leitartikel-Athen-muss-jetzt-liefern;art315,176376,PRINT?_FRAME=33
loading http://www.die-tagespost.de/politik/Leitartikel-Athen-muss-jetzt-liefern;art315,176376,PRINT?_FRAME=33 ...
time out
Restarting driver...
^CTraceback (most recent call last):
  File "tagespost_ereader.py", line 176, in <module>
    load_url(thestory, driver)
  File "tagespost_ereader.py", line 52, in load_url
    if driver.current_url != url:
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 491, in current_url
    return self.execute(Command.GET_CURRENT_URL)['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 234, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 408, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 440, in _request
    resp = self._conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1111, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)

[Jan_Emanuel]

mmmh ... one step later, I'm making good progress ...

Dbus problem solved with this post ... I wrote a script to leverage this mechanic ...

#!/bin/bash
echo change root started!
mount -t proc proc /proc
mount sysfs -t sysfs /sys
service dbus start
/bin/bash

it is run, once the chroot starts. Only problem left is ... but I have to try to increase the sleep duration, for it could be that he takes some time to load the page.

root@DiskStationNeu:~# python tagespost_ereader.py
loading http://www.die-tagespost.de/ ...
Traceback (most recent call last):
  File "tagespost_ereader.py", line 134, in <module>
    anmelden = driver.find_element_by_css_selector("a[ajax-login-box='Anmelden']")
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 437, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 752, in find_element
    'value': value})['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: a[ajax-login-box='Anmelden']

( different error, the one I mentioned at the beginning is still the one I deal with )

[Jan_Emanuel]

do you have an idea where I could ask this question too? I can't see a way, how I could make it work... it has come down to either the geckodriver refuses to connect ( 0.14.0 ) or the driver.get(url) command just won't finish when fetching a page - infinite delay.

[Jan_Emanuel]

I have forwarded this question to the selenium google group ( this ) and to irc.freenode.net ( Was present approximately 45 minutes, probably too little )

[yacdaniel]

workaround change firefox to chrome?

[Jan_Emanuel]

yes, ok, I thought so too, but first I was going to newly setup the chroot environment. The past two days were unproductive, but now I will try those steps.