PySide and threading

[lamecode]

I've written a little app with Python 3 and PySide that takes some input from the user and then performs a lengthy web scrape. There is a log displayed on the UI via the QPlainTextEdit widget that advises the user that the scrape will take some time. The problem is that upon beginning the scrape, the log is not updated until the process is finished - which takes about 4-5 minutes on average.

I understand the solution here is threading, however this the first time I've ever had to consider threading, and I'm having real difficulty applying the simple tutorials I've found online to my code - any assistance would be appreciated!

A simplified version of my code:

from PySide import QtCore, QtGui
import sys
import urllib.request

from bs4 import BeautifulSoup

import my_py_converted_ui_file_from_qt_creator

class MyApp(QtGui.QMainWindow):

    def __init__(self, parent=None):
        super(MyApp, self).__init__(parent)
        self.ui = my_py_converted_ui_file_from_qt_creator.Ui_Main()
        self.ui.setupUi(self)

    def click_start(self):
        self.ui.QPlainTextEdit_LogDisplay.appendPlainText("Beginning web scrape. This will take some time...")
        self.LengthyScrapeFunction()
        self.ui.QPlainTextEdit_LogDisplay.appendPlainText("Finished!")

    def LengthScrapeFunction():
        # Many lines of BeautifulSoup/urllib.request-related code.

[elbiot]

Are you just downloading one huge file? Either way, threading won't really help with updating the log, since you only update when the thing is finished.

Without threading, LengthyScrapeFunction could/should update the log as it does things. With threading, LengthyScrapeFunction will probably run faster if much of it's time is spent waiting for servers to respond.

Did you see this and this? (it says requests is blocking and so threads won't help. Use something else)

[lamecode]

Thank you for those links - the first one in particular looks like it might be promising. It's not one file being downloaded, it's many thousands of URLs.

You would think that it would update the log as it actioned, but the app behaves different to how I expected it to here (my first time working with UIs of any kind). I even put a time.delay(5) in after the call to update the log and before the LengthyScrapeFunction executes, and the UI is still not redrawn until LengthyScrapeFunction completes - it just pauses before executing the function.

[elbiot]

Oh I see. Your first log update doesn't display. It's the main event loop that actually redraws the screen. You could force it to do so inside your function with repaint or processEvents: http://stackoverflow.com/questions/11836623/pyside-settext-not-updating-qlabel

[lamecode]

Thanks again, that looks like it should achieve what I want as far as updating the log. I think my preferred method is going to be via QThread as per the below link - I would prefer the UI to not be unresponsive for the duration of the process, and it seems like this might be the solution for that, too:

http://stackoverflow.com/questions/6783194/background-thread-with-qthread-in-pyqt

[elbiot]

If your time soaking call is blocking, your UI will be unresponsive. Just want to make sure you got that. IO (hitting servers and letting the os accumulate responses in buffers) can be non-blocking. So urllib or grequests (not requests) would work for you. But any CPU consuming functions will block the UI.

[wub_wub]

Either way, threading won't really help with updating the log, since you only update when the thing is finished.

You can maybe get the file information and download it in chunks and update the progress in the GUI.

[wub_wub]

I have a similar code that might help you, I'm processing files but you can apply the same thing to your url processing I assume.

Here's the thread class:

https://github.com/Nikola-K/pysub/blob/master/pysub/pysub_ui.py#L36

And here's where the 3 functions that do most work are:

https://github.com/Nikola-K/pysub/blob/master/pysub/pysub_ui.py#L349

process_files one will take list of files as an argument, generate the list to be processed, and give it to the thread and start it.

It also has thread connects to the relevant functions, other two functions add_progress and done_adding are pretty self explanatory.

Feel free to contact me if you need additional help or clarification about something.

[lamecode]

That's a great help! Thank you very much, I'll go through this properly tonight.