How to run asynchronous tasks in Python GObject Introspection apps

Question

I'm writing a Python + GObject app that needs to read a non-trivial amount of data from disk upon start. The data is read synchronously and it takes about 10 seconds to finish the read operation, during which time the loading of the UI is delayed.

I'd like to run the task asynchronously, and get a notification when it's ready, without blocking the UI, more or less like:

def take_ages():
    read_a_huge_file_from_disk()

def on_finished_long_task():
    print "Finished!"

run_long_task(task=take_ages, callback=on_finished_long_task)
load_the_UI_without_blocking_on_long_task()

I've used GTask in the past for this sort of thing, but I'm concerned that its code hasn't been touched in 3 years, let alone been ported to GObject Introspection. Most importantly, it's no longer available in Ubuntu 12.04. So I'm looking for an easy way to run tasks asynchronously, either in a standard Python way or in a GObject/GTK+ standard way.

Edit: here's some code with an example of what I'm trying to do. I've tried python-defer as suggested in the comments, but I could not manage to run the long task asynchronously and let the UI load without having to wait for it to finish. Browse the test code.

Is there an easy and widely used way of running asynchronous tasks and get notified when they're finished?

Answer 1

Your problem is a very common one, therefore there are tons of solutions (sheds, queues with multiprocessing or threading, worker pools, ...)

Since it is so common, there is also a python build-in solution (in 3.2, but backported here: http://pypi.python.org/pypi/futures) called concurrent.futures. 'Futures' are available in many languages, therefore python calls them the same. Here are the typical calls (and here is your full example, however, the db part is replaced by sleep, see below why).

from concurrent import futures
executor = futures.ProcessPoolExecutor(max_workers=1)
#executor = futures.ThreadPoolExecutor(max_workers=1)
future = executor.submit(slow_load)
future.add_done_callback(self.on_complete)

Now to your problem, which is much more complicated than your simple example suggests. In general you have threads or processes to solve this, but here is why your example is so complicated:

1. Most Python implementations have a GIL, which makes threads not fully utilize multicores. So: do not use threads with python!
2. The objects you want to return in slow_load from the DB are not pickelable, which means that they can not simply be passed between processes. So: no multiprocessing with softwarecenter results!
3. The library you call (softwarecenter.db) is not threadsafe (seems to include gtk or similar), therefore calling these methods in a thread results in strange behaviour (in my test, everything from 'it works' over 'core dump' to simple quitting without results). So: no threads with softwarecenter.
4. Every asynchronous callback in gtk should not do anything except sheduling a callback which will be called in the glib mainloop. So: no print, no gtk state changes, except adding a callback!
5. Gtk and alike does not work with threads out of the box. You need to do threads_init, and if you call a gtk or alike method, you have to protect that method (in earlier versions this was gtk.gdk.threads_enter(), gtk.gdk.threads_leave(). see for example gstreamer: http://pygstdocs.berlios.de/pygst-tutorial/playbin.html).

I can give you the following suggestion:

1. Rewrite your slow_load to return pickelable results and use futures with processes.
2. Switch from softwarecenter to python-apt or similar (you probably don't like that). But since your employed by Canonical, you could ask the softwarecenter developers directly to add documention to their software (e.g. stating that it is not thread safe) and even better, making softwarecenter threadsafe.

As a note: the solutions given by the others (Gio.io_scheduler_push_job, async_call) do work with time.sleep but not with softwarecenter.db. This is, because it all boils down to threads or processes and threads to not work with gtk and softwarecenter.

Answer 2

Here's another option using GIO's I/O Scheduler (I've never used it before from Python, but the example below seems to run fine).

from gi.repository import GLib, Gio, GObject
import time

def slow_stuff(job, cancellable, user_data):
    print "Slow!"
    for i in xrange(5):
        print "doing slow stuff..."
        time.sleep(0.5)
    print "finished doing slow stuff!"
    return False # job completed

def main():
    GObject.threads_init()
    print "Starting..."
    Gio.io_scheduler_push_job(slow_stuff, None, GLib.PRIORITY_DEFAULT, None)
    print "It's running async..."
    GLib.idle_add(ui_stuff)
    GLib.MainLoop().run()

def ui_stuff():
    print "This is the UI doing stuff..."
    time.sleep(1)
    return True

if __name__ == '__main__':
    main()

Answer 3

Use the introspected Gio API to read a file, with its asynchronous methods, and when making the initial call, do it as a timeout with GLib.timeout_add_seconds(3, call_the_gio_stuff) where call_the_gio_stuff is a function which returns False.

The timeout here is necessary to add (a different number of seconds may be required, though), because while the Gio async calls are asynchronous, they are not non-blocking, meaning that the heavy disk activity of reading a large file, or large number of files, can result in blocked UI, as the UI and I/O are still in the same (main) thread.

If you want to write your own functions to be async, and integrate with the main loop, using Python's file I/O APIs, you'll have to write the code as a GObject, or to pass callbacks around, or use python-defer to help you do it. But it's best to use Gio here, as it can bring you a lot of nice features, especially if you're doing file open/save stuff in the UX.

Answer 4

You can also use GLib.idle_add(callback) to call the long running task once the GLib Mainloop finishes all it's higher priority events (which I believe includes building the UI).

Answer 5

I think it bears noting that this is a convoluted way to do what @mhall suggested.

Essentially, you've got a run this then run that function of async_call.

If you want to see how it works, you can play with the sleep timer and keep clicking the button. It's essentially the same as @mhall's answer except that there's example code.

Based on this which is not my work.

import threading
import time
from gi.repository import Gtk, GObject
calls f on another thread
def async_call(f, on_done):
    if not on_done:
        on_done = lambda r, e: None
def do_call():
    result = None
    error = None

    try:
        result = f()
    except Exception, err:
        error = err

    GObject.idle_add(lambda: on_done(result, error))
thread = threading.Thread(target = do_call)
thread.start()


class SlowLoad(Gtk.Window):
def __init__(self):
    Gtk.Window.__init__(self, title="Hello World")
    GObject.threads_init()        

    self.connect("delete-event", Gtk.main_quit)

    self.button = Gtk.Button(label="Click Here")
    self.button.connect("clicked", self.on_button_clicked)
    self.add(self.button)

    self.file_contents = 'Slow load pending'

    async_call(self.slow_load, self.slow_complete)

def on_button_clicked(self, widget):
    print self.file_contents

def slow_complete(self, results, errors):
    '''
    '''
    self.file_contents = results
    self.button.set_label(self.file_contents)
    self.button.show_all()

def slow_load(self):
    '''
    '''
    time.sleep(5)
    self.file_contents = "Slow load in progress..."
    time.sleep(5)
    return 'Slow load complete'




if name == 'main':
    win = SlowLoad()
    win.show_all()
    #time.sleep(10)
    Gtk.main()

Additional note, you have to let the other thread finish before it will terminate properly or check for a file.lock in your child thread.

Edit to address comment:
Initially I forgot GObject.threads_init(). Evidently when the button fired, it initialized threading for me. This masked the mistake for me.

Generally the flow is create the window in memory, immediately launch the other thread, when the thread completes update the button. I added an additional sleep before I even called Gtk.main to verify that the full update COULD run before the window was even drawn. I also commented it out to verify that the thread launch doesn't impede window drawing at all.