Cleaning up Python threads

Compared to a lot of other threading environments, Python threads are pretty good, but there are a couple of features that annoy me. Fortunately, you can fix them with a little clean-up code.

Semaphore methods

First, the methods for Python semaphores are called acquire and release, which is a perfectly reasonable choice, but after working on this book for a couple of years, I am used to signal and wait. Fortunately, I can have it my way by subclassing the version of Semaphore in the threading module:

import threading
 
class Semaphore(threading._Semaphore):
    wait = threading._Semaphore.acquire
    signal = threading._Semaphore.release

Once this class is defined, you can create and manipulate Semaphores using the syntax in this book.

mutex = Semaphore()
mutex.wait()
mutex.signal()

Creating threads

The other feature of the threading module that annoys me is the interface for creating and starting threads. The usual way requires keyword arguments and two steps:

import threading

def function(x, y, z):
    print x, y, z

thread = threading.Thread(target=function, args=[1, 2, 3])
thread.start()

In this example, creating the thread has no immediate effect. But when you invoke start, the new thread executes the target function with the given arguments. This is great if you need to do something with the thread before it starts, but I almost never do. Also, I think the keyword arguments target and args are awkward.

Fortunately, we can solve both of these problems with four lines of code.

class Thread(threading.Thread):
    def __init__(self, t, *args):
        threading.Thread.__init__(self, target=t, args=args)
        self.start()

Now we can create threads with a nicer interface, and they start automatically:

thread = Thread(function, 1, 2, 3)

This also lends itself to an idiom I like, which is to create multiple Threads with a list comprehension:

threads = [Thread(function, i, i, i) for i in range(10)]

Handling keyboard interrupts

One other problem with the threading class is that Thread.join can’t be interrupted by Ctrl-C, which generates the signal SIGINT, which Python translates into a KeyboardInterrupt.

So, if you write the following program:

import threading, time

class Thread(threading.Thread):
    def __init__(self, t, *args):
        threading.Thread.__init__(self, target=t, args=args)
        self.start()

def parent_code():
    child = Thread(child_code, 10)
    child.join()

def child_code(n=10):
    for i in range(n):
        print i
        time.sleep(1)
    
parent_code()

You will find that it cannot be interrupted with Ctrl-C or a SIGINT¹².

My workaround for this problem uses os.fork and os.wait, so it only works on UNIX and Macintosh. Here’s how it works: before creating new threads, the program invokes watcher, which forks a new process. The new process returns and executes the rest of the program. The original process waits for the child process to complete, hence the name watcher:

import threading, time, os, signal, sys

class Thread(threading.Thread):
    def __init__(self, t, *args):
        threading.Thread.__init__(self, target=t, args=args)
        self.start()

def parent_code():
    child = Thread(child_code, 10)
    child.join()

def child_code(n=10):
    for i in range(n):
        print i
        time.sleep(1)

def watcher():
    child = os.fork()
    if child == 0: return
    try:
        os.wait()
    except KeyboardInterrupt:
        print 'KeyboardInterrupt'
        os.kill(child, signal.SIGKILL)
    sys.exit()

watcher()
parent_code()

If you run this version of the program, you should be able to interrupt it with Ctrl-C. I am not sure, but I think it is guaranteed that the SIGINT is delivered to the watcher process, so that’s one less thing the parent and child threads have to deal with.

I keep all this code in a file named threading_cleanup.py, which you can download from greenteapress.com/semaphores/threading\_cleanup.py

The examples in Chapter [pysync] are presented with the understanding that this code executes prior to the example code.