Implement a priority queue

question

You want to implement a queue that sorts elements with a given priority and returns the element with the highest priority each time a pop operation occurs.

solution

The following class implements a simple priority queue using the heapq module

import heapq

class PriorityQueue:
    def __init__(self):
        self._queue = []
        self._index = 0

    def push(self, item, priority):
        heapq.heappush(self._queue, (-priority, self._index, item))
        self._index += 1
        
    def pop(self):
        return heapq.heappop(self._queue)[-1]

Here is an example of how to use this class:

>>> class Item:
... 　　def __init__(self, name):
... 　　　　self.name = name
...
... 　　def __repr__(self):
... 　　　　return 'Item({!r})'.format(self.name)
...
>>> q = PriorityQueue()
>>> q.push(Item('foo'), 1)
>>> q.push(Item('bar'), 5)
>>> q.push(Item('spam'), 4)
>>> q.push(Item('grok'), 1)

>>> q.pop()
Item('bar')
>>> q.pop()
Item('spam')
>>> q.pop()
Item('foo')
>>> q.pop()
Item('grok')

Note that the element returned the first time the pop() operation is performed has the highest priority. We also observe that two elements with the same priority (foo and grok) are returned in the same order as they were inserted into the queue

Discuss

The core of the code snippet above lies in the use of the heapq module. The functions heapq.heappush() and heapq.heappop() implement the insertion and removal of elements from the list _queue, respectively, and ensure that the first element in the list has the lowest priority (as described in the previous section). The heappop() method always returns the "smallest" element, so this is the key to getting the right element to pop from the queue. In addition, since the complexity of both push and pop operations is O(logN), where N represents the number of elements in the heap, these operations are very efficient even if the value of N is large.
In this code, the queue is composed of elements (-priority, self._index, item). Negative priority values are used to allow the queue to be arranged in order of element priority from high to low. This is the opposite of the normal heap queue order, where heaps are generally sorted in ascending order.
The purpose of the variable index is to arrange elements with the same priority in the proper order. By maintaining an ever-increasing index, elements are arranged in the order in which they were enqueued. However, index also plays an important role in comparing elements with the same priority
To illustrate that Item instances cannot be compared sequentially, let's look at the following example:

>>> a = Item('foo')
>>> b = Item('bar')

>>> a < b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: Item() < Item()
>>>

If the elements are represented as a tuple (priority, item), they can be compared only if they differ only in priority. However, if the two tuples have the same priority value, the comparison will still fail as before. E.g:

>>> a = (1, Item('foo'))
>>> b = (5, Item('bar'))
>>> a < b )
True
>>> c = (1, Item('grok'))
>>> a < c
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: Item() < Item()
>>>

This problem can be completely avoided by introducing an extra index value to build a tuple in the form of (priority, index, item). Because no two tuples will have the same index value (once the result of the comparison operation can be determined, Python will not compare the remaining tuple elements)

>>> a = (1, 0, Item('foo'))
>>> b = (5, 1, Item('bar'))
>>> c = (1, 2, Item('grok'))
>>> a < b
True
>>> a < c
True

If you want to use this queue for inter-thread communication, you also need to add appropriate lock and signal mechanisms.

Implement a priority queue

Guess you like