The blog does not allow removing old posts and does not handle hard spaces properly at all places. I hope you can decipher my code nevertheless.

Alfe

class Worker(threading.Thread):
    def __init__(self, taskQueue, resultQueue):
self.taskQueue = taskQueue
self.resultQueue = resultQueue
super(threading.Thread, self).__init__()
def run(self):
while True: # until end marker found
try:
url = self.taskQueue().get(True)
if url is None: # end-marker?
self.resultQueue.put(self) # report being finished
f = urllib.urlopen(url)
contents = f.read()
f.close()
self.resultQueue.put(contents)
except Exception, e:
self.resultQueue.put(e)

def get_files(urls):
taskQueue = Queue()
for url in urls:
taskQueue.put(url)
resultQueue = Queue()
workers = set([ Worker(taskQueue, resultQueue) for i in range(4) ])
for worker in workers:
worker.start()
taskQueue.put(None) # one end marker for each worker thread
while workers:
result = resultQueue.get(True)
if result in workers:
workers.remove(result)
else:
yield result

class Worker(threading.Thread):
    def __init__(self, taskQueue, resultQueue):
        self.taskQueue = taskQueue
self.resultQueue = resultQueue
        super(threading.Thread, self).__init__()
    def run(self):
while True: # until end marker found
try:
url = self.taskQueue().get(True)
if url is None: # end-marker?
self.resultQueue.put(self) # report being finished
f = urllib.urlopen(url)
contents = f.read()
f.close()
self.resultQueue.put(contents)
except Exception, e:
self.resultQueue.put(e)

def get_files(urls):
taskQueue = Queue()
for url in urls:
taskQueue.put(url)
resultQueue = Queue()
workers = set([ Worker(taskQueue, resultQueue) for i in range(4) ])
for worker in workers:
worker.start()
taskQueue.put(None) # one end marker for each worker thread
while workers:
result = resultQueue.get(True)
if result in workers:
workers.remove(result)
else:
yield result

Personally I would stick the jobs to be performed (in this case URLs to be downloaded) in a queue then create n (in this case 4) threads to work through that queue. If necessary, these threads can place the results in another queue to be processed by some other number of consumer threads.

This solution would have the huge advantage that jobs can be added as you go along, which is absolutely crucial if you want to chain a few of these things together.

I also do not understand the point of creating a Thread class to replace the get_files function. Ostensibly you do this to make things simpler, but actually I think it’s more complicated (at least for illustration purposes). Potentially you might want your tasks to be objects so your pool of threads would take task objects from a queue, do something to those objects and put the “finished” objects into another queue for further processing. But I can’t see why you need your tasks to be thread objects.

and

urllib.urlopen(self.url)

Also, the main page is at: Scruffy Thinking