[2016-09-03] A process pool in python

When you need to spawn a lot of processes, so many that you don't want to spawn them at the same time this PopenPool can help you.

    def run(self,):
        while self.running_tasks or self.upcomming_tasks:
            while self.upcomming_tasks and len(self.running_tasks) < self.pool_size:
                self.consume_task()
            still_running_tasks = deque()
            while self.running_tasks:
                task = self.running_tasks.popleft()
                if task.poll() is None:
                    # task is not finished yet
                    still_running_tasks.append(task)
                else:
                    self.finished_tasks.append(task)
            self.running_tasks = still_running_tasks

This code will spawn the maximum number of processes and as soon one of them finishes the next one is spawned.

The full source can be found here

python
python-2
python-3