[2016-09-03] A process pool in python

When you need to spawn a lot of processes, so many that you don't want to spawn them at the same time this PopenPool can help you.

    def run(self,):
        while self.running_tasks or self.upcomming_tasks:
            while self.upcomming_tasks and len(self.running_tasks) < self.pool_size:
            still_running_tasks = deque()
            while self.running_tasks:
                task = self.running_tasks.popleft()
                if task.poll() is None:
                    # task is not finished yet
            self.running_tasks = still_running_tasks

This code will spawn the maximum number of processes and as soon one of them finishes the next one is spawned.

The full source can be found here