Criticism This Python Code (crawler With Threadpool)

how good this python code ? need criticism) there is a error in this code, some times script do print 'ALL WAIT - CAN FINISH!' and freeze (no more actions are happend..) but i can'

Solution 1:

You are sharing state between threads (i.e., in is_all_wait) without synchronization. Plus, the fact that all threads are "waiting" is not a reliable indicator that the queue is empty (for instance, they could all be in the process of getting a task). I suspect that, occasionally, threads are exiting before the queue is truly empty. If this happens often enough, you will be left with tasks in the queue but no threads to run them. So queue.join() will wait forever.

My recomendation is:

1. Get rid of is_all_wait -- it's not a reliable indicator
2. Get rid of the task state -- it's not really necessary
3. Rely on queue.join to let you know when everything is processed

If you need to kill the threads (for example, this is part of a larger, long-running program), then do so after the queue.join().

Solution 2:

I have basic python knowledge but threading in python isn't useless? I've seen tons of articles criticizing the global lock interpreter for this.