The GIL Detector

Ever wonder if your flavor of python has a Global Interpreter Lock? The recipe gil_detector.py checks that.

In this day and age CPython is still core-locked; though we are seeing improvements thanks to Antoine Pitrou and other great people of the Python community. Wanting to measure just how bad the problem is – my way of looking at it was through a number I call the “python effective core count”.

Python Effective Core Count

How many cores does python see? If you bought a quad core, how many cores can each python process utilize? Measuring how long it takes to complete a given amount of work, W, then measuring how long it takes for python to run 2W using 2 threads, 3W on 3 threads, etc. The script gil_detector.py calculates:

effective_cpus = amount_of_work / (time_to_finish / baseline)

Where the baseline is the time_to_finish for 1 work unit. E.g. if it took the same amount of time to finish 4W (amount_of_work = 4) on 4 threads as it took 1W on 1 thread – python is utilizing 4 cores.

Results

I recommend reading the whole output to see the exact numbers.

Implementation Effective Core Count
Jython 2.5.2 3.8/4 cores
IronPython 2.7 3.2/4 cores
PyPy-1.5 1.0/4 cores
Stackless Python 3.2 1.0/4 cores
CPython 3.2 1.0/4 cores
CPython 2.7 0.2/4 cores

Basically, Jython has the best multithreading with IronPython not too far behind. I know multiprocessing is really easy in python, but it’s still harder. We have to solve this before 8 core CPU’s become the standard at grandma’s house. Those other languages (C/C++) utilize 3.9-4.0 cores of a quad core machine easily, why can’t we? An honorable mention goes to PyPy which by far was the fastest to execute the benchmark (10x faster). PyPy is definitely the future of Python, hopefully they can save us all. A note about CPython 2.7 – yes that number is under 1 because adding threads on python-cpu-intensive tasks hurt performance badly on older versions of the GIL, Antoine fixed that on CPython 3.2.

In my opinion the community should call this a bug and have a unittest in there yelling at us until we fix it, though it’s easy for me to complain when I’m not the core-dev. Maybe I can join PyPy and see what I can do when I find some free time. Hopefully so.

Edit – updated gil_detector.py as corrected at reddit.

What python lacks

I do like python a lot, but I feel some things are still missing. Correct me if I’m wrong.

* How do I know what can I import? I can use help(os) but how do I know that ‘os’ exists (from within the interpreter). I want a command like dir that can search through the default import directories. “mdir” (like module dir) or “pdir” (package dir).
* Tkinter sucks, why isn’t wx or anything else included?
* The php documentation really makes me jealous because the user contributed comments are VERY useful. Why doesn’t python docs have some web 2.0 in it? Like digg’s comment system.
* So many crappy IDE’s, I can’t find one that’s good for everything. At first I liked Wing but nowadays it’s way too slow and heavy so I switched to SPE which is great except it can crash for no reason.
* The GIL needs to be taken more seriously with a dedicated page explaining why it’s in place, how to escape it’s suckiness, and a roadmap to minimize the damage. Alot of people care about performance more than idiot-proof concurrency.

And I think i have a few more rants to go on about this, I’ll post ’em when I recall.