The Python 3 Wall of Shame

Posted on February 12, 2011 by ubershmekel

Here’s my attempt at motivating package maintainers to port to python 3 and you can check out the code that generates the chart. I basically scraped PyPI (13,000 webpages) for this info and formatted it into an HTML table which is uploaded to appspot. I added a ‘cron’ task on my PC to recrawl PyPI every Sunday so the chart stays fresh. When developing the WOS I used filecache (from the previous blog post) so I could just write the code that parses the crawl as though it just now scraped PyPI while infact everything was cached to disk. Without filecache I would have had to either wait for ages, or write code that stores the scrapes, and reparses them. About 10 lines turn into an import and a decorator, now that’s magic.

I hope no one takes offense from this. The situation we’re starting at is pretty bad. Only 11 out of the top 100 packages on PyPI are labeled as supporting python3. There are a few glitches (e.g. multiprocessing) where the developers simply haven’t yet labeled the package as python 3 compliant.

Once we pass the 50% mark I guess we can change it from “Wall of Shame” to “World of Strength” or something because the subdomain is “wos”.

31 thoughts on “The Python 3 Wall of Shame”

cwillu on February 12, 2011 at 19:15 said:

The situation is under control, and precisely on schedule. 5 years was the target, back in 2008. That’s not to say that a go/no-go list isn’t handy, but the title is inflammatory.

Relevant: http://sayspy.blogspot.com/2011/01/my-semi-regular-reminder-that-python-3.html

Reply ↓
- ubershmekel on February 13, 2011 at 09:12 said:
  
  I hardcoded the title to change to “Python 3 Wall of Superpowers” once it goes past 50%. I’m not joking.
  
  Reply ↓
Noufal Ibrahim on February 12, 2011 at 21:09 said:

We’re working on something similar at http://getpython3.net/ in the same vein as http://isitruby19.com/.

Do take a look and maybe we can collaborate.

Reply ↓
- ubershmekel on February 13, 2011 at 09:03 said:
  
  @Noufal – I think your website was part of my inspiration, so sure. I’m all for collabs.
  
  Reply ↓
Peritus on February 12, 2011 at 21:33 said:

http://pypi.python.org/pypi/zc.buildout is missing, despite having 455304 downloads (as of now)

Reply ↓
- ubershmekel on February 13, 2011 at 09:04 said:
  
  Thanks! I fixed it. I was parsing the first table in the PyPI page and not the last one.
  
  Reply ↓
Brett on February 12, 2011 at 22:44 said:

First off, nice website! Looks good and easily communicates what it needs to.

But there are a couple of things you could do to improve the reporting. One, It isn’t so much that e.g., multiprocessing have not updated their labels, it’s that they never will thanks to the modules being included in the stdlib. I count four projects which have been merged into Python 3.2 (or earlier) and thus are implicitly ported to Python 3.

You also count Django 3 times: 1.2.3, 1.2.1, and 1.2.4. You might want to only consider the latest release instead of (in Django’s case) triple counting it as Django is not about to add support for Python 3 in a micro release.

Lastly, you may want to color projects that specify *no* specific Python version in another way (e.g., docutils). If they are not going to bother to specify even what Python 2 versions they rely on then they are most certainly not going to bother to list Python 3 support. And in docutils’ case it does have Python 3 support.

But thanks for doing this! It’s nice to visibly see this sort of thing.

Reply ↓
- ubershmekel on February 13, 2011 at 09:09 said:
  
  I’m going to hardcode blacklisting those 4 packages soon.
  
  Concerning multiples versions it’s going to be a bit annoying because some packages have multiple pages per version and some have one page with all of them. But I hope I can get to that some time this week.
  
  About especially coloring projects that haven’t designated anything, wouldn’t that miss the whole point?
  
  Reply ↓
  - Michael Foord on February 13, 2011 at 23:40 said:
    
    Well, the problem is that at the moment you’re listing in Red some libraries that *do* have Python 3 support. Isn’t *that* missing the point?
Michael Foord on February 13, 2011 at 18:38 said:

unittest2 has python 3 support through a different distribution (same package name if you install it): unittest2py3k. There are python 3 version of virtualenv and docutils too.

multiprocessing and uuid are both in the standard library now and so aren’t *needed* as separate python 3 versions.

Reply ↓
- ubershmekel on February 13, 2011 at 23:55 said:
  
  I just now hardcoded ‘multiprocessing’, ‘simplejson’, ‘argparse’, and ‘uuid’ as irrelevant to the WOS. So I can manually add ‘unittest2’, ‘virtualenv’ and ‘docutils’. Though I would have preferred an automatic way of figuring this out so I don’t have to manually remove each package that forks…
  
  Reply ↓
  - Michael Foord on February 13, 2011 at 23:58 said:
    
    Yeah, I agree hardcoding is nasty. Unless pypi provides me with metadata to describe the fact that unittest2 for python 3 is a different package I don’t see how to do it though. It would certainly be incorrect to say that the unittest2 distribution itself works with python 3 (it doesn’t).
    
    As it happens I’m *considering* switching to a single distribution (shipping both source code packages in a single distribution and deciding at install time which to install – this isn’t without its own problems however).
Alex Grönholm on February 14, 2011 at 16:11 said:

A couple notes:
– distribute is the py3k compatible version of setuptools, so setuptools shouldn’t be listed at all (it will never turn green)
– pymongo was ported to py3k by yours truly (it’s named pymongo3 on pypi) and it might take a very long time before the package under the “pymongo” name will be py3k compatible

Reply ↓
- ubershmekel on February 16, 2011 at 00:34 said:
  
  As noted, we need a way to link from py2 packages to their py3 counterparts or to devise a way of containing both packages in one setup.py. We should ask python-dev about this.
  
  Reply ↓
  - Alex Grönholm on February 16, 2011 at 05:10 said:
    
    Neither method will fix the setuptools showing up as red issue. The author of setuptools has essentially refused to acknowledge distribute as the legitimate successor to setuptools, despite the project being led by the distutils maintainer. You may still have to hardcode this case.
Jean-Paul Calderone on February 15, 2011 at 13:39 said:

pyOpenSSL supports Python 3, but you have it listed as red. How useful is the website if so much of the data it presents is wrong?

Reply ↓
- ubershmekel on February 15, 2011 at 18:05 said:
  
  If you’ll add the python 3 classifier to pyOpenSSL then you’ll get the right color. I’m sorry for the inconvenience. Is there another way I could have extrapolated pyOpenSSL supports py3k? If there is then I don’t mind implementing it.
  
  Reply ↓
  - Alex Grönholm on February 15, 2011 at 18:09 said:
    
    Likewise, is there a way to indicate that a distribution is the py3k counterpart for another? See my previous comment about two distributions where this would be necessary.
Joaquin Abian on March 1, 2011 at 19:56 said:

numpy has python 3 support from long ago. There is a python 3.1 package in pypi. However in your list it is indicated in red.

Reply ↓
- ubershmekel on March 2, 2011 at 00:07 said:
  
  You’ll notice that currently numpy doesn’t have the python3 classifier. I wanted to notify the devs but then noticed https://github.com/numpy/numpy/blob/master/setup.py that it’s already taken care of. So next version of numpy will turn green.
  
  Reply ↓
Mike Verdone on March 5, 2011 at 18:28 said:

What about packages that change name when they go to Python3? e.g., virtualenv is Python 2 only, but the virtualenv5 fork is Python 2 and 3 compatible. PyPI makes it a total pain in the ass to release Py2 and Py3 libraries under the same name but with different source packages. e.g. my twitter package is Python2 and twitter3 is Python3.

Reply ↓
- Alex Grönholm on March 13, 2011 at 10:48 said:
  
  For some distributions, like setuptools, this is an appropriate concern. For virtualenv, however, it is not because virtualenv is actively developed and could get official Python 3 support as early as next week (during the PyCon sprints).
  As for your twitter package, could you not use two different source directories for py2 and py3 respectively and choose which one to use in your setup.py based on the Python version used? I recall that there are some distributions using this packaging style already (although I can’t name any).
  
  Reply ↓
Doug on March 13, 2011 at 04:17 said:

How is this for wall of shame…PyMongo will not install with Python 3.2 installed. The installer insists that Python 3.1 is required.

I found this website while googling with the following search string:
“pymongo3 does not install with python 3.2”

Reply ↓
- Alex Grönholm on March 13, 2011 at 10:32 said:
  
  The Windows installers for Python 3.2 were missing, that’s all. Those installers have now been uploaded to PyPI.
  
  Reply ↓
Doug on March 13, 2011 at 04:57 said:

Excuse me… PyMongo was the second of the 2 problems.

distribute will not work with python 3.2 either. I had to use version 6.12 and modify the sandbox file. See the following:

http://stackoverflow.com/questions/2831231/howto-install-distribute-for-python-3

FYI, 6.14 still gives a syntax error despite the correction being made to the file by the author.

Reply ↓
ericnathanwilson on July 6, 2011 at 13:30 said:

I think it would be encouraging if we could have a “Wall of Progress” that showed which libraries went from red to green in the last week, month, or year.

Reply ↓
Pingback: Python 3 Library updates, via twitter « Code and comments
Carl on September 7, 2011 at 00:42 said:

IPython does have a Python3.x branch. I use IPython0.11 with Python3.2 everyday, and it works really well. You can use it online at PythonAnywhere dot something. They should be made green.

Reply ↓
nanu on December 12, 2012 at 05:17 said:

The WoS shows xlrd and xlwt as red. These have been ported to python 3, but as new, renamed packages:
http://pypi.python.org/pypi/xlwt3/
http://pypi.python.org/pypi/xlrd3/

Reply ↓
- ubershmekel on December 12, 2012 at 21:19 said:
  
  These ports seem to have been abandoned, no?
  
  Reply ↓
JS on December 12, 2012 at 13:14 said:

A very useful Python Imaging Library (PIL) is missing from the list.

Reply ↓