This article is part of a series of tearful articles about Python packaging:
- The Can of Worms
- The Roots of Evil
- Delusions of Formats
- Files Everywhere (forthcoming)
- The Toolbox (forthcoming)
- The Expression of Needs (forthcoming)
- The Minimal Solution (forthcoming)
Before starting, we would like to send a lot of love to the members of the PyPA team. We complain a lot in this series, but we have a lot of respect for the sisyphean work already done.
That being said, let’s start (again) the whining 😭.
What’s the Problem?
If you are here, we can assume that you’ve already written some lines of
Python. You’ve already installed a Python project, with dependencies.
And for that, you may have had to do some stuff. A lot of stuff. Way
too much stuff. Not so much? We bet that you’ve already encountered
wheel… We can stop here, but you
know many more, without even realizing.
That’s too much, and that’s not normal.
A sentence from The Zen of Python says "There should be one—and preferably only one—obvious way to do it". You see the problem now, right?
The Horror of the Beginnings
You may not remember, but at some point you didn’t know about Python. Yes, it’s true.
Try to explain to someone at ease behind a keyboard (but who doesn’t know Python) how to install a Python program. In his or her place, as an average neophyte of the XXIe century, we would expect something like:
Install Python by following the super short tutorial from the super
documentation, or with the command
Install the program by clicking here, or with the command
Sorry to crush your dreams: reality is not like that. Not at all. The "super
short tutorial from the super documentation" to install Python doesn’t
exist. And too many
programs exist, all incompatible with each other, and not included in
Python. That’s why installing a program is unpleasant.
Yeah: 🤦. We agree.
Wow, Such Tutorial
Speaking about tutorials, one of the best sources is, indubitably, The Hitchhiker’s Guide to Python. A lot of things can always be said about documentation, but let’s bring a touch of optimism: the guide is translated in several languages, and some parts are really well written in a rather didactic way.
We won’t talk about the fact that it’s not the official documentation. We won’t talk about the fact that the whole guide is a not-so-well-hidden advertisement for the glory of the projects of its arguable author. No, don’t insist, we won’t do that.
Problem: this tutorial is not short, and not simple. You’re using macOS? Here is the first paragraph:
Before installing Python, you’ll need to install GCC. GCC can be obtained by downloading Xcode, the smaller Command Line Tools (must have an Apple account) or the even smaller OSX-GCC-Installer package.
We save you from the hell 🔥 of the installation on Windows, where more
or less wobbly solutions are listed. Python is in the Store (that’s
what Microsoft says),
in Chocolatey (that’s
what the Hitchhiker’s Guide says in English,
not in French),
but not its documentation).
What’s the best solution? We don’t know. In fact there’s none. And that’s very bad.
Wow, Such Installer
OK, the first point was kind of a disaster. But… Maybe we can do better
for the second one? Installing a package after installing Python should
be a child’s play. We do
pip install my-package, don’t we?
sudo pip install my-package?
No. No. No. No. No. No. No. No.
To install a package, you need to create a virtualenv, but, well, not always. To create a virtualenv, you’re going to use a module integrated in Python, but not always. And after that, we’re going to use a program integrated in Python, but not always. And after that, the package can be installed, but not always.
If you don’t want to complicate things, you can follow what the Hitchhiker’s Guide says and use Pipenv (we let you guess who created this wonderful tool). Of course you have to disregard the fact that Pipenv has been deserted during a year and a half in 2019 with a broken version and no update. Of course, you have to turn a blind eye to the fact that the project is sprawling, includes about forty (40!) duplicated (and sometimes modified) projects, and contains more than 200.000 lines of code 😱…
Hahahahahahaha. Haha. No.
The Loneliness of Package Creation
And we’ve just installed a package. If you want to create and share a package, summon up your patience, expect other surprises.
Let’s be honest: a tutorial and a guide exist to learn how to create packages. These documents are rather well written and should quickly help you to share your library. In theory, we’re taken by the hand. In theory.
In reality, things are way more chaotic. If we look at only three
NumPy), we have a long list
of files more or less related to package managment, without counting
LICENSE, and so on:
Bad news: Python doesn’t provide tools to create a new project ready to be shared. You’re going to be lost in third-party projects that create packages their own way (we’re still looking askance at you, Pipenv), or you’re going to learn the hard way what these weird files are used for.
(Small funny note:
among others, to list the dependencies of a project. They use 4
different file formats, and one of them is not even managed by the
Python standard library. True story.)
The different tutorials and online forums will only give you fragmented
(and often obsolete) information. And there’s a reason: good practice
is always changing. These tools have often been developed with no
specifications, with documentation at best incomplete (we are really
thinking about you,
setuptools). Everybody suggests ideas, new tools regularly
appear, everything quickly looks like a fashion moment that becomes
outmoded after a few years (even a few months if you’re unlucky)…
We ear you defending your beloved language: "Yes, but this stuff isn’t
that complicated, blah blah blah blah…". Do you want to cry? Go on
the Cargo documentation,
Rust’s package manager.
The first page
is the installation page, it contains 136 words (according to
including commands. It allows you to install Rust, with Cargo. 136
words, it’s one less than the Zen of Python. For real.
The second page gives you everything needed to create a simple package, with a tool that creates all the files. It contains the source code, the metadata file, the commands used and their results, a tree of files and folders in ASCII-art. Everything in 183 words.
There we are. It is possible. You know that, now.
And After That?
With Python, you’re going to learrn things randomly from your online reading, on doubtful blogs like this one. You’re going to find absolutely outdated solutions on StackOverflow, unspecified tricks using implementation details which will change next week. Do we include tests in packages, or not? Do we put them in the module, or not? Do we supply a source package, or not? Do we indicate exact versions of the dependencies, or not? Do we put the documentation in a folder aside, or not? So many questions, and many more, for which you’re in the deep loneliness of packages creation.
You probably think that everything is lost, that the people who developed everything are incompetent, that Python is in fact a language rotten beyond repair… Actually, no. We’ll try in the next episodes to understand how we got there, and how things changed during all those years, often for the best. Will we, one day, succeed in creating a nice package? Of course, we will.
To be continued…