This article is part of a series of tearful articles about Python packaging:
- The Can of Worms
- The Roots of Evil
- Delusions of Formats
- Files Everywhere
- The Toolbox
- The Expression of Needs
- The Minimal Solution
Before starting, we would like to send a lot of love to the members of the PyPA team. We complain a lot in this series, but we have a lot of respect for the sisyphean work already done.
That being said, let’s start (again) the whining 😭.
Where Do We Put What?
We saw in the previous article what we can put in our packages. That’s a lot of stuff to include, but we haven’t really defined where or how we can include it.
As this article is just an example among others, we won’t have any qualms to propose purely subjective opinions based on a peculiar sensitivity. It’s not the absolute truth, it’s just an opinion. But, of course, as it’s our own, it’s a good one, otherwise you would be reading something else.
The Wheel Package
The wheel package is the level 1 of packaging. It’s not only the simplest to create, but also the one that’ll be the most used.
The goal of this package is to provide the functional code already ready
for the target. The
pip command that will install it will
have nothing to do apart from decompressing the archive, retrieving
some information (like dependencies) and putting the decompressed
folder on the disk. No compilation, no arbitrary Python code execution,
no special case for a particular platform.
Incidentally, this means that if your code isn’t the same depending on the Python version or the operating system, you’ll have to create a different package for each combination on which your creation will be installed. So, you’ll have to work a little bit.
It doesn’t matter: we have all the tools we need to do that. What is
important is to agree on what we want to put inside. On this subject,
our opinion is simple: code, metadata, and maybe one or two files like
README or the license.
No need to cry. Who installs a wheel to look inside at the
CHANGELOG? Who installs a wheel to launch unit tests? Who
installs a wheel to rebuild the documentation of the module? If you
already did that, the time has come for you to discover an amazing tool
called Internet. It’s full of nice pages presenting the documentation
with colors, pictures and even links.
More seriously: the only purpose of wheels is to be installed. No
matter how they work, what they contain, the most important thing is
that their installation works properly with
Everything else is secondary. Really. Goodbye tests, goodbye
documentation, goodbye configuration files.
The Source Package
Sources are the basis, the alternative, the last resort, the ultimate comprehensiveness, and that’s why wheels will never send into oblivion this venerable format that has been supported by Python users since the beginning of package sharing.
Before defining what we’ll put inside, let’s settle a crucial issue right away: what will our source package be really used for? The answer is simple:
- reading sources,
- creating packages for other package managers (for example, for Linux distributions),
- installing packages, for hopeless cases.
For these cases, it’s often useful to have more information than what’s
included in the wheel. We like to have some files to discover changes
between versions, some tests to see how it works, some documentation
that can be read with a text editor… We may want to see how the package
is configured, to play with the
setup.py file, or even
to try to modify the code.
So, the source package will contain more or less the same content as
the versioned repository. We have, on broad terms, to retrieve what
we’d retrieved with
git clone (or the equivalent command
of your favorite version control software, no offense). Of course,
we’ll remove the versioning data and some details like the CI
With all these details, we’re now able to venture into package creation. We won’t explain each line of configuration or code to write, but we’ll try, at least, to lay the necessary foundation for the creation of your packages.
Simple Package, Simple Solution
We’ll get rid, with only a few words, of a huge unspoken fact that clutters us. You want to build a package? Let’s say that your code contains only Python, and that you want to follow the rules we’ve settled before. There, it’s already better.
Do we agree? Otherwise you can console yourself with another article,
somewhere on Internet, about
setuptools. We leave that to
Without further suspense, the tool we’ll use is Flit.
Flit is the hammer of package creation. It’s limited, it only does one thing, but it’s clear, limpid, efficient. We want to create and share packages with simple and foolish rules, that’s all.
Flit is also one of the tools behind
PEP 517 and
PEP 518. Yes,
it’s thanks to its creator Thomas Kluyver that now we can be free from
Flit is simplicity above all. If you don’t want to ask questions about package creation, if you don’t want to write more code than your module, it’s the choice of reason.
Forget the tons of files and endless configurations. Here, we aim simplicity, we’ll have to eradicate the obesity of root folders. We’re going to drastically lighten the home page of your repository.
In the past, we took examples of staggering overdoses. Without going back into all the projects we’ve already talked about, let’s take only one example of what we don’t want: Requests.
(Requests isn’t the ultimate evil, don’t make us say what we didn’t write. It’s just a good example of want we don’t want.)
The root folder contains 22 files and folders. Among them, we find the
usual suspects of package creation:
We also find nice configuration files for third-party tools: Tox,
Coverage, Pytest. Some metadata, some folders, here’s something to
impress a person who would like to use this famous project as an
example to understand how Python modules work.
In contrast to this project, we suggest 6 basic folders and files to include at the root of your folder:
- the folder containing the code of your project,
Of course, this is only a basis that you can adapt to your needs. But restricting ourselves to keep a light and clean root folder is also a good reason to think about the hygiene and the structure of projects. Let’s see what we can put in these little boxes…
In the folder containing the code of your project, you’ll first put… the code of your project. This seems to be obvious, but if we follow our idea of having a minimal wheel, we quickly understand that this folder will be the one ready to be decompressed. For the simple goal of installing the module, the rest is just decoration.
A consequence of this division is that this folder has to include the additional files required for the module. Pictures for your game? Include them in this folder. A lists of famous passwords for your NSA hacking tool? Include them in this folder.
That also explains why we don’t include tests or documentation in this folder. Everybody knows that tests and documentation are useless when the code is limpid and bug-free. However, in doubt, until all humans are omniscient, we could keep these relics, but only in the source package.
Tests, if they follow the naming conventions of your favorite tool,
will be automatically discovered. For this purpose,
seems to be an appropriate name, for humans and for tools (Flit or
Pytest, for example). Then, up to you to organize your tests as you
want, but you’ll help everyone by putting them in the same folder, at
the root of your project.
The documentation has, also, good reasons to be put in the source package. You give to the curious people the possibility to dig into the clear explanations about your project, next to your code, without Internet access requirement. You give to the Linux distributions the possibility to include an appreciable introduction to your tool, but also to optional manual pages. In fact, you give the possibility to anyone to do anything with content that helps people, and in this case you can never be safe from pleasant surprises.
This documentation is also the perfect place to store some information
that we may want to put in the root, like a
configuration samples. So, they’ll be available in a pleasant format,
in addition to text files. Code hosting platforms also offer dedicated
pages to some of this information, so that a lot of root files become
unnecessary. Nothing stops you, if you really want to, to put links
README file to guide people who only look at
the root folder.
README file is the basis of your project. In pure text
format or with a light markup, it’s the entry point of the majority of
people interested in code. There’s also a good chance that the file is
brought to the fore by your software forge and by PyPI.
That’s a good reason to work on your
README file. Beyond
the project description, you have to point all the necessary
information we like to quickly have when we discover a project: the
license, the supported versions of Python, the way to contact people
taking care of the project…
By the way, putting the license at the root of the project, in a
specific file, is very classic but questionable. After all, couldn’t
this legal information be in the documentation? Wouldn’t a line in the
README be enough to indicate which license applies to the
Yes, probably. But many tools expect to find this file at the root, and some of them even read it to deduct the license of the project. If it’s easy to change the habits of people, who’ll be happy with an indication in the documentation, it’s more complex to change the habits of machines. So… Let’s say this choice is a small agreement between ideals and reality. Let’s work to ensure that in some years we can get rid of this more simply.
At last, the main dish:
This file allows you to indicate
everything required to create a package.
The default choices of Flit being really good (objectively), you
shouldn’t need to change a lot of things to the proposed values. But
you should know that if you want to, you’ll find a long list of options
that will satisfy all your crazy ideas.
pyproject.toml is going to replace what you can
do with (at least)
MANIFEST.in. Of course,
possibilities are limited, just because you can’t write Python code to
commit atrocities executed during the creation or the installation of a
package. But it’s not a limitation, it’s a feature: stop playing with
this idea, interesting at the beginning but becoming more than filthy.
It may be more useful for the posteriority to write your module.
This file will also allow you to configure most of the third-party tools you use: Tox, Black, Pytest, Coverage.py, isort, Pylint… Yes, it means that you can say goodbye to this pile of configuration files, each using its own naming conventions and formats! The list of supported tools is getting bigger, don’t hesitate to have a look time to time to see if your favorite project dared to take the plunge.
From Creation To Deployment
Don’t expect a tutorial where we hold your hand, write your configuration files, and give you all the tricks to use Flit. The title of the article isn’t "7 things you don’t know about Flit—the 5th one will surprise you."
Why? Simply because the Flit’s documentation is amazing. You’ll find all you need to install and use Flit (almost) blindly. It’s limpid, it’s quick, and mainly it works.
publish. It’s all you need to model you little package
with love. No more need to suffer our derogatory prose, we leave you
among the delicate words of a delicate tool.
Enjoy flying on your own, let the wind .
Don’t lie to ourselves: Flit doesn’t solve everything. We already saw its limitations and its small wicknesses, but unfortunately there’s more.
Do you want another tool? Poetry can do the same as Flit, but it does more: managing virtual environments, solving dependencies, installing packages, numbering versions… If you like all-in-one tools that avoid the pitfall of becoming a tentacled and megalomaniac behemoths impossible to maintain (we won’t give names), you can find in Poetry an elegant substitute to Pipenv (oops, sorry).
But… whether it’s on Flit or Poetry, there’s a big shadow: abandonment.
Flit and Poetry are widely used, but they stay third-party tools that
aren’t as supported as
setuptools is. As well
as many open-source projects, they already go through some
and there will be other ones.
Fortunately, PEPs are now widely adopted, leaving the door open to
other future tools. Out of the shackles of
can use other tools based on
pyproject.toml. The keys and
the values of options will change, but at least we won’t need to depend
on a unique implementation stuck by the weight of legacy and the need
of backwards compatibility.
Wheels are wheels, sources are sources, and they lived happily ever after.
Tools change, but formats remain.