Python project progression

Things you may discover as your Python application grows over time

Getting started

So, you have discovered Python. You've singlehandedly written a useful program using only the standard library with relative ease, without the usual fuss associated with learning a new programming language and setting things up. You've found this experience quite enticing.

Managing dependencies

You've decided grow your program, which requires using a few third-party libraries. First, you need to decide how you'd like to manage your dependencies. You'll likely find several ways to do this, each providing some benefit that the previous method lacked, but also accompanied with its own shortcomings, in approximately this order:

  • Use your system's package manager
    • pro: doesn't require any extra software to manage dependencies
    • con: unless you or anyone running your code uses Windows
    • con: you're (typically) at the mercy of the singular version of a given library provided by your package manager (i.e. you do not get to choose the version)
  • Manually install things with pip
    • pro: straightforward
    • pro: you can choose the version of each direct dependency
    • con: imperative package management isn't great for repeatability or documentative purposes
  • Maintain a requirements.txt file containing your direct dependencies and use pip install -r requirements.txt to install your dependencies
    • pro: declarative package management
    • con: no control over transitive dependency versions between pip installs
  • Maintain a requirements.in file containing your direct dependencies, use pip-compile from pip-tools generate a locked requirements.txt file, and then use pip install -r requirements.txt to install your dependencies
    • pro: transitive dependency version pinning
    • con: no way to separate dependencies only used for tests/examples versus those required for library publication or application deployment
  • Do the above twice, once for runtime requirements and a second (with different filenames, probably like requirements{,-dev}.{in,txt}) files
    • pro: seperation of runtime and test/example dependencies
    • con: litters the root of your project with four files instead of just two
  • Use poetry
    • pro: you're back to only needing two files
    • con: it isn't exactly standard tooling
    • con: it tries to do a lot more than just manage dependencies

Managing dependencies part two

There's a big con of all of the above options (except poetry) that I neglected to mention: the act of installing packages is a global operation (by default), and could potentially break other Python software you're using or developing if the stars don't align (which they won't). Luckily, there are solutions to this problem; unluckily, there are a lot of them:

  • pyenv
    • manages versions of CPython per project
  • pyenv-virtualenv
    • an extension for the above
    • isolate things installed via pip per project
  • pyenv-virtualenvwrapper
    • extension for the above
    • adds some extra commands
  • virtualenv
    • isolate things installed via pip per project
  • virtualenvwrapper
    • extension for the above
    • adds some extra commands
  • pyvenv
    • isolate things installed via pip per project
  • pipenv
    • isolate things installed via pip per project
    • manages a lockfile
  • venv
    • isolate things installed via pip per project
    • this one was made specifically to be the canonical tool for this purpose
  • poetry
    • isolate dependencies per project
  • Nix + nix-direnv
    • completely unrelated to the Python ecosystem but can do what all of these tools do except with a few more very useful features
  • Bonus round: conda, anaconda, miniconda, mamba, micromamba

Understanding dependencies

Now that you've settled on a combination of tools, it's time to go on the hunt for helpful dependencies for your program. Over time you'll notice that a good fraction of projects present their documentation in different ways. It can be difficult going from one high profile project to another because not only do you need to relearn the API surface, but also how to navigate its documentation in the first place. Speaking of API surface, a lot of documentation is actually written as if it were a guide rather than an API reference, so determining the permissible uses of any given item is much harder and often ambiguous unless you read the source code of the project, or that of projects using the library in question.

Using dependencies

It's now time to actually add the dependencies you've deemed useful and worthy and start writing code using them. Uh oh, version resolution is failing? Looks like two of your direct dependencies require incompatible versions of a transitive dependency. Luckily, this is solved trivially: just kidding, it isn't.

Making your code nice (visually)

After getting out of that mess somehow, your codebase has grown in size significantly and you think it's probably time to start using an auto-formatter and a linter, especially since you'd like to bring in some extra help. For auto-formatters, you have the following choices:

  • autopep8
  • black
  • yapf
  • IDE-specific tooling (good luck running this in your CI pipeline)

You notice something peculiar: the tool you've chosen doesn't automatically sort your import statements at all. Indeed, you need a separate tool for this. Here are some options:

  • lexicographically sort them like some kind of madman
  • Use isort like the CIA probably does
  • IDE-specific tooling (good luck running this in your CI pipeline)

Make your code nice (cognitively)

Even though it looks nice, you still feel like there are some questionable lines of code here and there. A good number of these oddities can be automatically detected and complained about by a linter, which will help you clean things up a bit more. Again, you have a number of options:

  • autoflake
  • bandit
  • flake8
  • flakehell
  • prospector
  • pycodestyle
  • pydocstyle
    • This one actually lints your docstrings rather than your code, which is useful
  • pyflakes
  • pylsp
  • pylint
    • I've saved this one for last because it appears to be a superset of some of the previous linters (at least it is for flake8), and it seems to be the most commonly used

Configuring stuff

Now that you've chosen all of your tooling, you should probably configure them to your liking. Or at the very least, configure them to work with each other, as lot of them disagree by default. The rapidly growing list of configuration files is worrying you, so you decide to see if there's some alternative. To your delight, you discover PEP 518, which exists for this exact purpose. To your dismay, however, you come to learn that half of your chosen tools don't actually support this.

Gaining momentum

There are now multiple people working on your codebase, perhaps because you've started a company around this program of yours. The number of lines of code increased rapidly and you no longer have had eyes on all parts of the project. Perhaps lints stopped failing your CI pipelines in the name of feature velocity. You find yourself wishing more lints were capable of failing builds, that you and others had left more docstrings behind, or at least that you knew anything about the return value of the function you're trying to call. It would be cool if you could generate some sort of easily-navigable API reference to get a summary of how the new code works together, too. You find a few tools, only to discover that some of them are painful to set up on an existing project and that some of them refuse to document items considered "private" entirely. This is unfortunate considering you're writing an application and not a library, meaning basically all of your code is considered private.

Static typing

At the very least, you can start using yet more linters locally. More specifically, static analysis tooling. Again, you have a number of options:

  • mypy
  • pytype
  • pyre
  • pyright

These tools will help you figure out what's going on in this new code you haven't seen before. It gives you a map to get out of the woods you've been feeling lost in. It also helps to prevent a few kinds of duck-typing related bugs caused by incorrect assumptions about an object's properties. At this point, though, the code is so fargone that allowing one of these tools to fail CI is implausible due to the sheer amount of changes required to get CI to start passing again. And if the tools can't fail CI, you can't guarantee that other people will follow the conventions set by these new tools, which can significantly hinder their utility.

Statically typing exceptions

You've now noticed that PEP 484 provides no method to annotate exception types, nor is any existing static analysis tool capable of determining the list of exception types a given function may raise. You may be told to "just read the docs", to which you point out that there are often not docs, and sometimes they lie. Or maybe they said "just read the source code", which you know is ridiculous because that would mean you'd have to read the entire source tree of every function you'd like to know about. Or maybe "you should only handle exception types you know you can handle", to which you ask "how can I decide whether I can handle an exception if I don't know what the possible exceptions are?" and are left without an answer like the OP of this SO post.

Memory model

Suddenly you realize that you're not sure whether you're supposed to be able to mutate the original memory of the object being passed to the function you're currently bugfixing. You wonder whether it's possible to annotate or verify the intended ownership semantics of the object and are disappointed to learn that you (currently?) cannot. Perhaps you also wonder if a system like this could improve synchronization primitive ergonomics by forcing you to acquire a lock before granting you read or write access to the underlying data. You look for prior art on this topic and discover it has indeed been done before. Neat.

Performance

The performance of your application is showing signs of becoming a problem, so naturally you seek to introduce some form of parallelism (presumably you've already been using async/await for IO-bound concurrency). The obvious answer is multithreading, so you dispatch parallelizable work to a threadpool to be executed, adding locks throughout your code as necessary. Since lock acquisition/release isn't enforced particularly strongly as you learned earlier, perhaps you hit some deadlocks or data races or other parallelism-related bugs along the way.

Looking at your CPU usage, you notice that the program is still only using a single core at a time for some reason. It turns out that this is because threads aren't real (in CPython at least). A popular way to solve this problem is mutliprocessing via some IPC mechanism like UDS or TCP or HTTP or AMQP or something. You pick one of these and successfully further your goal of scaling vertically, but deep down, you're keenly aware of the overhead of IPC versus shared memory and locks and it leaves a bad taste in your mouth.

During your research about parallelism in Python, you discover some discourse on the topic of removing the GIL, but it seems like the consensus is that this is a bad idea because, somewhat ironically, it would actually hinder performance quite significantly. The reason for this is that the GIL acts as a single lock around the CPython interpreter's internals, and a single lock is very fast and has low contention. If you remove this lock (and still desire programs to run correctly), you'll need to add a bunch of smaller locks around all the objects. Even though there tends to be low contention on all these objects, Python's refcounting memory model now requires atomics, which absolutely annihilates CPU cache as new objects are created and destroyed all the time.

If only there were an alternative memory management scheme to alleviate this problem, like a different flavor of garbage collector or perhaps some way to statically track object lifetimes. But at that point, it probably wouldn't be Python anymore.

Further reading