AI-translated. Read the original in Korean

setup.py, Stop Right There!

Sep 13, 2021

Software Engineer

Packaging is the most efficient way to share Python code with others or distribute a project without unnecessary files. Python has a long history, and its packaging approaches have evolved along with it. This post explains two problems in Python packaging and the recommended, standards-based approach at the time of writing.

A brief history of packaging

While researching the history I knew, I found an excellent article. Rather than repeat it, I’ll point you there: https://ryanking13.github.io/2021/07/11/python-packaging.html

Am I using a legacy approach?

The following are signs of a legacy packaging setup:

A setup.py file exists.
Your final packaging command starts with python setup.py ...

If you use Poetry, you aren’t using that legacy approach. The appendix below may still help simplify your release process.

Problem 1: packaging requires setup.py

What is wrong with setup.py?

No real alternative

Python already standardizes its package formats: wheel for binary distributions, and source distributions. Put files in the expected locations with the correct names, and a package can be installed just like one produced by setuptools or Poetry. But the packaging process itself lacked a standard. Instead, setuptools and setup.py served as the de facto standard, even though setuptools is a third-party library. Reading setup.py requires setuptools, so any packaging tool has to depend on it.

Who manages build-time dependencies?

Runtime dependencies are recorded in the package, so pip installs them automatically. But what if setup.py imports a particular library to manage versions during packaging? Without that library installed, python setup.py raises ImportError. You have to open the file, identify the dependency, and install it yourself.

The solution: configuration instead of scripts

If executing setup.py creates these dependency problems, let’s remove the script from the equation.

`pyproject.toml`

PEP-518 introduced this file to declare build-time dependencies. Its role has since expanded to hold project metadata and configuration for tools such as pytest, mypy, and various linters.

Examples

Building a wheel with setuptools

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

Using Poetry

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

With this configuration in pyproject.toml and a packaging tool that implements PEP-517, installing the build system and dependencies and creating the package can be automated—even in an environment without setuptools or Poetry preinstalled.

The PEP-517 tools I found were build and pep517. At the time of writing, the latter is deprecated and build appears to be the usual choice. Poetry also supports PEP-517, though my tests suggested it worked only when Poetry was the backend.

With build, these two commands are sufficient. There is no need to install the backend, such as Poetry or setuptools, first.

pip install build
python -m build .

Problem 2: no shared metadata format

What is package metadata?

The build-system configuration alone isn’t enough. Beyond essential information such as package name, version, and dependencies, Python packages can include a README, tags, entry points, and other metadata.

A standard for declaring this information does exist, but it was finalized only recently: PEP-621 became Final on March 1, 2021. At the time of writing, major build tools have not yet adopted it.

Build system	Status	Link
setuptools	Under discussion	https://github.com/pypa/setuptools/issues/1688
Poetry	Under discussion	https://github.com/python-poetry/poetry/issues/3332
Flit	Supported	https://flit.readthedocs.io/en/latest/pyproject_toml.html#pyproject-toml-project

We therefore still need each tool’s specific format: setup.cfg for setuptools, or [tool.poetry] in pyproject.toml for Poetry.

Below are examples and documentation for those formats, followed by the standard format intended to work across build tools in the future.

setuptools

Official documentation: https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html#declarative-config

Declare the fields previously held in setup.py in the INI-format setup.cfg file.

[metadata]
name = my_package
version = attr: src.VERSION
description = My package description
long_description = file: README.rst, CHANGELOG.rst, LICENSE.rst
keywords = one, two
license = BSD 3-Clause License
classifiers =
    Framework :: Django
    License :: OSI Approved :: BSD License
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3.5

[options]
zip_safe = False
include_package_data = True
packages = find:
scripts =
    bin/first.py
    bin/second.py
install_requires =
    requests
    importlib; python_version == "2.6"

[options.package_data]
* = *.txt, *.rst
hello = *.msg

[options.entry_points]
console_scripts =
    executable-name = package.module:function

[options.extras_require]
pdf = ReportLab>=1.2; RXP
rest = docutils>=0.3; pack ==1.1, ==1.3

[options.packages.find]
exclude =
    src.subpackage1
    src.subpackage2

[options.data_files]
/etc/my_package =
    site.d/00_default.conf
    host.d/00_default.conf
data = data/img/logo.png, data/svg/icon.svg
fonts = data/fonts/*.ttf, data/fonts/*.otf

Poetry

Official documentation: https://python-poetry.org/docs/pyproject/

Unlike setuptools, Poetry was developed more recently and used pyproject.toml instead of setup.py from the beginning. Because it predates the acceptance of PEP-621, however, it stores metadata in its own slightly different format under [tool.poetry] rather than [project].

[tool.poetry]
name = "poetry"
version = "1.2.0a2"
description = "Python dependency management and packaging made easy."
authors = [
    "Sébastien Eustace <sebastien@eustace.io>"
]
license = "MIT"

readme = "README.md"

include = [
    { path = "tests", format = "sdist" }
]

homepage = "https://python-poetry.org/"
repository = "https://github.com/python-poetry/poetry"
documentation = "https://python-poetry.org/docs"

keywords = ["packaging", "dependency", "poetry"]

classifiers = [
    "Topic :: Software Development :: Build Tools",
    "Topic :: Software Development :: Libraries :: Python Modules"
]

[tool.poetry.build]
generate-setup-file = false

# Requirements
[tool.poetry.dependencies]
python = "^3.6"

poetry-core = "^1.1.0a6"
cleo = "^1.0.0a4"
crashtest = "^0.3.0"
requests = "^2.18"
cachy = "^0.3.0"
requests-toolbelt = "^0.9.1"
cachecontrol = { version = "^0.12.4", extras = ["filecache"] }
pkginfo = "^1.5"
html5lib = "^1.0"
shellingham = "^1.1"
tomlkit = ">=0.7.0,<1.0.0"
pexpect = "^4.7.0"
packaging = "^20.4"
# temporarily clamped due to https://github.com/pypa/pip/issues/9953
virtualenv = ">=20.4.3,<20.4.5"
keyring = ">=21.2.0"
entrypoints = "^0.3"
importlib-metadata = {version = "^1.6.0", python = "<3.8"}
dataclasses = {version = "^0.8", python = "~3.6"}

[tool.poetry.dev-dependencies]
pytest = "^6.2"
pytest-cov = "^2.8"
pytest-mock = "^3.5"
pre-commit = { version = "^2.6", python = "^3.6.1" }
tox = "^3.0"
pytest-sugar = "^0.9"
httpretty = "^1.0"
zipp = { version = "^3.4", python = "<3.8"}
deepdiff = "^5.0"

[tool.poetry.scripts]
poetry = "poetry.console.application:main"

The future: PEP-621 and Flit

Official documentation:

PEP-621: https://www.python.org/dev/peps/pep-0621/#details
Python Packaging User Guide: https://packaging.python.org/specifications/declaring-project-metadata/
Flit: https://flit.readthedocs.io/en/latest/pyproject_toml.html#pyproject-toml-project

Once setuptools and Poetry support PEP-621, metadata written in this format will be interchangeable across tools. As the table shows, however, those tools have not yet implemented the standard at the time of writing. To package with this format now, the available choice is Flit.

[project]
name = "spam"
version = "2020.0.0"
description = "Lovely Spam! Wonderful Spam!"
readme = "README.rst"
requires-python = ">=3.8"
license = {file = "LICENSE.txt"}
keywords = ["egg", "bacon", "sausage", "tomatoes", "Lobster Thermidor"]
authors = [
  {email = "hi@pradyunsg.me"},
  {name = "Tzu-Ping Chung"}
]
maintainers = [
  {name = "Brett Cannon", email = "brett@python.org"}
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python"
]

dependencies = [
  "httpx",
  "gidgethub[httpx]>4.0.0",
  "django>2.1; os_name != 'nt'",
  "django>2.0; os_name == 'nt'"
]

[project.optional-dependencies]
test = [
  "pytest < 5.0.0",
  "pytest-cov[all]"
]

[project.urls]
homepage = "example.com"
documentation = "readthedocs.org"
repository = "github.com"
changelog = "github.com/me/spam/blob/master/CHANGELOG.md"

[project.scripts]
spam-cli = "spam:main_cli"

[project.gui-scripts]
spam-gui = "spam:main_gui"

[project.entry-points."spam.magical"]
tomatoes = "spam:main_tomatoes"

Appendix: simpler releases for Poetry projects

TL;DR

Use python -m build instead of the poetry command in Dockerfiles and CI pipelines.

Poetry handles packaging, virtual environments, dependency resolution, package publishing, and other development tasks together. But invoking poetry just for packaging inside a Dockerfile or CI pipeline means installing all that functionality.

Poetry itself has many dependencies, so installation takes time. Alpine images make this more complicated: dependencies such as cryptography contain C code, requiring build tools and compilation. At the time of writing, the relevant wheel support is unavailable for Alpine. If you aren’t using Poetry interactively on your local host, the build command described above is therefore the simpler option, even for a Poetry-managed project.

Here is a Dockerfile example.

Before

FROM python:3.9-alpine
WORKDIR /src
COPY . .
RUN apk --update add gcc musl-dev openssl-dev libffi-dev cargo
# 아래가 rust, c 코드 컴파일로 몇분이 소요됨
RUN pip install poetry
RUN poetry build

After

FROM python:3.9-alpine
WORKDIR /src
COPY . .
RUN pip install build
RUN python -m build

setup.py, Stop Right There!

A brief history of packaging

Am I using a legacy approach?

Problem 1: packaging requires setup.py

What is wrong with setup.py?

No real alternative

Who manages build-time dependencies?

The solution: configuration instead of scripts

`pyproject.toml`

Examples

Building a wheel with setuptools

Using Poetry

Problem 2: no shared metadata format

What is package metadata?

setuptools

Poetry

The future: PEP-621 and Flit

Appendix: simpler releases for Poetry projects

TL;DR

Before

After

Breaking Down asyncio, Part 3: Handling SIGTERM (CTRL+C) Properly

Breaking Down asyncio, Part 2: Working with Futures

Breaking Down asyncio, Part 1: Coroutines and the Event Loop

setup.py, Stop Right There!

A brief history of packaging

Am I using a legacy approach?

Problem 1: packaging requires setup.py

What is wrong with setup.py?

No real alternative

Who manages build-time dependencies?

The solution: configuration instead of scripts

pyproject.toml

Examples

Building a wheel with setuptools

Using Poetry

Problem 2: no shared metadata format

What is package metadata?

setuptools

Poetry

The future: PEP-621 and Flit

Appendix: simpler releases for Poetry projects

TL;DR

Before

After

Related articles

Breaking Down asyncio, Part 3: Handling SIGTERM (CTRL+C) Properly

Breaking Down asyncio, Part 2: Working with Futures

Breaking Down asyncio, Part 1: Coroutines and the Event Loop

`pyproject.toml`