Python Package Dev

A general workflow for creating and developing python packages

This provides a bare-bones example of the workflow for creating and developing a python package. It is basically the workflow described in Beuzen & Timbers’ Python Packages book but with some tweaks/some of my notes.

And here’s an example repo

Step 0: Install Tools

We need the poetry and cookiecutter tools to help manage dependencies and set up package templates. We can install these with pipx

$ pipx install poetry
$ pipx install cookiecutter

Step 1: Create pkg structure with cookiecutter

#use the python packages book cookiecutter template
$ cookiecutter https://github.com/py-pkgs/py-pkgs-cookiecutter.git

Also in this step we probably want to add version control and make a virtual env

$ git init
#etc -- do a first commit, add a remote, etc.

To make a virtual env, I like to use VS Code to do this (via the command palette > Python: Create Virtual Environment)

then activate the venv by:

$ .venv/Scripts/activate

Step 2: Write python code and add dependencies

Now we can write whatever python code we want (within /src). Whenever we need to add a dependency, we can add it via the shell:

$ poetry install <dependency_name>

We’ll want to make sure we’re doing this in our virtual environment.

Step 3: Install our pkg and try it out

We can install our in-development package with

$ poetry install

One useful note is that we can change code in the package and we don’t need to reinstall it – it’ll just work with the new code

Step 4: Test code

First, we write tests for our code inside /tests. Test files should be prefixed with test_.

We want to add pytest (for testing) and pytest-cov (for test coverage) as dependencies. But our main package doesn’t actually depend on these things – only the testing does. Since they’re a development dependency, we can specify this in poetry via:

$ poetry add --group dev pytest pytest-cov

and then to run our tests, we can do (in the shell):

$ pytest tests/ --cov=<pkg_name>

Where the --cov flag says we also want to see the test coverage.

Step 5: Write and render documentation

Some of the boilerplate documentation will be pre-generated via the cookiecutter package we used. We’ll definitely want to update the README, though, and possibly some other files.

We’ll definitely want to include some docstrings within our functions – probably using the numpy docstring standard (n.b. that DuetAI seems to be really good at generating these).

We also might want to include a vignette showing a worked example of a common workflow in the package. We can create these as Jupyter notebooks within the docs/ directory.

Finally, we’ll want to render the documentation. Given our cookiecutter template, we can do this via the shell:

#add tools to render documentation
$ poetry add --group dev myst-nb sphinx-autoapi sphinx-rtd-theme
$ cd docs
$ make html
$ cd ..

The make html command in the above is doing al of the heavy lifting.

Then we can optionally host the documentation on Github Pages or Read the Docs

Step 6: Tag, build, and publish

These probably should be multiple steps, but whatever.

When we make changes, we probably want to tag these as part of a release, via:

$ git tag

The Python Packages book has more info on tagging versions here

We can build distributions of our package via:

poetry build

And if we want to publish our package to TestPyPI as a “dry run” before publishing to PyPI:

$ poetry config repositories.test-pypi \
  https://test.pypi.org/legacy/
$ poetry publish -r test-pypi
$ pip install --index-url https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple \
  <pkg_name>

And then if that works, we can publish to PyPI

$ poetry publish
$ pip install <pkg_name>