Adding PyPI Update Github Action to Existing Project

I’ve now been updating a few projects to also automatically update PyPI with each new release (see https://foggyprogrammer.com/github-actions-updates-pypi-package for context), but have run into a couple snags:

TestPyPI submission fails when the version is not updated. I mostly work on the master branch and each new commit tries to push a new build with the same version to TestPyPI. It seems to make more sense that I should run both testpypi and pypi only when a new tag/version/release is ready.
Running testpypi on every submit, especially when it will fail (and I learn nothing) seems like a waste of computing resources.
The steps to update a new project were less trivial than I expected. So, this post is mainly a reminder of how I should do this.

Updates to github workflow

I updated the trigger file to only run when a tag is pushed on the master branch (I think the ‘master branch only’ bit is implied).

name: Publish Python distribution to PyPI and TestPyPI
on:
  push:
    tags:
      - '*'

I can then remove the if statements on the individual steps, since the job will only run when a new release is has been pushed. Also, if the testpypi job fails, the pypi will not even run.

I also found an option to get the latest ubuntu machine with ubuntu-latest. I’m a little afraid to try the same thing with Python in the event that a package isn’t yet available/compatible. Everything else is the same, and here it is for completeness:

name: Publish Python distribution to PyPI and TestPyPI
on:
  push:
    tags:
      - '*'
jobs:
  build-n-publish:
    name: Publish Python distribution to PyPI and TestPyPI
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
      - name: Setup Python 3.8
        uses: actions/setup-python@v1
        with:
          python-version: 3.8
      - name: Install flit
        run: >-
          python -m
          pip install
          flit
      - name: Build a binary wheel and a source tarball
        run: >-
          python -m
          flit build
      - name: Publish distribution to TestPyPI
        uses: pypa/gh-action-pypi-publish@master
        with:
          user: __token__
          password: ${{ secrets.testpypi_token }}
          repository_url: https://test.pypi.org/legacy/
      - name: Publish distribution to PyPI
        uses: pypa/gh-action-pypi-publish@master
        with:
          user: __token__
          password: ${{ secrets.pypi_token }}

pyproject.toml

I initially forgot to incorporate my pyproject.toml file, but this is required by flit. Also, it took a while to find out how to convert various other options from setup.py to flit.ini, primarily due to the fact that flit automatically handles a lot of the configurations that setup.py required explicitly. E.g., package_dir={'': 'src'}, packages=, and package_data= are all handled implicitly. Console scripts are handled in a separate section called [tool.flit.scripts] using a similar format to setup.py.

Here’s a more complete example:

[build-system]
requires = ['flit_core >=2,<4']
build-backend = 'flit_core.buildapi'

[tool.flit.metadata]
module = 'module'
author = 'author'
author-email = '@email.com'
home-page = 'https://github.com/author/module'
requires = ['pandas']
requires-python = '>=3.7'
description-file = 'README.md'
keywords = 'something special'
classifiers = [
    'Development Status :: 5 - Production/Stable',
    'Programming Language :: Python :: 3 :: Only',
    'License :: OSI Approved :: MIT License',
]

[tool.flit.scripts]
name = 'module.pyfile:main'

[tool.flit.metadata.requires-extra]
test = ['pytest']

Update package-level init.py

The top-level __init__.py needs to have both a pydoc description and a version number. (I think flit is still reading my version from setup.py, so I’ve continued updating both until I take the time to figure it out.)

# in: package/__init__.py
"""Does something really cool."""
__version__ = '2.5.0'

Update .gitignore

Make sure everything in your project is committed or ignored, otherwise you’ll get this exception when using flit publish:

flit_core.common.VCSError: Untracked or deleted files in the source directory. Commit, undo or ignore these files in your VCS. ()

You can see what hasn’t yet been added with git status or

git ls-files --others --exclude-standard

Or just remove all untracked files (probably not always a good idea):

# REMOVES all untracked files
git clean -df

With Pycharm, I’ve found a few files in the .idea/ directory are not added by default (I use the .ignore plugin), but I’ve never included them in my project (maybe there’s a good reason to?), so I’ll manually include in .gitignore.

.idea/.gitignore
.idea/inspectionProfiles/
.idea/misc.xml
.idea/modules.xml
.idea/regexify.iml
.idea/vcs.xml
.idea/[project].xml

Add secret keys to project

If the project doesn’t yet exist on PyPI and TestPyPI, run flit publish locally. (See: https://foggyprogrammer.com/add-python-package-to-pypi for more details.) For TestPyPI, ensure that the .pypirc files is fully specifice the docs — not required for just PyPI.
Generate api keys which are limited to a particular project in PyPI and TestPyPI, with a name like f'github-{project}'.
Add pypi_token and testpypi_token to the github > settings > secrets.

Finally…

Once these changes are completed and committed, bump the version, add a tag with that version information, and push to Github. Go to the project page to ensure that the build succeeded.