I had wanted to follow the guide for automated PyPI releases here (https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/), but ended up getting carried away with the PyPI packaging process. Now, I can turn to updating my three repositories to automatically deploy new releases to PyPI.
Begin by creating new tokens for the PyPI package on both testpypi and pypi proper — ideally place ‘Github’ somewhere in the name to remember what it’s purpose is.
Navigating to ‘Settings’ > ‘Secrets’, and adding two new secrets: pypi_token and testpypi_token (or something like those).
Returning to my local code, I need to create a new .yml file in a .github/workflows directory and add it to git:
And here’s the content which the tutorial recommends. Basically, give the job a name, and then describe the steps that the workflow should take. The example uses ‘ubuntu-18.04’ and ‘Python 3.7’, but I’ve selected ‘ubuntu-20.04’ and ‘Python 3.8’.
name: Publish Python distribution to PyPI and TestPyPI
on: push
jobs:
build-n-publish:
name: Publish Python distribution to PyPI and TestPyPI
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@master
- name: Setup Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Install pep517
run: >-
python -m
pip install
pep517
--user
- name: Build a binary wheel and a source tarball
run: >-
python -m
pep517.build
--source
--binary
--out-dir dist/
- name: Publish distribution to TestPyPI
uses: pypa/gh-action-pypi-publish@master
with:
password: ${{ secrets.testpypi_token }}
repository_url: https://test.pypi.org/legacy/
- name: Publish distribution to PyPI
if: startsWith(github.event.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
with:
password: ${{ secrets.pypi_token }}
Unfortunately, the references to a package based on PEP 517 (and its companion PEP 518) and the reference to a pyproject.toml file are new to me. This seems like the proper way to do things…
pyproject.toml
PEP 517 looks to add other methods for building Python distributions (as opposed to just distutils/setuptools), and PEP 518 defines a pyproject.toml file to define the build process. I’m not really sure what this is supposed to look like practically, except that one of the build packages, flit seems to help you build the file.
pip install flit flit init # prompts to fill in information
Increased documentation on what option flit wants are available here: https://flit.readthedocs.io/en/latest/pyproject_toml.html.
flit appears to get version information and a short description from the module/__init__.py file (in my case, src/regexify/__init__.py). This must at minimum have a docstring and version number:
"""Regular expression containers and helper functions to simplify the use and deployment of regular expressions.""" __version__ = '0.1.1'
Running flit publish to push the package to PyPI, I received an error about untracked or deleted files in git:
flit_core.common.VCSError: Untracked or deleted files in the source directory. Commit, undo or ignore these files in your VCS. ()
I committed my nascent workflow file and pyproject.toml before running git status which shows .idea/. I guess I need to handle this in .gitignore?
Opening .gitignore with the .ignore plugin in Pycharm, I have an option to add unversioned files:
.idea/.gitignore .idea/inspectionProfiles/ .idea/misc.xml .idea/modules.xml .idea/regexify.iml .idea/vcs.xml
Okay, now commit the new .gitignore.
flit publish runs excep thtat PyPI reports that File Already Exists. I suspect this is because I didn’t bump the version number. Updated both in setup.py and pyproject.toml and it works, building and uploading both to TestPyPI and PyPI.
Back on Github Actions
Well, the github build failed, but this is expected since I hadn’t used flit in the pyproject.toml file. The red X alerts me to this failure and provides more details if clicked.
I updated the publish-to-pypi.yml to install and run flit, and swapped in ubuntu-18.04 for 20.04 since no such environment exists. I am, however, still getting an error when it’s running on Github actions: Exception: Could not find username for upload. I think this means that the username __token__ is not being specified.
The action we are using is pypa/gh-action-pypi-publish, which provides some additional help. I’m realizing that the call to flit publish is building and publishing — this is having trouble identifying a .pyprc file. Apparently this can be set as an environmental variable? First, I’m going to try just using flit build. If that doesn’t work, I’ll setup some environment variables in the VM and pass user/password/repo to those. (See, e.g., https://flit.readthedocs.io/en/latest/upload.html.)
This time, however, it worked!
I’m going to bump the version info and tag it to see if I can also publish to PyPI. This is working, but it looks like there’s a race condition when publishing to TestPyPI as both the tag and the commit are firing off the TestPyPI job. I’ll make it so that the release/tagged version only fires off PyPI and skips the TestPyPI.
Success!
And I can run pip install regexify --upgrade to get version 0.1.5 (it took a few runs to get his working). Now that I have a working form, adding my other projects should go pretty smoothly. Here’s what I’m using:
name: Publish Python distribution to PyPI and TestPyPI
on: push
jobs:
build-n-publish:
name: Publish Python distribution to TestPyPI
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout@master
- name: Setup Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Install flit
run: >-
python -m
pip install
flit
- name: Build a binary wheel and a source tarball
run: >-
python -m
flit build
- name: Publish distribution to TestPyPI
if: github.event_name == 'push' && !startsWith(github.event.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.testpypi_token }}
repository_url: https://test.pypi.org/legacy/
- name: Publish distribution to PyPI
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.pypi_token }}
Update: see https://foggyprogrammer.com/adding-pypi-update-github-action-to-existing-project