Background

One gripe I have with Python in my professional work relates to package management. My first job out of university had me working on Node backend servers. After changing teams I started writing Python code full time instead of Javascript. Shortly after this switch I started looking back fondly on my time with npm and yarn. That fondness may be due to someone else doing the work of getting the private packages set up with npm. Python was newer for the company and didn’t have as much tooling set up for it. It may have been due to npm handling the majority of the work. The company paid npm for each developer to access the private repositories. Either way, I came to miss that ease of use with private packages within the organization.

Many years later I still work with python and have the same gripe. With some free time on my hands, I tried setting up a private pypi repository. My goal was to get a working demo of publishing and downloading a package to my local machine. The constraint for this project was that the solution worked with Google Cloud. This is the Cloud Provider my current work uses so the restraint was a practical one. Ideally the solution would also work with a python dependency manager. I prefer simpler tools like Pipenv or Poetry which make Python projects as easy as possible. I came up with the following objectives for this work.

Ease of use. The smoother it is for a developer to download or publish a package, the better.
Low maintenance. The less work required to keep the repository up and running, the better.
Price. The cheaper it is to run the server, the better.

I found that Artifact Registry with Poetry fit the requirements. The next section of this post shows how I got it working. It should be self explanatory how it fits the objectives set out above.

Demo

This section of the blog post covers how I got the demo working. If you’re only interested in my final thoughts, feel free to skip to the Summary.

Setup

First, this demo assumes you have Poetry and gcloud installed on your machine.

which poetry
which gcloud

It also assumes that you have a GCP project you can use for this demo. You’ll need to make sure you’re authenticated with gcloud and the project has Artifact Registry’s API enabled.

gcloud auth login
gcloud auth application-default login
gcloud services enable artifactregistry.googleapis.com

Finally, we need to configure Poetry to include google cloud artifact registry keyring. This will make sure that Poetry can authenticate with Google’s Artifact Registry.

poetry self add keyrings.google-artifactregistry-auth

Repository

First thing we need is the Artifact Repository to push packages to and pull packages from. GCP provides some great documentation on this, so I won’t repeat that here.

Google also provides a CLI command for fetching the repository’s push and pull url. These will be important later to configure Poetry.

gcloud artifacts print-settings python --repository YOUR_REPOSITORY_NAME

Publish

With the repository set up, we can now publish a package to it. To help with this demo, I created a git repository to scaffold the majority of the python code. You can clone the repository with the command below.

git clone git@github.com:steve148/poetry-artifact-registry-demo.git

From the top level of the repo, you should see two folders. my-dumb-library is a python project which includes a package we want to publish to our pypi repository. my-dumb-app contains a python app which wants to use the dumb library code. Before we can look at downloading the package, we need to publish it to our repository. To do that let’s move into the my-dumb-library folder.

cd my-dumb-library

The following files within that folder are important for this demo.

pyproject.toml - This is the configuration file for our Python project. Alongside the basic poetry configuration, it defines the dependency of pydantic for this package.
poetry.toml - This is the configuration file for Poetry. It defines the name and url for the private pypi for package publishing.
my_dumb_library/__init__.py - This is the entry point for the package. It exports the Character model.

To publish this package to your private pypi repository, we need to update the publish url for Poetry. You can get the url from the output of the gcloud artifacts print-settings command. With that url we can run the following command to update Poetry.

poetry config --local repositories.my-dumb-pypi YOUR_PUBLISH_URL

Now onto the final step. We need to build the python wheel and publish it to Artifact Registry. Poetry makes this easy to do.

poetry publish --build --repository my-dumb-pypi

We can check that the package was published by going to the Artifact Registry page in the GCP console. We should see a package called my-dumb-library with a version of 0.1.0.

Viewing the python package on Google Cloud

Download

Now that we have our package published, we now want to try to download it. Let’s switch over to our app folder.

cd ../my-dumb-app

The only thing we need to change for this project is the pull url for your Artifact Registry. Like before, get the pull url from the output of the gcloud artifacts print-settings command. With that url we can run the following command to update Poetry.

poetry source add --priority=supplemental my-dumb-pypi YOUR_DOWNLOAD_URL

Now we can test installing the python package in the app’s virtual environment.

poetry install

We can run the snippet below to test that the app code works.

poetry run python my_dumb_app/__init__.py

Summary

This isn’t a perfect demo. I hope it is useful for someone else interested in this problem and wondering how to get started. At bare minimum, this post will remind myself how to set this up in the future. This approach felt very low friction for me. Little setup is required from later users once the initial configuration is done. Ideally someone joining the project would only need to run poetry install to download the dependencies. Authentication with Google Cloud is unnoticeable to the user. If you’re serious about using this approach, I would suggest doing a quick cost estimate. I didn’t end up doing that here so it’d be prudent to figure that out before putting this into production.

Time moves forward and the Python developer experience improves.

Private Pypi Server on GCP