Towards testable repository automation workflows

I have recently found myself fully maintaining multiple CI/CD workflows for the first time. It's been exciting setting up the initial workflow, but now it's a huge pain making small changes to it. I'm going to be exploring automated testing of the CI/CD workflows themselves while exploring a few specific types of (reusable) workflows.

I'm calling it relez^[1].

Repository automation

Now, let's talk about what I'm actually trying to tackle here. CI/CD covers a lot of things, and I'm going after a very small, specific portion of it, in a way tailored for a very particular crowd. There's probably a better name already out there (and I'm open to better suggestions), but I'm going to term it "repository automation".

What relez will not cover is the testing and release actions themselves. There are plenty of methods of running tests, builds, and releases, all generic to different degrees. Make, Bazel, doit, npm scripts, etc. all do their job just fine for the purposes of this exploration. relez will cover the questions of when, how, and why those are run.

I'll be (vaguely) defining a Git flow and collection of automations within relez that I've dubbed relez-flows. It's described in-depth in My ideal versioning and releasing workflow^[2], so what I'd like to cover here is the experimentation around testing those flows. relez-flows will initially cover the most complex workflow described there.

Testing that the tests are run

Maybe more importantly, I'd like to enable my own flows experimentation through test automation. A full set of automations quickly becomes as complex as a minor application, and when making even small changes to CI/CD workflows you are effectively testing in production. You can dogfood in a private/less important repo, but that doesn't make the A-Z workflow easier to test.

To tackle this challenge, I'm experimenting with testing relez-flows under the name relez-tests. This set of tests and utility tools will test the Github Workflows and Actions, for now using Bats and Github CLI to run the automation in real repositories. By building them in parallel I have a reason to build the tests, as well as a way to test the flows.

relez's future

I don't really feel a need to define a future for relez, but I'd like to be able to use it in my own repositories as soon as possible. It'll need to make quite a few big steps to make this possible. After finishing the flows, relez will need a method of deployment as well as a method of pushing updates to dependent repositories. relez will need to cover the different versioning workflows to allow using it in my backend repository.

Additionally, it would be nice if this produced something usable by others. If this ends up being useful for myself, I will most likely write more blog posts about it. Potentially, it could also grow into a set of libraries and utilities that could be reused to build other reusable flows.

The dream here is for both relez-flows and relez-tests to be platform-agnostic. At some point, most source management/CI platforms do the same things and it should be possible to deploy the same workflows across them.

In conclusion...

I won't make any promises here on relez's functionality, direction, or even continuation. Currently, it's an experiment that hopefully turns into something useful for myself.

If you're interested in this at all, feel free to come hang out in the repository's discussions tab, star the repo, PR, or message me on Twitter or Mastodon.