Delivery pipelines

Created on 2023-08-11 09:14

Published on 2023-08-11 10:18

The industry is buzzing currently with term as Continuous Delivery, Continuous Deployment an Pipelines. Pipelines are techniques to seed up the processing time by sequentially running a set of tasks on specialized stages, at every moment of time one step consumes the output of the previous one. The concept is not new, it has been started in the early 1900 when it was applied to factories and assembly lines. But the current pace of software development just put a lot of spotlight on them.

The software pipeline most often do a set of operations as check-out code from VCS, compile/build it, package it, run tests on it, deploy the artifacts to a repository and inform the stakeholders of the result, update various metrics and dashboards. These actions require a script to put things in the right order and check the results in every step.

In software the pipelines are generally defined through a specific language, either a visual one or a text based one - generally YAML or Groovy. These are run by specialized software as Jenkins, GitHub Actions, CircleCI, etc. The high level pipeline described in YAML is calling various other scripts that define the individual steps - like micro-operations.

The build step for example is the invocation of a Makefile, shell script or a Maven build. The pipeline will wait for the completion of the build step before passing the artifacts for testing or packaging. Such a situation is interesting because there will be two points of control in the pipeline. One at pipeline level and one at step level but their responsibilities will bleed into each-other.

This might lead to some problems:

1. Inefficient pipelines: If the result of a step can be guessed before the whole step completes the pipeline should fail fast. It makes no sense to wait for the completion. Imagine that you have a multi project build. In the trivial case the projects are built and packaged. But what if one of them has an issue? Should we wait for completion of all builds and then juts cancel the packaging step? What if there are modifications on a single project that has no egress dependencies? Again - it makes no sense to package all other projects as they haven't been modified. However with the split logic this is hard to achieve as the operations on pipeline level are quite coarse-grained. On the other hand most build tools (Gradle, MSBuild, ...) are perfectly able to run more parts of the process by themselves - create archives, containers, publish artifacts - and probably they should do it. Fail fast and incremental builds are essential for accelerated release cycles. The build tool is in a better position to understand what it is building by looking in the source code than the pipeline that is merely an orchestrator of some loosely coupled processes. Caching build results and artifacts, using parallel builds could reduce build times from 20 minutes to less than a minute - so some DORA metrics will look way better and will make both developers and managers less impatient. Speculative execution and rescheduling is well known in CPU pipelines - software engineers should have some inspiration from the clever solutions that hardware folks are successfully applying for more than two decades.
2. Portability: The high level pipeline logic is hard to be moved from one type of executor to the other. Pipelines built for Jenkins will be hard to be ported to another CI/CD system. The worst part is that it won't be easy for a developer to run the pipelines locally in order to have similar results as on the server. Dev/Prod parity should be not only on the tooling versions but also on the environment where code is built. Being able to run the pipeline locally would mean that a developer can also debug it if it's the case - it creates better visibility in the whole project - enabling a DevOps culture. Having an ops/build team that is managing the pipeline in secrecy is in my own view an anti-pattern. The devs will happily throw any issue away to the build/ops as "it works on their computers" - creating some knowledge towers. Also the posibility of running the pipelines on the local machines will ultimatly reduce the load on build machines and the queuing - improving again the metrics.
3. Mutability: The environment on which the pipeline runs should be immutable in order to produce consistent results. This is well treated by GitHub actions but for Jenkins (or other on premises CI/CD solutions) or even locally on the developer machine this is slightly complicated. However this can be solved if the pipeline can run in a container which is immutable. Many IDEs today offer the possibility of a development container that provides trusted and stable environments. This is extremely important nowadays in order to mitigate supply chain attacks. The immutability of the environment would mitigate issues as those described in Ken Thompson's paper from 1984 "Reflections on Trusting the Trust". This immutability contrasts partially with the caching needed for mitigating the speed but this can be solved quite elegantly nowadays with crypto methods so no cache poisoning can be inflicted.

Containerized pipelines solve both portability and mutability issues - so the developers could have both freedom of choosing tools and rigor for their builds in the same time. I learnt about a company that creates Visual Studio customized installations and pushes them every night to the developer's machines in order to solve the issues above. This is not only inefficient (hundred of gigabytes transferred and computers never in standby) but it is also error prone as a there are a lot of machine specific issues that might interfere - so in the end there is no certitude that the configuration is identical on all computers and that there is no drift. Imagine that in a large project a single library has some different settings - it will take hours for the developer to investigate an error caused by an obscure glitch that happened overnight. Running pipelines locally in the container would enable the developers use Rider or VS Code on Windows while still being able to test and build on trusted environments and deliver Linux software. Jenkins has this feature also, but is somehow exposing it in a clumsy way despite its huge value. Contrariwise GitHub Actions make it completely transparent for the consumers of the steps - one has to look in the action's source to know that it uses containers.

Speed is a crosscutting concern and can be addressed regardless if the pipeline runs or not in a container. A slow build won't get faster if done in docker. An aggregated approach - with the right tools could improve the development experience, speed and security of an organization.

#pipeline #containers