Delivery pipelines

Created on 2023-08-11 09:14

Published on 2023-08-11 10:18

The industry is buzzing currently with term as Continuous Delivery, Continuous Deployment an Pipelines. Pipelines are techniques to seed up the processing time by sequentially running a set of tasks on specialized stages, at every moment of time one step consumes the output of the previous one. The concept is not new, it has been started in the early 1900 when it was applied to factories and assembly lines. But the current pace of software development just put a lot of spotlight on them.

The software pipeline most often do a set of operations as check-out code from VCS, compile/build it, package it, run tests on it, deploy the artifacts to a repository and inform the stakeholders of the result, update various metrics and dashboards. These actions require a script to put things in the right order and check the results in every step.

In software the pipelines are generally defined through a specific language, either a visual one or a text based one - generally YAML or Groovy. These are run by specialized software as Jenkins, GitHub Actions, CircleCI, etc. The high level pipeline described in YAML is calling various other scripts that define the individual steps - like micro-operations.

The build step for example is the invocation of a Makefile, shell script or a Maven build. The pipeline will wait for the completion of the build step before passing the artifacts for testing or packaging. Such a situation is interesting because there will be two points of control in the pipeline. One at pipeline level and one at step level but their responsibilities will bleed into each-other.

This might lead to some problems:

1. Inefficient pipelines: If the result of a step can be guessed before the whole step completes the pipeline should fail fast. It makes no sense to wait for the completion. Imagine that you have a multi project build. In the trivial case the projects are built and packaged. But what if one of them has an issue? Should we wait for completion of all builds and then juts cancel the packaging step? What if there are modifications on a single project that has no egress dependencies? Again - it makes no sense to package all other projects as they haven't been modified. However with the split logic this is hard to achieve as the operations on pipeline level are quite coarse-grained. On the other hand most build tools (Gradle, MSBuild, ...) are perfectly able to run more parts of the process by themselves - create archives, containers, publish artifacts - and probably they should do it. Fail fast and incremental builds are essential for accelerated release cycles. The build tool is in a better position to understand what it is building by looking in the source code than the pipeline that is merely an orchestrator of some loosely coupled processes. Caching build results and artifacts, using parallel builds could reduce build times from 20 minutes to less than a minute - so some DORA metrics will look way better and will make both developers and managers less impatient. Speculative execution and rescheduling is well known in CPU pipelines - software engineers should have some inspiration from the clever solutions that hardware folks are successfully applying for more than two decades.
2. Portability: The high level pipeline logic is hard to be moved from one type of executor to the other. Pipelines built for Jenkins will be hard to be ported to another CI/CD system. The worst part is that it won't be easy for a developer to run the pipelines locally in order to have similar results as on the server. Dev/Prod parity should be not only on the tooling versions but also on the environment where code is built. Being able to run the pipeline locally would mean that a developer can also debug it if it's the case - it creates better visibility in the whole project - enabling a DevOps culture. Having an ops/build team that is managing the pipeline in secrecy is in my own view an anti-pattern. The devs will happily throw any issue away to the build/ops as "it works on their computers" - creating some knowledge towers. Also the posibility of running the pipelines on the local machines will ultimatly reduce the load on build machines and the queuing - improving again the metrics.
3. Mutability: The environment on which the pipeline runs should be immutable in order to produce consistent results. This is well treated by GitHub actions but for Jenkins (or other on premises CI/CD solutions) or even locally on the developer machine this is slightly complicated. However this can be solved if the pipeline can run in a container which is immutable. Many IDEs today offer the possibility of a development container that provides trusted and stable environments. This is extremely important nowadays in order to mitigate supply chain attacks. The immutability of the environment would mitigate issues as those described in Ken Thompson's paper from 1984 "Reflections on Trusting the Trust". This immutability contrasts partially with the caching needed for mitigating the speed but this can be solved quite elegantly nowadays with crypto methods so no cache poisoning can be inflicted.

Containerized pipelines solve both portability and mutability issues - so the developers could have both freedom of choosing tools and rigor for their builds in the same time. I learnt about a company that creates Visual Studio customized installations and pushes them every night to the developer's machines in order to solve the issues above. This is not only inefficient (hundred of gigabytes transferred and computers never in standby) but it is also error prone as a there are a lot of machine specific issues that might interfere - so in the end there is no certitude that the configuration is identical on all computers and that there is no drift. Imagine that in a large project a single library has some different settings - it will take hours for the developer to investigate an error caused by an obscure glitch that happened overnight. Running pipelines locally in the container would enable the developers use Rider or VS Code on Windows while still being able to test and build on trusted environments and deliver Linux software. Jenkins has this feature also, but is somehow exposing it in a clumsy way despite its huge value. Contrariwise GitHub Actions make it completely transparent for the consumers of the steps - one has to look in the action's source to know that it uses containers.

Speed is a crosscutting concern and can be addressed regardless if the pipeline runs or not in a container. A slow build won't get faster if done in docker. An aggregated approach - with the right tools could improve the development experience, speed and security of an organization.

#pipeline #containers

2022 Highlights

2022 was quite interesting. I made some changes in my toolbox.

The most interesting bits were:

1. Vercel and Supabase usage that worked flawlessly. 

2. I have ditched Docker completely and moved to Rancher Desktop and Podman. 

3. Still on Fedora 36 for all my machines. 

4. Moved some personal workloads to Oracle Cloud.

5. Visted Norway

6. Toured Romania

Felt dumb most of the year. And somehow powerless - cannot really get a grab, authority wise on the development of the  project.

Not much here, huh?

 I wrote most of my rants elsewhere. But this kind of sucks...

 

Legacy

Created on 2022-11-17 12:29

Published on 2022-11-19 19:56

The life that we are currently living is shaped in many ways by "legacy". In most cases, the word legacy carries positive connotations related to wealth, culture, and tradition. However, this is not the case with software. Here the word "legacy" has a lot of negative meanings associated with it. What is "legacy" in software? Why do we consider it bad?

Legacy software simply means that the code base is old, probably unmaintained, and hard to read. It says nothing about the value or the quality of the code, it just states that is old. Then why do we consider it bad? In the real world legacy is a source of wealth, something we grew upon. Probably the answer lies in our laziness. How many of us can fluently speak or write in Latin for example? Or classical Greek, old Norse, medieval French maybe? A few of us can, although is a pity, many fundamental works of mankind were written in those languages. Still, we do not read them in original, it is more convenient to read modern versions of those in a familiar language.

The same happens in software. Programmers are using contemporary languages and lost the ability to read older ones, so the code they stopped understanding became mysterious and potentially dangerous for them. As with classical languages, there is just of handful of people that have the patience to study and understand older code and those are the ones still able to explain the values that we, in our pride and ignorance, cannot see in the old code.

Many critical infrastructures and day-to-day codebases are "legacy", still, they work well, and they keep supporting our daily life. Banking still relies on Cobol and RPG, scientific computations still use Fortran, and operating systems still build on C.

I have to praise a relatively unknown "legacy" programming language and runtime: Concept. Concept is a 4GL (4th generation language) that started in Norway during the '80s. Googling it might yield 0 results, nevertheless, it delivers daily for a few million people. The language has all the features one would expect UI library, database connectivity, and it can run both server side and client side, it's kind of memory safe. The syntax seems a little dated but is still expressive enough to implement huge projects. As the community of Concept developers is not large it starts to lack new talent and also tooling. Despite this, there is still maintenance effort and is kept as much as possible in line with the latest industry trends: REST, JSON, MQs, 64bit code generation, containers, and such.

Understanding older languages is never easy. As I said before, due to our laziness we tend to ignore them and rewrite everything with no guarantee of a better job. Probably it is often a better idea to rewrite, but we first need to understand what we are replacing. We need tooling for understanding older code bases, especially when the original specifications of the software are lost. We need helper tools that guide us through the syntax and structure of the code and could provide handlers for the business logic already written in the old code bases enabling true reuse. Developing tooling that would transparently retarget old languages on new platforms will guarantee that the legacy received is still producing the expected results and we can build more interesting things in a more cooperative way. Rewriting software with feeble specifications or no specification at all, using the legacy system as a model but not fully grokking it, is in my opinion far more dangerous than keeping battle-tested code running.

The Five Ideals

Created on 2022-09-10 06:35

Published on 2022-09-10 14:15

Locality and Simplicity; Focus, Flow, and Joy; Improvement of Daily Work; Psychological Safety; Customer Focus - these are the five ideals of an organization that might blossom into a unicorn.

While some of them can be grown from the inside, starting from the development and operations teams and evolving them into a 'DevOps' culture, others are leveraged mostly by managers.

Helping teams improve for the first four ideals in the company creates more room for the latter. Those create a lot of turmoil and non-functional requirements but in the end, the price paid yields probably squared.

This is why managers that act toward the five ideals are as precious as mythical animals. This is also why when such a manager leaves it's quite sad.

To low code or to not low code

Created on 2022-01-31 19:43

Published on 2022-01-31 21:37

In 2005 I had my first encounter with a low-code platform. It was Alcatel's SCE/SDE later known as PrOSPer. It was a pretty capable environment for that period targeted for the generation of IN services composed of SIBs (Service Independent Blocks). Most of the IN abstractions were encapsulated inside the SIBs, the service creator was chaining the SIBs visually to create a new service. When it was needed a new SIB could be also created by implementing a set of interfaces needed for a SIB to run in its SLEE environment or to be seen inside the SCE/SDE interface.

The system generated C/C++ code, compiled it and then created the deployment descriptors. The idea behind it was a great one. It offered to domain experts the means for describing an IN service in a high-level language, with a good visual interface. Howeve,r there were some issues with it. The quality of the generated code was not optimal, the generator was a write-onlyy" one that missed any form of round-trip engineering, sometime the generated code had to be patched by hand and that made the high level description obsolete because the patched code couldn't be imported back in the SCE. Versioning of the code was also a nightmare in the SVN/ClearCase environment. All in all the developers were avoiding the SCE/SDE and were trying to handcraft their own code with simpler call-flows and better control over the implementation, keeping only the parts that were absolutely necessary to communicate with the SLEE.

The next low code experience I had was with Apache NiFi. NiFi is a flow based programming system that is used for all kind of data manipulation. It is again composed by a set of blocks that perform various operations on the data streams that were sent to their inputs. New blocks can be easily added or generic scripted blocks could be inserted so that the system is easily extensible. Versioning is still not great but is not terrible also, the system works pretty well but it's still missing some advanced features as meta-models, types and so on. There are also several other data-flow systems as NodeRed but most of them are in the same category of "write only" generators.

A major step forward was made by the Eclipse foundation with its EclipseSirius modelling workbench. This one was indeed based on higher level abstractions. Being constructed on top of GEF/EMF it had superpowers as reverse engineering, round-trip engineering, and easy creation of user interfaces that were not limited just to a data flow like systems before it.

The interesting questions that those low-code solutions raise is in the sphere of languages. What is a language? What are the entities a language operates on? What are the rules that make the language correct?

Professor Jordi Cabot reaches the conclusion (see slide 13) that low-code is in fact a fancy syntax and a marketing term used for some model-driven architectures and development environments. All the "low code" systems described are in fact visual syntaxes that describe the interaction between some entities (called blocks, or SIBS).

Having models and meta-models make the "low code" even more interesting as now other abstractions can be created. Constructions in the new language can have some formal semantics, type systems can be applied. Testing can be moved from generated code to the visual syntax itself. Probably, as also Eric Evans describes in its Domain Driven Design book the most important quality of the "low code" is that it enables the human interaction and comprehension. Maybe lawyers will not operate with visual law designing tools (although it would be interesting) but for sure they could grasp models of law, fact, proof, patent that would permit them to use a low code solution in their own bounded context.

So far the "low code" solutions I referred to were visual but this is not the only way of having it. There can be for sure low code textual languages or DSLs.

The above example encapsulates lots of domain entities: tax payer, taxable income - entities that would be pretty hard to be manipulated in normal programming languages due to the amount of boilerplate code needed. If we make another step maybe the situation could look like:

If we accept that a "low code" visual solution is just an alternative syntax for a DSL then we can link the two worlds. Most of the problems that were hard to solve in the visual syntax can be now rewritten in the DSL so we get many advantages:

• as before we can have type systems and meta-models
• we get an AST that facilitates transformations of the models
• we get sane versioning as textual representations are VCS friendly.
• we get somehow a human understandable/maintainable code

There are also two other aspects that also emerge here. One is the possibility of a "language engineering workbench" where domain specific languages to be created and augmented with thick semantic layers. This has huge applicability in domains where is a lot of formalism and already developed systems.

The other interesting one, that in my opinion has a major impact, is the "projectional editing" that would permit the language creator to manipulate the language internals easier, in some situations without resorting to traditional lexers or parsers as the language internals would be exposed as models in their own right. Modern language workbenches already offer great tooling for projectional editing making the development of DSLs easier.

What I am trying to conclude here is that "low code" is not a single term but rather a combination of paradigms so in my opinion evaluation of "low code" cannot be done in total separation from domain modeling and language engineering. My arguments are probably naive, but I see value in higher level systems description, although this is not always necessary or desirable. Higher level abstractions are beneficial not only in the human-machine languages but they are also great for human-to-human communication as they reduce the miscommunication. Well human engineered languages could result into better implementation.

Many thanks to Jennek Geels for introducing me to his concepts of domain modeling.

References:

https://modeling-languages.com/low-code-vs-model-driven/

https://martinfowler.com/dsl.html

https://martinfowler.com/bliki/DslBoundary.html

Dutch tax DSL: https://resources.jetbrains.com/storage/products/mps/docs/MPS_DTO_Case_Study.pdf

Language workbench: https://web.cecs.pdx.edu/~apt/onward14.pdf

Technology Grafting

Created on 2021-12-23 09:48

Published on 2021-12-23 17:01

Wikipedia defines grafting as: "a horticultural technique whereby tissues of plants are joined to continue their growth together". Grafting is often used for producing new varieties of fruits on older trunks that are more resilient or adapted to a climate.

When working with legacy software we can see modernization as grafting. The rootstock is the older technology - proven and battle-tested while the scion is a newer technology that will enhance the rootstock's qualities. The technological grafting needs two things: interfaces/protocols and encapsulation.

The interface layer ensures that the data can be understood by both scion and rootstock. If the rootstock uses some very specific encodings (EBCDIC, various other locale-specific charsets) then the translation between them must be consistent in both directions. Things seem to be easy with textual protocols (although there might be also some issues) than with binary proprietary protocols. If a fast bidirectional translation is not possible (very different protocols) then a middleware might be necessary but in this case, it would be best that this middleware

On the other hand, encapsulation permits the scion to maintain all its properties and continue working as if it would be still attached to its original trunk. When grafting a pear branch on an apple tree the pear branch will continue to function as pear and will not hinder the apple trunk. The same behavior should be present when grafting software components. However, this strong encapsulation requires the open interfaces discussed before.

One of the most interesting technological graftings I have seen lately is in the form of modernizing older web applications using micro frontends packed as web components. The idea is not a novel one, web components are one of the ways of composing micro frontends, but this has massive implications for all the functional and nonfunctional aspects of the application. Imagine old PHP or Struts code bases getting an instant makeup with nice web components that reduce the server load and break monolithic applications into simpler parts. The web components isolate the new developments from the old code base and will permit the evolution of the older code base.

Grafting techniques and patterns might prove very useful in all kinds of modernization scenarios as there are a lot of use cases for this. The use cases can range from, as I wrote above, UI modernization to more interesting ones like adapting 50 years old systems to web or IoT. If the results of grafting are successful then this might help the rootstock also evolve (like Autocad's migration to the web). Grafting permits the progressive evolution of the software without disruptions in base technologies used at a minimal extra cost. The complicated problem is to determine where is the best place to insert the scion and how to make it behave on the same security and functional constraints as the rest of the application.