Locality and Simplicity; Focus, Flow, and Joy; Improvement of Daily Work; Psychological Safety; Customer Focus - these are the five ideals of an organization that might blossom into a unicorn.
While some of them can be grown from the inside, starting from the development and operations teams and evolving them into a 'DevOps' culture, others are leveraged mostly by managers.
Helping teams improve for the first four ideals in the company creates more room for the latter. Those create a lot of turmoil and non-functional requirements but in the end, the price paid yields probably squared.
This is why managers that act toward the five ideals are as precious as mythical animals. This is also why when such a manager leaves it's quite sad.
In 2005 I had my first encounter with a low-code platform. It was Alcatel's SCE/SDE later known as PrOSPer. It was a pretty capable environment for that period targeted for the generation of IN services composed of SIBs (Service Independent Blocks). Most of the IN abstractions were encapsulated inside the SIBs, the service creator was chaining the SIBs visually to create a new service. When it was needed a new SIB could be also created by implementing a set of interfaces needed for a SIB to run in its SLEE environment or to be seen inside the SCE/SDE interface.
The system generated C/C++ code, compiled it and then created the deployment descriptors. The idea behind it was a great one. It offered to domain experts the means for describing an IN service in a high-level language, with a good visual interface. Howeve,r there were some issues with it. The quality of the generated code was not optimal, the generator was a write-onlyy" one that missed any form of round-trip engineering, sometime the generated code had to be patched by hand and that made the high level description obsolete because the patched code couldn't be imported back in the SCE. Versioning of the code was also a nightmare in the SVN/ClearCase environment. All in all the developers were avoiding the SCE/SDE and were trying to handcraft their own code with simpler call-flows and better control over the implementation, keeping only the parts that were absolutely necessary to communicate with the SLEE.
The next low code experience I had was with Apache NiFi. NiFi is a flow based programming system that is used for all kind of data manipulation. It is again composed by a set of blocks that perform various operations on the data streams that were sent to their inputs. New blocks can be easily added or generic scripted blocks could be inserted so that the system is easily extensible. Versioning is still not great but is not terrible also, the system works pretty well but it's still missing some advanced features as meta-models, types and so on. There are also several other data-flow systems as NodeRed but most of them are in the same category of "write only" generators.
A major step forward was made by the Eclipse foundation with its EclipseSirius modelling workbench. This one was indeed based on higher level abstractions. Being constructed on top of GEF/EMF it had superpowers as reverse engineering, round-trip engineering, and easy creation of user interfaces that were not limited just to a data flow like systems before it.
The interesting questions that those low-code solutions raise is in the sphere of languages. What is a language? What are the entities a language operates on? What are the rules that make the language correct?
Professor Jordi Cabot reaches the conclusion (see slide 13) that low-code is in fact a fancy syntax and a marketing term used for some model-driven architectures and development environments. All the "low code" systems described are in fact visual syntaxes that describe the interaction between some entities (called blocks, or SIBS).
Having models and meta-models make the "low code" even more interesting as now other abstractions can be created. Constructions in the new language can have some formal semantics, type systems can be applied. Testing can be moved from generated code to the visual syntax itself. Probably, as also Eric Evans describes in its Domain Driven Design book the most important quality of the "low code" is that it enables the human interaction and comprehension. Maybe lawyers will not operate with visual law designing tools (although it would be interesting) but for sure they could grasp models of law, fact, proof, patent that would permit them to use a low code solution in their own bounded context.
So far the "low code" solutions I referred to were visual but this is not the only way of having it. There can be for sure low code textual languages or DSLs.
The above example encapsulates lots of domain entities: tax payer, taxable income - entities that would be pretty hard to be manipulated in normal programming languages due to the amount of boilerplate code needed. If we make another step maybe the situation could look like:
If we accept that a "low code" visual solution is just an alternative syntax for a DSL then we can link the two worlds. Most of the problems that were hard to solve in the visual syntax can be now rewritten in the DSL so we get many advantages:
as before we can have type systems and meta-models
we get an AST that facilitates transformations of the models
we get sane versioning as textual representations are VCS friendly.
we get somehow a human understandable/maintainable code
There are also two other aspects that also emerge here. One is the possibility of a "language engineering workbench" where domain specific languages to be created and augmented with thick semantic layers. This has huge applicability in domains where is a lot of formalism and already developed systems.
The other interesting one, that in my opinion has a major impact, is the "projectional editing" that would permit the language creator to manipulate the language internals easier, in some situations without resorting to traditional lexers or parsers as the language internals would be exposed as models in their own right. Modern language workbenches already offer great tooling for projectional editing making the development of DSLs easier.
What I am trying to conclude here is that "low code" is not a single term but rather a combination of paradigms so in my opinion evaluation of "low code" cannot be done in total separation from domain modeling and language engineering. My arguments are probably naive, but I see value in higher level systems description, although this is not always necessary or desirable. Higher level abstractions are beneficial not only in the human-machine languages but they are also great for human-to-human communication as they reduce the miscommunication. Well human engineered languages could result into better implementation.
Many thanks to Jennek Geels for introducing me to his concepts of domain modeling.
Wikipedia defines grafting as: "a horticultural technique whereby tissues of plants are joined to continue their growth together". Grafting is often used for producing new varieties of fruits on older trunks that are more resilient or adapted to a climate.
When working with legacy software we can see modernization as grafting. The rootstock is the older technology - proven and battle-tested while the scion is a newer technology that will enhance the rootstock's qualities. The technological grafting needs two things: interfaces/protocols and encapsulation.
The interface layer ensures that the data can be understood by both scion and rootstock. If the rootstock uses some very specific encodings (EBCDIC, various other locale-specific charsets) then the translation between them must be consistent in both directions. Things seem to be easy with textual protocols (although there might be also some issues) than with binary proprietary protocols. If a fast bidirectional translation is not possible (very different protocols) then a middleware might be necessary but in this case, it would be best that this middleware
On the other hand, encapsulation permits the scion to maintain all its properties and continue working as if it would be still attached to its original trunk. When grafting a pear branch on an apple tree the pear branch will continue to function as pear and will not hinder the apple trunk. The same behavior should be present when grafting software components. However, this strong encapsulation requires the open interfaces discussed before.
One of the most interesting technological graftings I have seen lately is in the form of modernizing older web applications using micro frontends packed as web components. The idea is not a novel one, web components are one of the ways of composing micro frontends, but this has massive implications for all the functional and nonfunctional aspects of the application. Imagine old PHP or Struts code bases getting an instant makeup with nice web components that reduce the server load and break monolithic applications into simpler parts. The web components isolate the new developments from the old code base and will permit the evolution of the older code base.
Grafting techniques and patterns might prove very useful in all kinds of modernization scenarios as there are a lot of use cases for this. The use cases can range from, as I wrote above, UI modernization to more interesting ones like adapting 50 years old systems to web or IoT. If the results of grafting are successful then this might help the rootstock also evolve (like Autocad's migration to the web). Grafting permits the progressive evolution of the software without disruptions in base technologies used at a minimal extra cost. The complicated problem is to determine where is the best place to insert the scion and how to make it behave on the same security and functional constraints as the rest of the application.
Most of us have started with a small application that solved a business need. Then we had some success with it therefore we added functionalities, improved UI, made it multi tenant and so forth. In the meanwhile the code has grown and every "if" condition we added made it more complex.
We added "ifs" because we had to respond to changes created by customer requests, changing laws and policies, access to different markets. Some of us were smart and resorted early to design patterns but some of us continued with the old way of adding ifs... In the end application is huge and hard to debug because it's codebase no longer reflects it's initial purpose. There is no "isomorphism" between the problem that application solved and the implementation of the application. Even applications that were well engineered have this issue as the changes they accumulated over time are not described in the original universe where the application was spawned, in the business part, but rather the engineering created approximations in the implementation domain that are maintained. To make a forced comparison the situation is similar to getting the image of a forest in autumn simply by creating a huge palette of colours without thinking of what we are trying to represent with them.
So moving along this line, how do we keep applications from going adrift? How do we keep in sync the reality where applications emerged with their implemented behaviour? Here formalism appears. Describing application behaviour in a formal way, close to what people understand is the key. This formalism have many names ranging from "Domain Driven Design" to DSLs but all focus on capturing the domain information on a higher level, not directly in code. Code is generally a byproduct of those, formal descriptions are needed for any DSL and low-code solution because, quoting Markus Volter: "models are semantically rich for their domains" whilst code is not semantically rich, just behaviouraly rich.
One of the challenges I am thinking about is: how do we retrofit formalism into application so that we make them more maintainable? There is no definitive answer but especially for legacy application refactoring to models is a key. We should start small, refactor user interface behaviour for example to state charts (in form of Redux or XState). In the meanwhile the refactored interactions could be again reorganized both server and client side into some business process descriptions (it would be ideal to use a dedicated DSL for those in this stage - but this is generally hard, so refactoring to state charts is a good beginning). Iterating over this a couple of time we will reach better models on the domain and clearer actions defined on the set of the models - hence we get a crude formal definition of the domain. At this point the process, described probably in a specific language can be understood again by business people therefore they can evolve it while it is still generating lower level formalisms (as the statecharts) that developers or specific tools can generate code from.
The new developments nowadays have the chance of starting clean as the current tooling is quite advanced (Jetbrains MPS, Eclipse Sirius). New projects can start from DSLs or from BPMN visual representations and generate rich models that can be visually manipulated in low code solutions. Moreover a repository of well described models and processes can be shared inside a company so people can reuse the high level abstractions, manipulate them in their own code and getting interoperability and common behaviour for free. Also testing is positively impacted as tests clarity increases because models carry more semantics inside. What would be easier to test: 500 lines of written code or 12-15 lines of DSL or visual representation? Where are the errors simpler to catch?
Well crafted formal definitions reduce the implementation and design burden for the implementation team as they shunt most of the repetitive design and boilerplate tasks in an application (rich models already contain the logic that otherwise has to be expressed and maintained in code). Continuous effort of expressing behaviours in high level, formal languages and refactoring older code towards the same seems a way of increasing maintainability and quality. DSLs, modeling and low-code are in fact facets of the same base problem - how to capture and communicate succinctly, correctly and in a non ambiguous way domain knowledge both for people and for machines.
PS: There are better approximations of a process definitions than Amazon's StepFunctions or BPMN. To quote a friend - they are like GoTo programming and do not incorporate transactional aspects.
More than a year ago we were talking about backups. My solution then was something based on duplicity and rclone and I was explaining it to one of my colleagues who had another approach. I was pretty happy with it although it had some quirks. Then one of the other colleagues asked us: "Have you tried restic?". I did not know about it and I found it quite interesting. Months passed and I had to implement again an off-site backup system. Although I had the old tools in the pocket I remmembered restic and I soon realised that is the perfect fit for my incremental backup task.
I imagine that I would have lost probably some days trying to use other tools and craft an adapter layer on top of the old tools. Listening to others brang me pretty much advantage both in terms of efficiency as well as in terms of knowledge - another tool on my belt.
One thing that I miss while working remotely is the posibility of sharing knowledge in an informal way. When enginers discuss around the coffee machine some crazy geekish ideas (from Raspberry Pi autonomous lawnmowers to a new programming language) or they lucidly dream some interesting features in their code then innovation is about to happen. Many things that are in an incumbent state are hard to find in mainstream publications and blogs but sometimes there is one who has a passion about "obscure-ish" (eg. https://qconsf.com/system/files/presentation-slides/control_theory_in_container_orchestration.pdf) subjects and a wise remark at the right moment can cut months of development because that insight solves in a creative way an otherwise hard task.
I had my share of these happenings: timer_fd (simplified state machines), NancyFX (instead of all OWIN stuff), Swig (for genetrating bindings) and many others but the key fact was that I was among peers who gave me back at one moment valuable information. This is one of the things that videoconference/Skype/Meet still cannot substitute yet. They are fantastic tools for this time but unfortunately they are still unable to create the context for this kind of communication. I remember that many ideas that were later translated into patents emerged in this kind of informal gatherings and sometime a "What about..." created the spark.
I haven't said much lately on the blog nor on LinkedIn. It was quiet and dull-ish in day to day. The year started pretty bad, my father-in-law died in January. In July my daughter had her exams for Gymnasium.
Basically those two moments are what I can remmember from 2021 so far.
Shards of events:
1. Always backup SSH keys.
2. At my previous job they are still not able to impement some cloud strategy as I forecasted. People started to flee from there
3. I slacked with all my side projects
4. Lost the will of going out - something that was a pleasure slowly turned into a burden. Also lost most of the social skills.
5. Minor but reccurent health issues. No Corona though
6. Got the COVID-19 shot
7. Hackintoshed my Huawei - works decently well with Intel WiFi (10x itlwm)
8. Pleasant working experience in Fedora 34.
9. Rust langusge is getting better and better
10. Svelte is pretty nice although it has quirks
11. Oracle seems to focus om Micronaut - probably a second bet that I won.
"Anything that can go wrong will go wrong" (Murphy's Law)
"Preppers" are persons that spend a lot of time preparing for an imminent disaster. In case of doomsday, they might survive it as they planned minutely for any possible outcome long before. Preppers are stocking equipment, clothes, food and often create shelters so that they could face even the most atrocious situations with relative calm. Prepping puts some financial strain on people but not as much one might think as preppers' most important assets are knowledge and improvisation. Regardless how elaborated the disaster recovery plan is, it's never perfect so creativity should always be welcomed.
Preppers use some techniques that IT should learn from:
Rehearsal - preppers really exercise their plan. They try surviving wilderness, with their bushcraft skills, thus testing their theories and equipment. In IT from time to time it's worthy simulating a nasty situation in the infrastructure. It doesn't need to be a complicated one that would imply the whole SimianArmy but at least simulate a disaster, solve it and take notes. Ask colleagues to stop a service or make a ethernet loop in a switch. Learn where to look for clues and when the incident is solved try to automate the solution. There are organizations that take rehearsals very seriously. I've been part of probably three fire simulation drills at Adastral Park in less than a month but I can say that It was the only occasion where I've seen so much calm and organization as people knew their drill.
Share information - preppers are sometime organized in clubs and they share lots of relevant information on materials and techniques. Community driven DRP might be interesting as people come with different viewpoints that might reveal flaws or provide better solutions. Information sharing might be the most important source of learning.
Use cheap and easy to find materials. For example many things can be simulated on scaled down versions of the real systems. A full K8S/EKS cluster can be simulated with a petty RaspberryPi running K3S. Most of the things will be the same but costs will be negligible. The differences can be documented and there are also lots of third parties that can simulate other services cheaply (Lambdas, Object Storage). In some situation one will have to improvise massively to recover from a disaster and then knowing what to do with almost off-the-shelve components becomes invaluable.
Invest in understandable systems - preppers try to understand nature and mechanisms so that they increase their survival odds. Although automation is a must it should be transparent to the engineers and clearly present what's behind the scenes. If the state is extremely bad and automation magic stopped working then understanding the system might be handy at least to go back in a restorable state with minimum effort. I for one was in a similar situation when I discovered that the backup I made had to be restored on a disk with different physical characteristics. Knowing what was the backup process I was able to hack the restore on the new drive although the tool was not supporting it.
Keep stashes - preppers often have hidden stashes of food and tools around the house. Engineers should also make stashes of resources if possible offsite or in different clouds. If something goes horribly wrong at least some things could be partially running. An OCI solution, replicated on a different vendor's platform, would make a lot of sense so if a provider is unavailable restoring the solution on another would not mean an entire rewrite but rather a reconfiguration. Vendor agnostic IAC solutions and CNI packed services are handy in this case - they can be easily run on different environments.
Have spare batteries - preppers don' t rely on a single centralised solution – e.g. electricity; they always have alternate independent energy sources that they can use - batteries, generators, etc. In the light of the recent OVH fire let's imagine the following scenario: the company's automation solution (Chef, Puppet, Octopus) is also affected somehow then the situation will be similar to electricity or transport failure. On the other hand, having the IAC implemented through standalone tools (Terraform, Ansible) increase the resilience as there is no longer a single point of failure, any engineer could replicate the infrastructure from his/her own machine as long as one can connect to the datacenter/cloud.
Learn how to use and build tools - preppers spend time in learning how to build bows, arrows, tents and also how to use them. Off the shelf tools are fine (backup/restore, IAC, etc) but sometime one needs to do specific operations (e.g. recover data from an ZFS formatted drive). Then building small and specific tools would be invaluable so a programming language that permits rapid development is handy - currently I see that people are looking towards Python and Go.
What I am trying to say is that pessimism and preparation are key for survivability in case of a disaster. Even a digital one.
