Legacy

Created on 2022-11-17 12:29

Published on 2022-11-19 19:56

The life that we are currently living is shaped in many ways by "legacy". In most cases, the word legacy carries positive connotations related to wealth, culture, and tradition. However, this is not the case with software. Here the word "legacy" has a lot of negative meanings associated with it. What is "legacy" in software? Why do we consider it bad?

Legacy software simply means that the code base is old, probably unmaintained, and hard to read. It says nothing about the value or the quality of the code, it just states that is old. Then why do we consider it bad? In the real world legacy is a source of wealth, something we grew upon. Probably the answer lies in our laziness. How many of us can fluently speak or write in Latin for example? Or classical Greek, old Norse, medieval French maybe? A few of us can, although is a pity, many fundamental works of mankind were written in those languages. Still, we do not read them in original, it is more convenient to read modern versions of those in a familiar language.

The same happens in software. Programmers are using contemporary languages and lost the ability to read older ones, so the code they stopped understanding became mysterious and potentially dangerous for them. As with classical languages, there is just of handful of people that have the patience to study and understand older code and those are the ones still able to explain the values that we, in our pride and ignorance, cannot see in the old code.

Many critical infrastructures and day-to-day codebases are "legacy", still, they work well, and they keep supporting our daily life. Banking still relies on Cobol and RPG, scientific computations still use Fortran, and operating systems still build on C.

I have to praise a relatively unknown "legacy" programming language and runtime: Concept. Concept is a 4GL (4th generation language) that started in Norway during the '80s. Googling it might yield 0 results, nevertheless, it delivers daily for a few million people. The language has all the features one would expect UI library, database connectivity, and it can run both server side and client side, it's kind of memory safe. The syntax seems a little dated but is still expressive enough to implement huge projects. As the community of Concept developers is not large it starts to lack new talent and also tooling. Despite this, there is still maintenance effort and is kept as much as possible in line with the latest industry trends: REST, JSON, MQs, 64bit code generation, containers, and such.

Understanding older languages is never easy. As I said before, due to our laziness we tend to ignore them and rewrite everything with no guarantee of a better job. Probably it is often a better idea to rewrite, but we first need to understand what we are replacing. We need tooling for understanding older code bases, especially when the original specifications of the software are lost. We need helper tools that guide us through the syntax and structure of the code and could provide handlers for the business logic already written in the old code bases enabling true reuse. Developing tooling that would transparently retarget old languages on new platforms will guarantee that the legacy received is still producing the expected results and we can build more interesting things in a more cooperative way. Rewriting software with feeble specifications or no specification at all, using the legacy system as a model but not fully grokking it, is in my opinion far more dangerous than keeping battle-tested code running.

The Five Ideals

Created on 2022-09-10 06:35

Published on 2022-09-10 14:15

Locality and Simplicity; Focus, Flow, and Joy; Improvement of Daily Work; Psychological Safety; Customer Focus - these are the five ideals of an organization that might blossom into a unicorn.

While some of them can be grown from the inside, starting from the development and operations teams and evolving them into a 'DevOps' culture, others are leveraged mostly by managers.

Helping teams improve for the first four ideals in the company creates more room for the latter. Those create a lot of turmoil and non-functional requirements but in the end, the price paid yields probably squared.

This is why managers that act toward the five ideals are as precious as mythical animals. This is also why when such a manager leaves it's quite sad.

To low code or to not low code

Created on 2022-01-31 19:43

Published on 2022-01-31 21:37

In 2005 I had my first encounter with a low-code platform. It was Alcatel's SCE/SDE later known as PrOSPer. It was a pretty capable environment for that period targeted for the generation of IN services composed of SIBs (Service Independent Blocks). Most of the IN abstractions were encapsulated inside the SIBs, the service creator was chaining the SIBs visually to create a new service. When it was needed a new SIB could be also created by implementing a set of interfaces needed for a SIB to run in its SLEE environment or to be seen inside the SCE/SDE interface.

The system generated C/C++ code, compiled it and then created the deployment descriptors. The idea behind it was a great one. It offered to domain experts the means for describing an IN service in a high-level language, with a good visual interface. Howeve,r there were some issues with it. The quality of the generated code was not optimal, the generator was a write-onlyy" one that missed any form of round-trip engineering, sometime the generated code had to be patched by hand and that made the high level description obsolete because the patched code couldn't be imported back in the SCE. Versioning of the code was also a nightmare in the SVN/ClearCase environment. All in all the developers were avoiding the SCE/SDE and were trying to handcraft their own code with simpler call-flows and better control over the implementation, keeping only the parts that were absolutely necessary to communicate with the SLEE.

The next low code experience I had was with Apache NiFi. NiFi is a flow based programming system that is used for all kind of data manipulation. It is again composed by a set of blocks that perform various operations on the data streams that were sent to their inputs. New blocks can be easily added or generic scripted blocks could be inserted so that the system is easily extensible. Versioning is still not great but is not terrible also, the system works pretty well but it's still missing some advanced features as meta-models, types and so on. There are also several other data-flow systems as NodeRed but most of them are in the same category of "write only" generators.

A major step forward was made by the Eclipse foundation with its EclipseSirius modelling workbench. This one was indeed based on higher level abstractions. Being constructed on top of GEF/EMF it had superpowers as reverse engineering, round-trip engineering, and easy creation of user interfaces that were not limited just to a data flow like systems before it.

The interesting questions that those low-code solutions raise is in the sphere of languages. What is a language? What are the entities a language operates on? What are the rules that make the language correct?

Professor Jordi Cabot reaches the conclusion (see slide 13) that low-code is in fact a fancy syntax and a marketing term used for some model-driven architectures and development environments. All the "low code" systems described are in fact visual syntaxes that describe the interaction between some entities (called blocks, or SIBS).

Having models and meta-models make the "low code" even more interesting as now other abstractions can be created. Constructions in the new language can have some formal semantics, type systems can be applied. Testing can be moved from generated code to the visual syntax itself. Probably, as also Eric Evans describes in its Domain Driven Design book the most important quality of the "low code" is that it enables the human interaction and comprehension. Maybe lawyers will not operate with visual law designing tools (although it would be interesting) but for sure they could grasp models of law, fact, proof, patent that would permit them to use a low code solution in their own bounded context.

So far the "low code" solutions I referred to were visual but this is not the only way of having it. There can be for sure low code textual languages or DSLs.

The above example encapsulates lots of domain entities: tax payer, taxable income - entities that would be pretty hard to be manipulated in normal programming languages due to the amount of boilerplate code needed. If we make another step maybe the situation could look like:

If we accept that a "low code" visual solution is just an alternative syntax for a DSL then we can link the two worlds. Most of the problems that were hard to solve in the visual syntax can be now rewritten in the DSL so we get many advantages:

• as before we can have type systems and meta-models
• we get an AST that facilitates transformations of the models
• we get sane versioning as textual representations are VCS friendly.
• we get somehow a human understandable/maintainable code

There are also two other aspects that also emerge here. One is the possibility of a "language engineering workbench" where domain specific languages to be created and augmented with thick semantic layers. This has huge applicability in domains where is a lot of formalism and already developed systems.

The other interesting one, that in my opinion has a major impact, is the "projectional editing" that would permit the language creator to manipulate the language internals easier, in some situations without resorting to traditional lexers or parsers as the language internals would be exposed as models in their own right. Modern language workbenches already offer great tooling for projectional editing making the development of DSLs easier.

What I am trying to conclude here is that "low code" is not a single term but rather a combination of paradigms so in my opinion evaluation of "low code" cannot be done in total separation from domain modeling and language engineering. My arguments are probably naive, but I see value in higher level systems description, although this is not always necessary or desirable. Higher level abstractions are beneficial not only in the human-machine languages but they are also great for human-to-human communication as they reduce the miscommunication. Well human engineered languages could result into better implementation.

Many thanks to Jennek Geels for introducing me to his concepts of domain modeling.

References:

https://modeling-languages.com/low-code-vs-model-driven/

https://martinfowler.com/dsl.html

https://martinfowler.com/bliki/DslBoundary.html

Dutch tax DSL: https://resources.jetbrains.com/storage/products/mps/docs/MPS_DTO_Case_Study.pdf

Language workbench: https://web.cecs.pdx.edu/~apt/onward14.pdf

Technology Grafting

Created on 2021-12-23 09:48

Published on 2021-12-23 17:01

Wikipedia defines grafting as: "a horticultural technique whereby tissues of plants are joined to continue their growth together". Grafting is often used for producing new varieties of fruits on older trunks that are more resilient or adapted to a climate.

When working with legacy software we can see modernization as grafting. The rootstock is the older technology - proven and battle-tested while the scion is a newer technology that will enhance the rootstock's qualities. The technological grafting needs two things: interfaces/protocols and encapsulation.

The interface layer ensures that the data can be understood by both scion and rootstock. If the rootstock uses some very specific encodings (EBCDIC, various other locale-specific charsets) then the translation between them must be consistent in both directions. Things seem to be easy with textual protocols (although there might be also some issues) than with binary proprietary protocols. If a fast bidirectional translation is not possible (very different protocols) then a middleware might be necessary but in this case, it would be best that this middleware

On the other hand, encapsulation permits the scion to maintain all its properties and continue working as if it would be still attached to its original trunk. When grafting a pear branch on an apple tree the pear branch will continue to function as pear and will not hinder the apple trunk. The same behavior should be present when grafting software components. However, this strong encapsulation requires the open interfaces discussed before.

One of the most interesting technological graftings I have seen lately is in the form of modernizing older web applications using micro frontends packed as web components. The idea is not a novel one, web components are one of the ways of composing micro frontends, but this has massive implications for all the functional and nonfunctional aspects of the application. Imagine old PHP or Struts code bases getting an instant makeup with nice web components that reduce the server load and break monolithic applications into simpler parts. The web components isolate the new developments from the old code base and will permit the evolution of the older code base.

Grafting techniques and patterns might prove very useful in all kinds of modernization scenarios as there are a lot of use cases for this. The use cases can range from, as I wrote above, UI modernization to more interesting ones like adapting 50 years old systems to web or IoT. If the results of grafting are successful then this might help the rootstock also evolve (like Autocad's migration to the web). Grafting permits the progressive evolution of the software without disruptions in base technologies used at a minimal extra cost. The complicated problem is to determine where is the best place to insert the scion and how to make it behave on the same security and functional constraints as the rest of the application.

Necessary formalism

Created on 2021-12-19 14:39

Published on 2021-12-19 16:02

Most of us have started with a small application that solved a business need. Then we had some success with it therefore we added functionalities, improved UI, made it multi tenant and so forth. In the meanwhile the code has grown and every "if" condition we added made it more complex.

We added "ifs" because we had to respond to changes created by customer requests, changing laws and policies, access to different markets. Some of us were smart and resorted early to design patterns but some of us continued with the old way of adding ifs... In the end application is huge and hard to debug because it's codebase no longer reflects it's initial purpose. There is no "isomorphism" between the problem that application solved and the implementation of the application. Even applications that were well engineered have this issue as the changes they accumulated over time are not described in the original universe where the application was spawned, in the business part, but rather the engineering created approximations in the implementation domain that are maintained. To make a forced comparison the situation is similar to getting the image of a forest in autumn simply by creating a huge palette of colours without thinking of what we are trying to represent with them.

So moving along this line, how do we keep applications from going adrift? How do we keep in sync the reality where applications emerged with their implemented behaviour? Here formalism appears. Describing application behaviour in a formal way, close to what people understand is the key. This formalism have many names ranging from "Domain Driven Design" to DSLs but all focus on capturing the domain information on a higher level, not directly in code. Code is generally a byproduct of those, formal descriptions are needed for any DSL and low-code solution because, quoting Markus Volter: "models are semantically rich for their domains" whilst code is not semantically rich, just behaviouraly rich.

One of the challenges I am thinking about is: how do we retrofit formalism into application so that we make them more maintainable? There is no definitive answer but especially for legacy application refactoring to models is a key. We should start small, refactor user interface behaviour for example to state charts (in form of Redux or XState). In the meanwhile the refactored interactions could be again reorganized both server and client side into some business process descriptions (it would be ideal to use a dedicated DSL for those in this stage - but this is generally hard, so refactoring to state charts is a good beginning). Iterating over this a couple of time we will reach better models on the domain and clearer actions defined on the set of the models - hence we get a crude formal definition of the domain. At this point the process, described probably in a specific language can be understood again by business people therefore they can evolve it while it is still generating lower level formalisms (as the statecharts) that developers or specific tools can generate code from.

The new developments nowadays have the chance of starting clean as the current tooling is quite advanced (Jetbrains MPS, Eclipse Sirius). New projects can start from DSLs or from BPMN visual representations and generate rich models that can be visually manipulated in low code solutions. Moreover a repository of well described models and processes can be shared inside a company so people can reuse the high level abstractions, manipulate them in their own code and getting interoperability and common behaviour for free. Also testing is positively impacted as tests clarity increases because models carry more semantics inside. What would be easier to test: 500 lines of written code or 12-15 lines of DSL or visual representation? Where are the errors simpler to catch?

Well crafted formal definitions reduce the implementation and design burden for the implementation team as they shunt most of the repetitive design and boilerplate tasks in an application (rich models already contain the logic that otherwise has to be expressed and maintained in code). Continuous effort of expressing behaviours in high level, formal languages and refactoring older code towards the same seems a way of increasing maintainability and quality. DSLs, modeling and low-code are in fact facets of the same base problem - how to capture and communicate succinctly, correctly and in a non ambiguous way domain knowledge both for people and for machines.

PS: There are better approximations of a process definitions than Amazon's StepFunctions or BPMN. To quote a friend - they are like GoTo programming and do not incorporate transactional aspects.

Communication and innovation

Created on 2021-07-29 15:23

Published on 2021-07-29 15:55

More than a year ago we were talking about backups. My solution then was something based on duplicity and rclone and I was explaining it to one of my colleagues who had another approach. I was pretty happy with it although it had some quirks. Then one of the other colleagues asked us: "Have you tried restic?". I did not know about it and I found it quite interesting. Months passed and I had to implement again an off-site backup system. Although I had the old tools in the pocket I remmembered restic and I soon realised that is the perfect fit for my incremental backup task.

I imagine that I would have lost probably some days trying to use other tools and craft an adapter layer on top of the old tools. Listening to others brang me pretty much advantage both in terms of efficiency as well as in terms of knowledge - another tool on my belt.

One thing that I miss while working remotely is the posibility of sharing knowledge in an informal way. When enginers discuss around the coffee machine some crazy geekish ideas (from Raspberry Pi autonomous lawnmowers to a new programming language) or they lucidly dream some interesting features in their code then innovation is about to happen. Many things that are in an incumbent state are hard to find in mainstream publications and blogs but sometimes there is one who has a passion about "obscure-ish" (eg. https://qconsf.com/system/files/presentation-slides/control_theory_in_container_orchestration.pdf) subjects and a wise remark at the right moment can cut months of development because that insight solves in a creative way an otherwise hard task.

I had my share of these happenings: timer_fd (simplified state machines), NancyFX (instead of all OWIN stuff), Swig (for genetrating bindings) and many others but the key fact was that I was among peers who gave me back at one moment valuable information. This is one of the things that videoconference/Skype/Meet still cannot substitute yet. They are fantastic tools for this time but unfortunately they are still unable to create the context for this kind of communication. I remember that many ideas that were later translated into patents emerged in this kind of informal gatherings and sometime a "What about..." created the spark.

I have also found an interesting article (https://theconversation.com/companies-are-trying-to-connect-remote-workers-with-virtual-water-coolers-but-its-harder-than-it-sounds-146505) that reaches almost the same conclusions - apps cannot really mimmick spontaneity and generate context for innovation.

Quiet year

 I haven't said much lately on the blog nor on LinkedIn. It was quiet and dull-ish in day to day.
The year started pretty bad, my father-in-law died in January. In July my daughter had her exams for Gymnasium. 

Basically those two moments are what I can remmember from 2021 so far. 

Shards of  events:

1. Always backup SSH keys.

2. At my previous job they are still not able to impement some cloud strategy as I forecasted. People started to flee from there

3. I slacked with all my side projects

4. Lost the will of going out - something that was a pleasure slowly turned into a burden. Also lost most of the social skills.

5. Minor but reccurent health issues. No Corona though

6. Got the COVID-19 shot

7. Hackintoshed my Huawei - works decently well with Intel WiFi (10x itlwm)

8. Pleasant working experience in Fedora 34. 

9. Rust langusge is getting better and better

10. Svelte is pretty nice although it has quirks

11. Oracle seems to focus om Micronaut - probably a second bet that I won.