Technology Grafting

Created on 2021-12-23 09:48

Published on 2021-12-23 17:01

Wikipedia defines grafting as: "a horticultural technique whereby tissues of plants are joined to continue their growth together". Grafting is often used for producing new varieties of fruits on older trunks that are more resilient or adapted to a climate.

When working with legacy software we can see modernization as grafting. The rootstock is the older technology - proven and battle-tested while the scion is a newer technology that will enhance the rootstock's qualities. The technological grafting needs two things: interfaces/protocols and encapsulation.

The interface layer ensures that the data can be understood by both scion and rootstock. If the rootstock uses some very specific encodings (EBCDIC, various other locale-specific charsets) then the translation between them must be consistent in both directions. Things seem to be easy with textual protocols (although there might be also some issues) than with binary proprietary protocols. If a fast bidirectional translation is not possible (very different protocols) then a middleware might be necessary but in this case, it would be best that this middleware

On the other hand, encapsulation permits the scion to maintain all its properties and continue working as if it would be still attached to its original trunk. When grafting a pear branch on an apple tree the pear branch will continue to function as pear and will not hinder the apple trunk. The same behavior should be present when grafting software components. However, this strong encapsulation requires the open interfaces discussed before.

One of the most interesting technological graftings I have seen lately is in the form of modernizing older web applications using micro frontends packed as web components. The idea is not a novel one, web components are one of the ways of composing micro frontends, but this has massive implications for all the functional and nonfunctional aspects of the application. Imagine old PHP or Struts code bases getting an instant makeup with nice web components that reduce the server load and break monolithic applications into simpler parts. The web components isolate the new developments from the old code base and will permit the evolution of the older code base.

Grafting techniques and patterns might prove very useful in all kinds of modernization scenarios as there are a lot of use cases for this. The use cases can range from, as I wrote above, UI modernization to more interesting ones like adapting 50 years old systems to web or IoT. If the results of grafting are successful then this might help the rootstock also evolve (like Autocad's migration to the web). Grafting permits the progressive evolution of the software without disruptions in base technologies used at a minimal extra cost. The complicated problem is to determine where is the best place to insert the scion and how to make it behave on the same security and functional constraints as the rest of the application.

Necessary formalism

Created on 2021-12-19 14:39

Published on 2021-12-19 16:02

Most of us have started with a small application that solved a business need. Then we had some success with it therefore we added functionalities, improved UI, made it multi tenant and so forth. In the meanwhile the code has grown and every "if" condition we added made it more complex.

We added "ifs" because we had to respond to changes created by customer requests, changing laws and policies, access to different markets. Some of us were smart and resorted early to design patterns but some of us continued with the old way of adding ifs... In the end application is huge and hard to debug because it's codebase no longer reflects it's initial purpose. There is no "isomorphism" between the problem that application solved and the implementation of the application. Even applications that were well engineered have this issue as the changes they accumulated over time are not described in the original universe where the application was spawned, in the business part, but rather the engineering created approximations in the implementation domain that are maintained. To make a forced comparison the situation is similar to getting the image of a forest in autumn simply by creating a huge palette of colours without thinking of what we are trying to represent with them.

So moving along this line, how do we keep applications from going adrift? How do we keep in sync the reality where applications emerged with their implemented behaviour? Here formalism appears. Describing application behaviour in a formal way, close to what people understand is the key. This formalism have many names ranging from "Domain Driven Design" to DSLs but all focus on capturing the domain information on a higher level, not directly in code. Code is generally a byproduct of those, formal descriptions are needed for any DSL and low-code solution because, quoting Markus Volter: "models are semantically rich for their domains" whilst code is not semantically rich, just behaviouraly rich.

One of the challenges I am thinking about is: how do we retrofit formalism into application so that we make them more maintainable? There is no definitive answer but especially for legacy application refactoring to models is a key. We should start small, refactor user interface behaviour for example to state charts (in form of Redux or XState). In the meanwhile the refactored interactions could be again reorganized both server and client side into some business process descriptions (it would be ideal to use a dedicated DSL for those in this stage - but this is generally hard, so refactoring to state charts is a good beginning). Iterating over this a couple of time we will reach better models on the domain and clearer actions defined on the set of the models - hence we get a crude formal definition of the domain. At this point the process, described probably in a specific language can be understood again by business people therefore they can evolve it while it is still generating lower level formalisms (as the statecharts) that developers or specific tools can generate code from.

The new developments nowadays have the chance of starting clean as the current tooling is quite advanced (Jetbrains MPS, Eclipse Sirius). New projects can start from DSLs or from BPMN visual representations and generate rich models that can be visually manipulated in low code solutions. Moreover a repository of well described models and processes can be shared inside a company so people can reuse the high level abstractions, manipulate them in their own code and getting interoperability and common behaviour for free. Also testing is positively impacted as tests clarity increases because models carry more semantics inside. What would be easier to test: 500 lines of written code or 12-15 lines of DSL or visual representation? Where are the errors simpler to catch?

Well crafted formal definitions reduce the implementation and design burden for the implementation team as they shunt most of the repetitive design and boilerplate tasks in an application (rich models already contain the logic that otherwise has to be expressed and maintained in code). Continuous effort of expressing behaviours in high level, formal languages and refactoring older code towards the same seems a way of increasing maintainability and quality. DSLs, modeling and low-code are in fact facets of the same base problem - how to capture and communicate succinctly, correctly and in a non ambiguous way domain knowledge both for people and for machines.

PS: There are better approximations of a process definitions than Amazon's StepFunctions or BPMN. To quote a friend - they are like GoTo programming and do not incorporate transactional aspects.

Communication and innovation

Created on 2021-07-29 15:23

Published on 2021-07-29 15:55

More than a year ago we were talking about backups. My solution then was something based on duplicity and rclone and I was explaining it to one of my colleagues who had another approach. I was pretty happy with it although it had some quirks. Then one of the other colleagues asked us: "Have you tried restic?". I did not know about it and I found it quite interesting. Months passed and I had to implement again an off-site backup system. Although I had the old tools in the pocket I remmembered restic and I soon realised that is the perfect fit for my incremental backup task.

I imagine that I would have lost probably some days trying to use other tools and craft an adapter layer on top of the old tools. Listening to others brang me pretty much advantage both in terms of efficiency as well as in terms of knowledge - another tool on my belt.

One thing that I miss while working remotely is the posibility of sharing knowledge in an informal way. When enginers discuss around the coffee machine some crazy geekish ideas (from Raspberry Pi autonomous lawnmowers to a new programming language) or they lucidly dream some interesting features in their code then innovation is about to happen. Many things that are in an incumbent state are hard to find in mainstream publications and blogs but sometimes there is one who has a passion about "obscure-ish" (eg. https://qconsf.com/system/files/presentation-slides/control_theory_in_container_orchestration.pdf) subjects and a wise remark at the right moment can cut months of development because that insight solves in a creative way an otherwise hard task.

I had my share of these happenings: timer_fd (simplified state machines), NancyFX (instead of all OWIN stuff), Swig (for genetrating bindings) and many others but the key fact was that I was among peers who gave me back at one moment valuable information. This is one of the things that videoconference/Skype/Meet still cannot substitute yet. They are fantastic tools for this time but unfortunately they are still unable to create the context for this kind of communication. I remember that many ideas that were later translated into patents emerged in this kind of informal gatherings and sometime a "What about..." created the spark.

I have also found an interesting article (https://theconversation.com/companies-are-trying-to-connect-remote-workers-with-virtual-water-coolers-but-its-harder-than-it-sounds-146505) that reaches almost the same conclusions - apps cannot really mimmick spontaneity and generate context for innovation.

Quiet year

 I haven't said much lately on the blog nor on LinkedIn. It was quiet and dull-ish in day to day.
The year started pretty bad, my father-in-law died in January. In July my daughter had her exams for Gymnasium. 

Basically those two moments are what I can remmember from 2021 so far. 

Shards of  events:

1. Always backup SSH keys.

2. At my previous job they are still not able to impement some cloud strategy as I forecasted. People started to flee from there

3. I slacked with all my side projects

4. Lost the will of going out - something that was a pleasure slowly turned into a burden. Also lost most of the social skills.

5. Minor but reccurent health issues. No Corona though

6. Got the COVID-19 shot

7. Hackintoshed my Huawei - works decently well with Intel WiFi (10x itlwm)

8. Pleasant working experience in Fedora 34. 

9. Rust langusge is getting better and better

10. Svelte is pretty nice although it has quirks

11. Oracle seems to focus om Micronaut - probably a second bet that I won. 

Prepping

Created on 2021-03-12 21:41

Published on 2021-03-17 17:17

"Anything that can go wrong will go wrong" (Murphy's Law)

"Preppers" are persons that spend a lot of time preparing for an imminent disaster. In case of doomsday, they might survive it as they planned minutely for any possible outcome long before. Preppers are stocking equipment, clothes, food and often create shelters so that they could face even the most atrocious situations with relative calm. Prepping puts some financial strain on people but not as much one might think as preppers' most important assets are knowledge and improvisation. Regardless how elaborated the disaster recovery plan is, it's never perfect so creativity should always be welcomed.

Preppers use some techniques that IT should learn from:

1. Rehearsal - preppers really exercise their plan. They try surviving wilderness, with their bushcraft skills, thus testing their theories and equipment. In IT from time to time it's worthy simulating a nasty situation in the infrastructure. It doesn't need to be a complicated one that would imply the whole SimianArmy but at least simulate a disaster, solve it and take notes. Ask colleagues to stop a service or make a ethernet loop in a switch. Learn where to look for clues and when the incident is solved try to automate the solution. There are organizations that take rehearsals very seriously. I've been part of probably three fire simulation drills at Adastral Park in less than a month but I can say that It was the only occasion where I've seen so much calm and organization as people knew their drill.
2. Share information - preppers are sometime organized in clubs and they share lots of relevant information on materials and techniques. Community driven DRP might be interesting as people come with different viewpoints that might reveal flaws or provide better solutions. Information sharing might be the most important source of learning.
3. Use cheap and easy to find materials. For example many things can be simulated on scaled down versions of the real systems. A full K8S/EKS cluster can be simulated with a petty RaspberryPi running K3S. Most of the things will be the same but costs will be negligible. The differences can be documented and there are also lots of third parties that can simulate other services cheaply (Lambdas, Object Storage). In some situation one will have to improvise massively to recover from a disaster and then knowing what to do with almost off-the-shelve components becomes invaluable.
4. Invest in understandable systems - preppers try to understand nature and mechanisms so that they increase their survival odds. Although automation is a must it should be transparent to the engineers and clearly present what's behind the scenes. If the state is extremely bad and automation magic stopped working then understanding the system might be handy at least to go back in a restorable state with minimum effort. I for one was in a similar situation when I discovered that the backup I made had to be restored on a disk with different physical characteristics. Knowing what was the backup process I was able to hack the restore on the new drive although the tool was not supporting it.
5. Keep stashes - preppers often have hidden stashes of food and tools around the house. Engineers should also make stashes of resources if possible offsite or in different clouds. If something goes horribly wrong at least some things could be partially running. An OCI solution, replicated on a different vendor's platform, would make a lot of sense so if a provider is unavailable restoring the solution on another would not mean an entire rewrite but rather a reconfiguration. Vendor agnostic IAC solutions and CNI packed services are handy in this case - they can be easily run on different environments.
6. Have spare batteries - preppers don' t rely on a single centralised solution – e.g. electricity; they always have alternate independent energy sources that they can use - batteries, generators, etc. In the light of the recent OVH fire let's imagine the following scenario: the company's automation solution (Chef, Puppet, Octopus) is also affected somehow then the situation will be similar to electricity or transport failure. On the other hand, having the IAC implemented through standalone tools (Terraform, Ansible) increase the resilience as there is no longer a single point of failure, any engineer could replicate the infrastructure from his/her own machine as long as one can connect to the datacenter/cloud.
7. Learn how to use and build tools - preppers spend time in learning how to build bows, arrows, tents and also how to use them. Off the shelf tools are fine (backup/restore, IAC, etc) but sometime one needs to do specific operations (e.g. recover data from an ZFS formatted drive). Then building small and specific tools would be invaluable so a programming language that permits rapid development is handy - currently I see that people are looking towards Python and Go.

What I am trying to say is that pessimism and preparation are key for survivability in case of a disaster. Even a digital one.