Buckets and Sovereignity

Created on 2026-01-31 07:12

Published on 2026-01-31 07:26

S3 buckets have a problem. A developer needs a bucket, a single bucket. Something that'll take AWS about 30 seconds to spin up. However they have to open a Jira ticket. Then they ping you on Slack. Then they show up at your desk because the ticket's been sitting there for three days and their sprint is ending.

Then, to your chagrin, open the huge Terraform repo, add their bucket, run terraform plan, watch myriads of resources scroll by, get it reviewed, wait for the pipeline, and boom. One S3 bucket. Only took a week and three senior engineers.

This is stupid. Very stupid. But we keep doing it because we've convinced ourselves that "infrastructure as code" means all infrastructure must live in the same repository, managed by the same team.

Is it true? An S3 bucket that only exists to serve one application isn't infrastructure. It's part of the application.

Infrastructure is: VPC, EKS cluster, networks, etc. A bucket that stores upload images for the marketing website? That's application state that happens to live in AWS instead of Kubernetes. That could have been another API call to a object storage service. Managing application resources as infrastructure with an infrastructure tool is not friendly at all.

Enter AWS Controllers for Kubernetes.

ACK does something wonderfully simple: it turns AWS resources into Kubernetes resources. Want a bucket? Write a manifest. The ACK controllers watch these manifests and call the AWS APIs. The developers get self-service. You get to stop being the S3 bucket vending machine. There is no giving up control, you're shifting what you control. Th control increases in fact as the cluster will try to keep resources under control continuously, not only when terraform plan/apply happen. Instead of gatekeeping every single resource creation, you define the rules.

Another team needs a Postgres database? They drop this in their repo:

They commit it. The pipeline runs. The database appears. Your policies ensure it's in the right VPC, encrypted, backed up, tagged correctly. Tags are applied for correct cost allocation so the financial minded persons have a nice cost breakdown.

This is what platform engineering actually means. You build the platform, the foundation, the policies, the standards. Developers build on top of it. Both teams do what they're good at.

And when they delete that feature branch? The database goes away. No orphaned resources quietly racking up charges for six months because everyone assumed someone else would delete it. Everything's in Git, versioned with the application code. Developers can see infrastructure status with kubectl get dbinstance. The blast radius of any change is limited to that team's namespace.

The plot twist: digital sovereignty.

In the last year cloud shifted from pure convenience to strategic threat. European regulators are increasingly uncomfortable with critical data for European companies subject to American surveillance laws. GDPR was just the opening act. The Digital Services Act, the Data Governance Act, these are forcing real architectural decisions.

And here's the uncomfortable truth about ACK: it makes you really, really good at AWS. Every pattern, every practice, every custom resource is AWS-specific. Which is fine for US but not so much for Europe.

Enter Crossplane who does something clever. Instead of direct AWS API bindings, it gives you an abstraction layer. You define what a "Database" means for your organization, and Crossplane figures out whether that's RDS, or Cloud SQL, or, interestingly, a database on OVHcloud, Scaleway, or Open Telekom Cloud. They're European cloud providers, subject to European law, outside the reach of the CLOUD Act. For organizations handling European citizen data, running European critical infrastructure. This actually matters.

The developer experience stays the same. They still request a "Database." Your platform team just swaps what that provisions underneath. Today it's AWS. Tomorrow it's a sovereign provider. The workflow doesn't change.

Maybe the organization doesn't care about digital sovereignty. Maybe one is fine being all-in on AWS forever. That's legitimate. But the organizations that do care: regulated industries, government contracts, ones thinking five years ahead, are realizing that flexibility isn't just nice to have.

Because the geopolitical landscape that made AWS the obvious choice in 2015 might not be the landscape of 2030. ACK solves the S3 bucket problem beautifully. But Crossplane solves it and gives you an exit strategy. In a world where "which country's laws apply to this data" is becoming as important as "how many nines of uptime do we get," that's worth thinking about.

Red Aurora

The snow outside was the same shade of pale grey as the concrete blocks that lined Oslo’s People’s Sector Seven. Inside the Productivity Hub, Comrade Morten stared at the dim glow of his workstation. 

A red icon pulsed in the corner: “AI Assistance Required by Order of the Central Efficiency Committee.” Morten sighed. He’d been a woodworker before the Great Automation Integration, before the Party had decreed that every citizen must harness state-approved AI to achieve optimal output. Now, every design he made was “enhanced” by Algorithm 14-K — smoothing lines, optimizing cuts, trimming away his individuality. 

“Comrade,” came a voice from behind. Morten turned to see Comrade Bjørn, draped in the standard-issue fur-trimmed coat, his breath steaming in the cold air of the underheated hall. “You’re falling behind quota,” Bjørn said, his voice both concerned and sharp. “The Party notices such things.” “I am meeting my numbers,” Morten protested. “Mostly.” Bjørn’s eyes narrowed. “Mostly is not enough. Have you forgotten Alexei Stakhanov? The coal miner who extracted fourteen times his quota in a single shift? The Party remembers him, even now. He is the model of the New Working Class Hero — not because he was forced, but because he believed.” Morten tapped the AI prompt reluctantly. Blueprints blossomed instantly on his screen — impossibly precise, impossibly fast. “The AI designs everything now,” he muttered. “What’s left for me to believe in?” 

Bjørn stepped closer, lowering his voice. “You believe in the result. Every chair you make, every beam you cut — they build the communal future. It doesn’t matter if your hands or the algorithm’s hands shape them. We are one machine, Comrade. You, me, and the AI — all tools of the Party.” Through the frosted window, the red aurora shimmered in the polar sky, cast by the orbital solar reflectors. It painted the snow crimson, as if the whole land bled for the collective. 

Morten looked at the glowing blueprints. Somewhere deep inside, a stubborn human pride whispered that he could do better without the machine. But another part of him — the part that wanted to survive the coming inspection — began to work faster. He pressed the Accept button. In the background, Bjørn’s voice echoed softly, almost like a hymn: “Production is devotion, Comrade. And devotion is forever.”

Recovering TP-LINK router

 My WDR430 router has been bricked for a while.

Initially I tied to reflash it using a SOIC clip but alas, that proved impossible as I had no level shifters for 1.8 V flash. Therefore I soldered an UART interface and tried to flash it via serial. That wasn't as easy as I was never able to pres the U-Boot interupt sequence `tpl` fast enough. 
The solution was to write an expect script and use that - worked perfectly :)
 

<

#!/usr/bin/expect -f

#!/usr/bin/expect -f

# Show all output
log_user 1

# Start cu
spawn cu -l /dev/ttyUSB0 -s 115200

# Increase timeout
set timeout 180

# Wait for autoboot
expect {
    -re "Autobooting.*1 seconds" {
        send "tpl\r"
        sleep 1
    }
    timeout {
        puts "Timeout waiting for autoboot message."
        exit 1
    }
}

# Send tftpboot
send "tftpboot\r"

# Wait for done after tftpboot
expect {
    -re ".*done.*" {
        send "erase\r"
    }
    timeout {
        puts "Timeout waiting for 'done' after tftpboot."
        exit 1
    }
}

# Wait for done after erase
expect {
    -re ".*done.*" {
        send "cp.b\r"
    }
    timeout {
        puts "Timeout waiting for 'done' after erase."
        exit 1
    }
}

# Wait for done after cp.b
expect {
    -re ".*done.*" {
        puts "All steps completed."
        interact
    }
    timeout {
        puts "Timeout waiting for 'done' after cp.b."
        exit 1
    }
}

The marshmallow experiment

Created on 2025-05-18 07:08

Published on 2025-05-18 08:17

Somewhere in the 70s psychologists at Stanford University made an experiment about "delayed gratification". Children were offered marsh mallows that they could eat immediately or, if they delayed it for 15 minutes, they would also get some extra treat as a bonus. Following up the results they found a correlation between those that were able to delay gratification and their later results in life being better in comparison to those who were not able to do it.

On the other hand we are pushed towards immediate gratifications by a lot of factors that our society is based. We are sold the idea that everything should be "Bigger, better, faster, more" and we need to "want it all, and want it now". If things are fast we pat ourselves on the back considering ourselves "efficient". Is it true? Are the results similar to the Marshmallow experiment?

https://ownmygrowth.com/2020/09/27/instant-gratification/

The current trend for boosting efficiency is the use of AI everywhere. The most relevant trend is now the "vibe coding" - fast results with minimal effort. Yes, results are impressive, the assistants are able to produce code, sometimes better code than the humans in a fraction of the time as they have indexed HUGE amounts of ALREADY WRITTEN CODE. Hence a task that used to take days could be theoretically be finished in hours if not even in minutes. That is great and saved a lot of time.

But the question is what we do with that time? How do we spend it? Do we take just another task from the Jira board and throw it to the army of assistants asking the correct prompts or prompting an assistant to prompt another assistant to do it? Or we try to "grok" it ourselves? Do we invest the gained time in something creative and new? In my opinion creativity doesn't boost in the same way, contrariwise, creativity comes generally from scarcity not from abundance. Having everything served immediately reduces the need for searching and understanding the solutions for the problem at hand. As a recent Microsoft study revealed we are just trusting the response as it seems plausible and after a while we take it for granted as being correct without double checking it.

Doing something good as a human requires effort, delayed gratification. Try learning a new instrument. It will take a long time to produce something that resembles music, especially for those that at their first musical experience. I was, and I still am, an horrible player, practiced for years until I was able to play something that sounded good (depending to whom you ask). But I enjoyed the trip. Before playing that solo i had to learn not only the instrument, chords, and finger-picking but I learnt about bands, trends , music in general and, probably the most important thing, I got to know people with similar interests. Putting effort into doing something is the key to progress as Derek Muller explains it.

The lack of delayed gratification leads to stagnation, lack of desire to do something, no imagination for what might be the end of the trip. A general state of INFANTILIZATION might appear as the skills needed to obtain something will be underdeveloped and tantrums will be satisfied unconditionally. The issue is that also decision makers will be affected by the same symptom as all their goals will apparently be satisfied instantly with no rebuttals.

AI is a wonderful tool, I am all in for that. The issue is not AI, as it is not sentient, it is us in our infantile way of using it to get immediate gratification instead using it wisely - try first, get feedback or advice, try again cycle. If we are more efficient day after day, we should take some time to understand it. How embedding works, what is quantization, what risks MCP poses. Try replicate and understand the mathematics behind it, go back to the Algebra and Calculus one neglected earlier in life, verify the responses by reading the literature suggested as source (if it really exist) or try learning to draw something with pen and paper just to understand the techniques after asking the AI to do it as an example. Try to delay the gratification and dream of some better reward. Most of the time it worth it.

To finish in a darker note:

https://www.orau.org/health-physics-museum/collection/radioactive-quack-cures/pills-potions-and-other-miscellany/vita-radium-suppositories.html

Somewhere in the '30s Radium suppositories were seen as a great thing, as Radium was something novel at that time as a cure to slowness as their advertising was reading. Not understanding something completely and using it JUST BECAUSE IT WAS ADVERTISED is dangerous at least. Short term gratification cycle is just followed once again.

Weak Discouraged Men! Now Bubble Over with Joyous Vitality Through the Use of Glands and Radium

“If YOU are showing signs of “slowing up” in your actions and duties, perhaps long before you should—if you have begun to lose your charm, your personality, your normal manly vigor—certainly you want to stage a “comeback.” The man who has lost these precious attributes of youth knows how to appreciate their value. He realizes that happiness depends on his ability to perform the duties of a REAL MAN. Sweet, glorious pleasures of life. Nature intended that you should enjoy them.”

“Now is the time to act! Today! RIGHT NOW! Tomorrow may never come.”

Links:

1. https://www.lifehack.org/353923/instant-gratification-short-lived-you-should-aim-for-long-term-goals

2. https://medium.com/@darinleavitt/the-epidemic-of-instant-gratification-aef2a8d38903

My response to "From Mere Engineer to True Artist"

Created on 2024-04-11 15:28

Published on 2024-04-11 18:58

Codecamp Timisoara took place today, and it was quite a nice conference. The lineup was top-notch, and the topics were quite diverse. There were also some talks that seemed overly specialized, possibly requested by sponsors, but overall, it was quite an entertaining edition.

James Coplien delivered an interesting talk called "From Mere Engineer to True Artist" at the end. Coplien is known for stating unpleasant truths, and this is okay; it provides food for thought.

He pointed out many elephants in the room somewhat bluntly, but radical candor is necessary to shake up the industry. However, some of his assertions today were not entirely true, as there are many nuances that I struggle with.

• What are architects? Everybody is an architect; there is nothing to argue about here. At a certain scale, architecture is present in every detail of the field; every commit is an architectural decision. However, the title "architect" carries more implications. The term "architect" originates from the Greek ἀρχιτέκτων (archi - first/over, techton - builder/mason). Architects were defined by Vitruvius as mediators, experienced workers able to mitigate stakeholders' needs. Architects have scars from previous battles and failed buildings, which gives them another "unnamed quality" of being able to steer and communicate to avoid repeating the same mistakes.

• The lack of importance of a "4-year CS degree" and the notion that "CS is not a science/engineering." I completely disagree. In my view, CS is a science, residing somewhere between a branch of mathematics, a cornerstone of electrical engineering, a niche of quantum physics, and a domain of philosophy, yet with some fundamental ideas of its own. Of course, talent and hard work can compensate for formal education, but formal CS studies help evolve it from a craft to an engineering discipline by incorporating the feedback of countless generations of practitioners, creating something as solid as civil or chemical engineering. Maybe CS is not a true science yet, but it is at least "science in the making", gaining substance year after year. A 1-year boot camp could theoretically create good programmers but not scientists or engineers. The graduates of these boot camps could be productive, but they would mostly align with the 90% that Coplien fears will disappear, especially if their motives for choosing CS as a profession are not fueled by passion and continuous learning. He argued that kids can create software. True, they can create impressive things from a young age, but sometimes their creations are naive, functional indeed but lacking the details that a seasoned practitioner would add — details that provide an edge in terms of usability, performance, and quality. Kid's creations, even if not completely abstract are full of abstractions

Unicorn drawn by a kid

vs.

Unicorn drawn by a professional

Coplien argued that Renaissance architects who delivered value had no architectural studies; still, they created wonderful buildings. In my opinion, it was not necessary at that point in time, as during that period, the arts and crafts were mostly similar, and practitioners were genuine polymaths.

• Commissioned work: Most of the examples of great architecture he presented (Brunelleschi, Michelangelo, da Vinci) worked on commissioned buildings. The artists worked at the order of a sponsor, as artists are often poor, in need of money. There are just a few self-sustained artists, mostly in the 20th century, who were able to market themselves so that they could create at their own will. Even these exceptions had to do commissioned work at the beginning until they were able to create independently. Starchitects like Oscar Niemeyer created "commissioned works of art" - for example, the city of Brasilia. It was their genius that transformed a mundane request into an unforgettable work of art, but they did not initiate the project themselves with their own funds.

• Abstractions are bad: Again, I do not agree. Abstractions are the foundation of science and art. If we observe the evolution of Picasso's bull, we see it goes from concrete to abstract, refining the shape to its bare essence. There is nothing that one can take out from the last image and still have a bull. Abstractions create flexibility, a quality that permits evolution. If we abandon abstractions, modern art would still resemble the bucolic style of the 16th century, and software would still be in Fortran dialects. Functional programming and object-oriented programming are based on abstract concepts that allow them to express real-world problems in an understandable manner and create higher-order simplifications of the problem at hand.

• Beautiful code: With some references to Christopher Alexander's "unnamed quality," Coplien wants us to deliver beauty. The problem lies in how he characterizes beauty. His aesthetic criteria are vague. Beautiful code is not only visually appealing; what if we cannot see? Perhaps one has a test-sensing organ that is sensitive to testable code and code with a lot of tests - a testable flower and its petals. We all strive to write beautiful code, even in suboptimal languages, but let's be honest - we do what we can to meet our deadlines. Maybe we write truly beautiful code once or twice in our careers, but often, we write code that brings value. Value can be delivered not only by ethereal, elegantly crafted code but also by solid, practical code. While he advocates for the axiom of delivering value first, his subsequent request for beauty is in stark contradiction.

Delivery pipelines

Created on 2023-08-11 09:14

Published on 2023-08-11 10:18

The industry is buzzing currently with term as Continuous Delivery, Continuous Deployment an Pipelines. Pipelines are techniques to seed up the processing time by sequentially running a set of tasks on specialized stages, at every moment of time one step consumes the output of the previous one. The concept is not new, it has been started in the early 1900 when it was applied to factories and assembly lines. But the current pace of software development just put a lot of spotlight on them.

The software pipeline most often do a set of operations as check-out code from VCS, compile/build it, package it, run tests on it, deploy the artifacts to a repository and inform the stakeholders of the result, update various metrics and dashboards. These actions require a script to put things in the right order and check the results in every step.

In software the pipelines are generally defined through a specific language, either a visual one or a text based one - generally YAML or Groovy. These are run by specialized software as Jenkins, GitHub Actions, CircleCI, etc. The high level pipeline described in YAML is calling various other scripts that define the individual steps - like micro-operations.

The build step for example is the invocation of a Makefile, shell script or a Maven build. The pipeline will wait for the completion of the build step before passing the artifacts for testing or packaging. Such a situation is interesting because there will be two points of control in the pipeline. One at pipeline level and one at step level but their responsibilities will bleed into each-other.

This might lead to some problems:

1. Inefficient pipelines: If the result of a step can be guessed before the whole step completes the pipeline should fail fast. It makes no sense to wait for the completion. Imagine that you have a multi project build. In the trivial case the projects are built and packaged. But what if one of them has an issue? Should we wait for completion of all builds and then juts cancel the packaging step? What if there are modifications on a single project that has no egress dependencies? Again - it makes no sense to package all other projects as they haven't been modified. However with the split logic this is hard to achieve as the operations on pipeline level are quite coarse-grained. On the other hand most build tools (Gradle, MSBuild, ...) are perfectly able to run more parts of the process by themselves - create archives, containers, publish artifacts - and probably they should do it. Fail fast and incremental builds are essential for accelerated release cycles. The build tool is in a better position to understand what it is building by looking in the source code than the pipeline that is merely an orchestrator of some loosely coupled processes. Caching build results and artifacts, using parallel builds could reduce build times from 20 minutes to less than a minute - so some DORA metrics will look way better and will make both developers and managers less impatient. Speculative execution and rescheduling is well known in CPU pipelines - software engineers should have some inspiration from the clever solutions that hardware folks are successfully applying for more than two decades.
2. Portability: The high level pipeline logic is hard to be moved from one type of executor to the other. Pipelines built for Jenkins will be hard to be ported to another CI/CD system. The worst part is that it won't be easy for a developer to run the pipelines locally in order to have similar results as on the server. Dev/Prod parity should be not only on the tooling versions but also on the environment where code is built. Being able to run the pipeline locally would mean that a developer can also debug it if it's the case - it creates better visibility in the whole project - enabling a DevOps culture. Having an ops/build team that is managing the pipeline in secrecy is in my own view an anti-pattern. The devs will happily throw any issue away to the build/ops as "it works on their computers" - creating some knowledge towers. Also the posibility of running the pipelines on the local machines will ultimatly reduce the load on build machines and the queuing - improving again the metrics.
3. Mutability: The environment on which the pipeline runs should be immutable in order to produce consistent results. This is well treated by GitHub actions but for Jenkins (or other on premises CI/CD solutions) or even locally on the developer machine this is slightly complicated. However this can be solved if the pipeline can run in a container which is immutable. Many IDEs today offer the possibility of a development container that provides trusted and stable environments. This is extremely important nowadays in order to mitigate supply chain attacks. The immutability of the environment would mitigate issues as those described in Ken Thompson's paper from 1984 "Reflections on Trusting the Trust". This immutability contrasts partially with the caching needed for mitigating the speed but this can be solved quite elegantly nowadays with crypto methods so no cache poisoning can be inflicted.

Containerized pipelines solve both portability and mutability issues - so the developers could have both freedom of choosing tools and rigor for their builds in the same time. I learnt about a company that creates Visual Studio customized installations and pushes them every night to the developer's machines in order to solve the issues above. This is not only inefficient (hundred of gigabytes transferred and computers never in standby) but it is also error prone as a there are a lot of machine specific issues that might interfere - so in the end there is no certitude that the configuration is identical on all computers and that there is no drift. Imagine that in a large project a single library has some different settings - it will take hours for the developer to investigate an error caused by an obscure glitch that happened overnight. Running pipelines locally in the container would enable the developers use Rider or VS Code on Windows while still being able to test and build on trusted environments and deliver Linux software. Jenkins has this feature also, but is somehow exposing it in a clumsy way despite its huge value. Contrariwise GitHub Actions make it completely transparent for the consumers of the steps - one has to look in the action's source to know that it uses containers.

Speed is a crosscutting concern and can be addressed regardless if the pipeline runs or not in a container. A slow build won't get faster if done in docker. An aggregated approach - with the right tools could improve the development experience, speed and security of an organization.

#pipeline #containers

2022 Highlights

2022 was quite interesting. I made some changes in my toolbox.

The most interesting bits were:

1. Vercel and Supabase usage that worked flawlessly. 

2. I have ditched Docker completely and moved to Rancher Desktop and Podman. 

3. Still on Fedora 36 for all my machines. 

4. Moved some personal workloads to Oracle Cloud.

5. Visted Norway

6. Toured Romania

Felt dumb most of the year. And somehow powerless - cannot really get a grab, authority wise on the development of the  project.