The essence of architectural work - Part 7

The previous post concluded the discussion of the What, the essential activities of architectural work using the 4E framework as guidance.

Leaving out the How

As I wrote in the first post of this series, I will not discuss the How of architectural work. While being a very interesting topic (and probably a topic you are also interested in), it is a huge topic. E.g., just think about collaborative modeling. It is one way to understand the business domain ¹. If we map this to the 4E framework, we discussed in the prior two posts, it covers part of the Examine section of the framework. And there are many other ways to understand the business domain besides collaborative modeling (not rating their relative effectiveness). But we find roughly two dozen methods for collaborative modeling alone, some of them covering whole books.

Just looking at this example, it becomes clear that it is impossible to cover the How of architectural work appropriately in a blog series. Additionally, I am not sure if it is possible at all to cover the essence of How. I know that we face quite a few people with strong opinions about “right” or “wrong”. Probably, those people also have a strong opinion about what is essential about How to do architectural work. Nevertheless, I am not one of them.

Personally, I think there is a reason why this field is so huge. I think a “right” method, the essential method does not exist. Many good methods exist for all four core activities of the 4E framework and their sub-activities. It is the concrete context in which method A may be more suitable than method B. Still, that does not mean that method A is always better than method B. It is just more suitable in the given context. In a different context, method B might be the more appropriate choice.

In the presentation, this blog series is loosely based on, the How part therefore primarily consisted of a few tips and tricks that served me well in my contexts. Additionally, I included a few hints towards relevant topics that I have seen neglected too often. However, this is not anything close to “essential” How.

Therefore, we will skip the How of architectural work and move on to the When and How much of architectural work. ²

No one-size-fits-all

When it comes to When and How much, we also face many strong opinions regarding “right” or “wrong”, accompanied by many heated – and usually pointless – debates. I say “pointless” because again, a “right” or “wrong” does not exist (even if some people insist it does). There are only more or less suitable approaches regarding the When and How much of architectural work.

If we look back to the pre-agile times, architectural work was BDUF (Big Design Up-Front) most of the time. While it worked relatively well in some contexts, it did not work well in other contexts. Then the IT world went Agile, and the agile advocates said that BDUF is evil. But it did not stop there. We saw a development over time:

• First, architectural work became NDUF (No Design Up-Front) in combination with emergent architecture. People quickly realized that this led to even worse architectures than the despicable BDUF.
• Then, architectural work became JDUF (Just Enough Design Up-Front) in conjunction with the “decide at the last responsible moment” principle. This worked better than the NDUF approach. However, architectural work had become very complicated, and in some contexts, it did not feel better than BDUF.
• Meanwhile, most companies are back to the incremental-iterative methods of the 1990s, delivering 4 releases a year. Of course, they do not call it RUP (Rational Unified Process) anymore but SAFe, and releases became PIs (Product Increments). While this suits the wants of many enterprise people better, who felt threatened by the original ideas of agile, it left us with an approach to architectural work somewhere between BDUF and JDUF. Depending on the task at hand, this approach feels more or less suitable.

The key point is: We cycled through a lot of approaches towards the When and How much of architectural work in the past two decades. Still, we did not find the one-size-fits-all approach towards architectural work. None of them worked well in all situations. In some situations, they worked very well. In others, they did not.

The question is: What is the key factor that determines how good an architectural approach works regarding its When and How much?

Uncertainty

Based on what I have learned over my career, the key factor is uncertainty.

This leaves the obvious two questions:

1. What do I mean by uncertainty?
2. How does it affect the When and How much of architectural work?

Let us begin with the first question and answer the second question along the way.

VUCA

If we research uncertainty, we quickly stumble upon VUCA, an acronym for the 4 terms: volatility, uncertainty, complexity, and ambiguity. If we look for definitions of the 4 terms, we find something along the lines ³:

• Volatility: the nature and dynamics of change, and the nature and speed of change forces and change catalysts.
• Uncertainty: the lack of predictability, the prospects for surprise, and the sense of awareness and understanding of issues and events.
• Complexity: the multiplex of forces, the confounding of issues, no cause-and-effect chain, and the confusion that surrounds organizations.
• Ambiguity: the haziness of reality, the potential for misreads, and the mixed meanings of conditions; cause-and-effect confusion.

Trying to phrase it in a more comprehensible way, VUCA describes the conditions we face more and more often these days:

• Volatility: Conditions and situations can change quickly. The perceived speed of change accelerates.
• Uncertainty: Changes are less predictable. We are increasingly often surprised by events and situations.
• Complexity: We face complex webs of interaction without simple (linear) cause-and-effect relations. We cannot reliably predict the effect of an action.
• Ambiguity: The messages and signals we read do not necessarily reflect reality. We must assume misinformation (deliberate and accidental).

All four aspects of VUCA ultimately lead to decision uncertainty – when to make which decision and how to come to it.

There are always those nagging questions: Which decision is right? Is there a right decision? Am I making the right decision? When do I need to make it? Do I know enough to make the decision? What if I make the wrong decision? What if the information I have is not correct? Etcetera.

The more VUCA we face, the harder it becomes to make an architectural decision, and the more work we need to invest in gathering and verifying the required information.

Drivers of VUCA

This immediately leads to the question of the drivers of VUCA. There are several drivers of VUCA in the context relevant to us. I discussed all of them in prior posts. Thus, I keep it short here. If you are interested in more details, please read the referenced posts. The most relevant drivers of VUCA regarding architectural work (and software development in general) are:

• Geopolitical and geo-economic developments – At a very high level, we can observe a political and economic power shift. Old structures and alliances crumble, new ones emerge, sometimes at surprising speed and in surprising ways. This makes it harder to make architectural decisions. A current example is digital sovereignty, which became a very relevant topic in Europe, where I live. “Go hyperscalers” was a very reasonable recommendation until two years ago (see, e.g., my The public cloud revolution blog series). However, right now things have become more complicated, and hyperscalers are not the obvious choice anymore.
• Post-industrial markets – I wrote a lot about post-industrial markets (see, e.g., my very first blog post, a discussion how market types affect IT work, or this more holistic consideration). In its very core, a post-industrial market is defined by a surplus of supply compared to demand. This creates a highly dynamic market where those companies drive the market that can adapt to the continuously changing demands of the customers best and quickest. As customers do not know their future demands until they see an offering satisfying them, we face a lot of VUCA, which in turn makes architectural decisions more challenging.
• Digital transformation (a.k.a. digitization, digitalization) – I also wrote quite a bit about digital transformation and its effects (see, e.g., my blog post explaining the waves of digital transformation, or this more holistic consideration). Regarding architectural work, digital transformation adds an extra – usually neglected – burden because software and IT have become indispensable and thus must work reliably, and they must be easily adaptable to changed needs (see also the post-industrial markets before). Additionally, digital transformation means that IT advances into domains where IT was not present before. This means a lack of experience in how to apply IT best in such domains, increasing uncertainty.
• Complexity of context and tasks – I discussed the topic for the first time on this blog in this early post: Software development has crossed the boundary from being a complicated task to being a complex task many years ago. At least for most software development projects, this means increased complexity and uncertainty.
• Disruptive technologies – I discussed in this old blog post that it takes roughly 20 years until the effects of a disruptive technology are understood by the mainstream. This also means, the companies that understood the effects of a disruptive technology earlier than others can leverage its effects to exert pressure on those companies that did not understand the effects yet. This pressure is perceived as high dynamics, ambiguity, and complexity because it feels as if those early adopters have some magic secret weapons, and it is totally unclear how to respond to them. Additionally, the new technology in itself also causes a lot of headaches and perceived complexity because it is still unclear how to use it properly.

These are the most relevant drivers of VUCA regarding architectural work that I have identified. Most likely, there are more drivers, but those are more than enough to create a lot of VUCA and thus decision uncertainty for most companies.

The effect of uncertainty

All this leads to the next question: What is the effect of (decision) uncertainty?

I discussed this question in detail in a former blog series about uncertainty. The short version: Under uncertainty, you cannot predict anymore if your decision will lead to value creation or not. This leads to a different approach towards decision-making – or at least should lead to a different approach. Instead of making few big decisions early on, big, i.e. expensive decisions are usually split up into a series of smaller, i.e., less expensive decisions that are continuously validated using feedback from the people affected by the decision.

This allows for figuring out early, i.e., without spending too much money, if an idea is promising or not. The promising ideas are developed further, while the non-promising ones are dropped. This increases the probability of making more decisions that actually create value. It also improves the ratio of business value created per euro spent under uncertainty.

The last responsible moment

In an architectural context, we can also observe another effect: Under decision uncertainty, we try to make decisions as late as possible. The underlying reasoning is that architectural decisions are often hard to revert ⁴. Therefore, it is important to make architectural decisions as late as possible because the longer we wait, there more information we tend to have. Several weeks or months into a project, we might have learned stuff that affect our decisions that we did not know at the beginning of the project (when BDUF decisions are made).

However, we cannot delay architectural decisions eternally, as they affect software development. If we do not make an architectural decision before code is written that is affected by the decision, the developer will make that decision implicitly due to the lack of an explicit decision. This tension between making a decision under uncertainty as late as possible while not delaying it until after it is needed became known as “the principle of the last responsible moment”. A lot of literature and opinions exist on when this last responsible moment is and how to identify it. As it is part of the How, I will not dive deeper into it.

Do not delay decisions unnecessarily

However, there is another principle regarding decision-making that is often neglected. Even if the amount of decision uncertainty has increased in software development projects over the years, not all projects are VUCA, and not all decisions face high degrees of uncertainty. We will dive deeper into this aspect in the next post of this series (link will follow).

For now, we will only discuss the complementing principle. The less VUCA we face, the more straightforward decisions can be made. This leads to the – usually forgotten – complementing principle: Do not delay decisions unnecessarily. If we have enough information to decide safely, we should decide. There is no value in delaying decisions if we have enough reliable information in place to make them. Quite the opposite: We leave something undecided that could be safely decided. This means we maintain an unnecessary additional degree of freedom, i.e., added mental load for all the people involved. This leads to the following effect:

• The more decision uncertainty we face, the later we should decide.
• The less decision uncertainty we face, the earlier we should decide.

Interlude

We started this post with the observation that there is no one-size-fits-all when it comes to architectural work. We discovered uncertainty as a key factor that determines how good an architectural approach works regarding its When and How much. We then discussed VUCA, the resulting decision uncertainty, and how it affects architectural work. Finally, we discussed the widespread principle of the last responsible moment and the less often considered complementing principle of not delaying decisions unnecessarily.

In the next post (link will follow), we will apply this knowledge and align the When and How much of architectural work with the degree of uncertainty.