Open Source Software – Part 1
When I started to discuss accidental complexity on the architectural level in my “Simplify!” blog series, I quickly realized that I needed to discuss Open Source Software (OSS) first as it often acts as a driver of accidental complexity on the architectural level today.
So, I started to discuss OSS inside the post. While doing this, I quickly realized that OSS deserves a more detailed discussion than just a few paragraphs in a post about a different topic. Especially, as so often it is important to understand the past evolution of a topic – here OSS – to be able to assess today’s situation correctly.
Therefore, I started to write this post. While writing, I realized that discussing OSS would become too much for a single post 1. Thus, I split it up in 3 parts:
- The rise of OSS and its benefits (this post)
- Misconceptions regarding OSS (next post)
- OSS in the world of today (next but one post – link will follow)
This post will be about the past of OSS, its rise and the effects it had on software development.
The time before OSS
Before we dive into the misconceptions regarding OSS that drive complexity, let us first have a brief look at the evolution of OSS and how it influenced software development.
In the early times of software development, OSS did not exist. All software was the property of the company that developed it. Sharing of software beyond algorithms shared via textbooks or alike was the exception. This also meant that companies needed to implement basically everything above the level of a 3GL programming language on their own, including all required base libraries and frameworks. 2
This limited the size of problems that could be solved reasonably with software 3. Quite some effort had to be spent on implementing base functionalities instead of solving the business problem at hand. In the beginning the code ratio probably was 90% base functionalities and 10% business problem. 4
Over time the 3GL programming language providers added more and more libraries, frameworks and other base functionalities to their compilers. Some vendors – typically also supplying the underlying OS – even provided whole programming and runtime environments including powerful abstractions. E.g., think about TSO as batch scheduler or IMS and later CICS as transaction monitors by IBM, including lots of libraries, frameworks and more base functionalities up to full-flavored IDEs and runtime support.
Roughly around the same time, standard software emerged. Instead of leaving all the business to the OS vendors, independent vendors started to implement software that could be installed on a given OS. This also gave rise to a completely new type of software that did not focus on IT productivity but on addressing business topics. The probably most prominent representative of this kind of software is SAP, but also lots of other standard software products emerged.
All these developments increased the productivity of software development a lot, i.e., the ratio of time spent on solving the actual business problem compared to implementing base functionalities rose. On the other hand, all these productivity enhancers were closed source and cost a lot of money. Additionally, hardware was still so expensive that usually only companies and other organizations of a certain size like, e.g., universities or government organizations were able to afford them.
This meant, the productivity of software development was bound to what vendors provided and what your respective company was able to buy which capped the overall productivity gain.
The rise of OSS
Over time computers became affordable for regular persons. Microcomputers emerged. Sharing of digital data became easier for individuals due to the rise of floppy disks acting as a cheap data exchange medium and the Internet (not yet the WWW, but all the protocols that existed before 5).
This evolution supported the forming of like-minded communities like, e.g. the GNU project that started to (re)build software solutions with the ideal of free access and usage for everyone. Simply put, starting with this “Public Domain” ideal over time several other variants emerged, with OSS becoming sort of an umbrella term. 6
In the beginning, relatively simple software solutions were (re)created as OSS solutions, some libraries or smaller applications. But over time, the solutions – often built on top of previously created OSS solutions – became more and more powerful.
Eventually, OSS solutions became so powerful and reliable that companies started picking up OSS – usually not relying on the powerful OSS community but requiring a support contract as they were used to from commercial software. 7
This in turn created a whole industry specialized in providing OSS support contracts. Yet, only a few companies were able to successfully implement this business model, i.e., were able to live from OSS support contracts. E.g., Red Hat was able to create a sustainable business model for a long time by licensing their own Linux distribution to enterprise customers, plus optional support contracts. But most companies were not able to create such a sustainable business model.
As a consequence the business models around OSS changed over time. Today, many of the popular, widespread “OSS” solutions are actually commercial solutions. Additionally, the companies behind the solutions also offer an OSS edition of the product, typically lacking essential enterprise features like, e.g., monitoring, HA capabilities or integrations with other widespread (commercial) products.
While it would be possible to use the OSS version of the software and add the missing enterprise features on your own, very few companies choose this path. Instead, they usually buy the commercial variant of the solution – which conveniently usually also includes a support contract option. The idea of the OSS edition in these cases primarily is to attract software engineers with the goal to eventually sell the commercial version of the product. 8
But also many other OSS solutions exist that are not connected to commercial offerings and do not incur license or subscription fees if used in an enterprise context. These solutions are backed in various ways, ranging from a single or a few idealistic individuals who work on the solution in their spare time to companies who pay some of their employees to work on an OSS solution (their motivations again ranging from quite idealistic to quite selfish – in the latter case usually being heavy users of the solution themselves).
Today, you find an OSS variant of almost everything imaginable on a technical level, often in very high quality and maturity. Also quite some business-related offerings are available as OSS. But the coverage is by far not as high as on the technical side. 9
Many companies implemented an “OSS first” strategy in the last 10 years. Yet, the reasons are usually not rooted in a deep conviction on the decision maker level regarding OSS. It rather is a mixture of developers preferring OSS solutions because they have better access to it and trust it more, and decision makers hoping for risk-free license cost reduction. Also, project managers and architects tend to like OSS as it often spares them tedious discussions about license costs. 10
Effects of OSS
The effects of OSS on software development cannot be overestimated. OSS was sort of a democratization wave regarding software development. As all democratization waves this had a lot, primarily positive effects. E.g.:
- Many more developers had access to programming languages, tooling, libraries and more. This increased the number of qualified software developers a lot. 11
- As a developer you were no longer dependent on the software your company was willing to buy. You had access to a lot of software building blocks, often in high quality.
- The productivity of developers increased a lot. They did not have to build all the functionalities from scratch anymore. They were able to start at a lot higher abstraction level.
- OSS boosted competition. Commercial vendors could no longer rest on their traditional monetization models selling relatively low-level software solutions at high prices (and higher-level solutions at extremely high prices). Low-level software solutions became available for free which urged the commercial vendors into higher-level segments, also making it harder to realize very high prices for their solutions.
- The whole software ecosystem evolution was accelerated. Building on lower-level OSS solutions, higher-level solutions were created up to complete application servers and more. The widespread availability of high quality OSS solutions acted as an accelerator for the whole industry.
Probably I missed some relevant effects, but I think the ones I listed already make clear, how much OSS influenced the whole software development domain. Especially in the early 2000s when companies started to pick up OSS at an increased rate, OSS created a huge productivity boost.
In this post we discussed the origins and rise of OSS. We can state that OSS was a tremendous success stories that brought a lot of positive effects. Especially in the period between approx. 1990 and 2010 it changed software development dramatically in many ways and shaped a whole generation of developers.
As so often, also some misconceptions sneaked in. But that is the story of the next post. So, stay tuned …
I know, that is a lot of realizations. To be honest, I am surprised myself often how things turn out. When I start writing a post, I usually know the topic and have a rough idea about the contents. But while writing, a lot of additional aspects come to my mind that initially did not have on my list. Also while writing I sometimes realize that a reasoning does not work and I need to change it. And so on. So, writing a blog post is also a discovery and learning process for me (and part of why I am doing it). ↩︎
There were times when even 3GL programming languages were not available. Yet, for this post it does not add any additional insights if we go back in time that far. ↩︎
Of course, very limited compute power also limited the size and scope of problems that could be addressed with software. Still, having in mind that people were able to land on the moon with computer hardware less powerful than a 3 Euro pocket calculator today, this probably was not the most limiting factor. ↩︎
Side note: This was also the time when computer science started to become obsessed with reusability. If you do not need to implement the base functionalities from scratch each time but can reuse parts from prior projects, it can become a huge productivity boost. While this made perfect sense for the situation back then, today we face a very different situation where most things worth being made reusable already exist. Yet, the obsession still persists – usually for the wrong reasons. ↩︎
Y’know, all that weird stuff like, e.g., BBS, Telnet, FTP or Gopher the elders talk about if they are in a nostalgic mood … ;) ↩︎
I know that this description is an oversimplification of what actually happened and I am sure that some members of the tribes that formed around OSS, Public Domain, Shareware, Freeware, etc. will heavily contradict. Yet, for the sake of this post, being more accurate would not add any additional value. That is why I decided to go with this oversimplification. ↩︎
The maybe most interesting aspect of insisting in support contracts is the related message, companies send to their software engineers: “We are convinced that you are not capable of fixing a bug in the OSS software you use if one should occur”. Even if nobody says it aloud, the message is omnipresent and I can only guess how much damage this message causes, psychologically but also in terms of money. If you treat people as if they were incapable, over time they will act as if they are incapable. The economic problem of such a setting: capable people create value, incapable people do not. It does not make a difference if the people are actually incapable or if they are just treated as if they were. Thus, as a company you should be really careful if you treat your engineers as if they were incapable because you will destroy lots of potential business value by doing so. ↩︎
To be clear: I think that is a perfectly valid business model and usually a lot more viable than the pure support contract variant. It just puts the widespread misconception “OSS is for free” into perspective. I will discuss this in more detail in the next post. ↩︎
Probably, one of the reasons for this is that new OSS developments typically start as pet projects in the spare time of one or a few software engineers. People usually only invest spare time in topics they are actually interested in. As most software engineers prefer technology over business functionality, I guess this could explain why we see more technical OSS solutions than business OSS solutions. ↩︎
Interestingly enough, in quite some companies it is easier to get 100.000 Euros of extra development costs signed off than 10.000 Euros for buying a product that would solve the exact problem at a tenth of the cost. It does not make any sense, but often this is enterprise reality. ↩︎
Of course, it also created some software developers of disputable qualification but that is not the fault of OSS. ↩︎