One important issue of software development is that problems can be resolved many ways and not all solutions are equal. Some solutions cannot be maintained because some developers cannot grasp how it works and insist on switching to something else. Some solutions cannot be deployed because the chosen providers don’t support it. Some solutions don’t adapt to the requirements of the customers or new requirements coming after the fact.
Following table summarizes the problems happening again and again in software development.
| Developers | Customers | Hosting providers |
|---|---|---|
| Limited knowledge of available solutions | Implicit requirements taken for granted | Limited set of platforms |
| Insist on using the solutions they know about and have others adapt and rewrite | Limited tolerance to errors in interpreting requirements | Too high cost for some customers |
| Narrow understanding of requirements | Limited tolerance to delays | Vendor lock in |
On one hand, developers know about some solutions and not others. They make their best at understanding requirements and designing something that works, but they cannot know everything. Usually, they are constrained by time so they prefer to go on with what they know rather than spend countless hours learning a new technology just for a new project. This is especially problematic when new developers, with different backgrounds, join a team. Each developer knows about different solutions and just cannot afford adapting to something new, so each will insist on moving everything to what he knows! This can slow down a project and even stop it altogether.
On the other hand, customers present requirements in a shallow way, taking many things for granted. When something granted is not implemented, this usually results in negative reactions making developers upset and, feel guilty and urged to fill the gap. Any additional delay is seen as a catastrophe that threatened the project, the business relationship with the customer and sometimes even the position of the affected developers. This results into more and more hacking and patching reducing the quality of the solution, causing other requirement mismatches, bugs and disappointment, for both customers and developers.
Then the provider deploying the solution imposes constraints as well. Sometimes, the platform is limited to only some technologies the developers may be even unaware of. Other times, the deployment cost is prohibitive for the customer. Some providers, especially cloud services, have specific terminology and system architectures making it hard to switch from one provider to the other without major refactoring, unless these problems are known in advance.
This article illustrates these problems using a very simple use case: a web application showing a clock to the user. There are many ways of implementing a clock and each solution has benefits and drawbacks. Choosing the wrong clock implementation is likely to require a rewrite of the whole system. While reading about this use case, think about how the complexity of real-world projects such as GMail, Facebook, Netflix, Amazon, etc., are orders of magnitude greater than a clock.
Initial requirement
Suppose we want a webpage showing an analog clock to the user. The clock needs to reflect the current time.
Static web page
The very simplest basic way of implementing the clock is using a simple Web page, represented as an HTML file, that will load an image showing the clock. This works in any browser, even the oldest. Some naïve developer may think the problem is solve. But how about showing the current time?
For this, one possibile solution is to create one image of the clock for each position and enhance the HTML page with a piece of Javascript code that would load the correct image based on current time. There will be many images, one for every minute of every hour, namely 720 images.
Some developer will go over the top, creating all these 720 clock images in a graphic program, managing to come up with a naming scheme and implement the piece of Javascript code that will pick the right image based on the current time. Then guess what? The customer will look at this and complain: he wanted three needles, not just two: one for the hours, one for the minutes, one for the seconds. So we end up with 43200 images of a clock to create! Even worse, the customer dislikes the layout, so all the initial 720 images need to be revised anyway.
Then after some thoughts and discussions, developers will agree on the need to create a program that will generate all these images. This could be done ahead of time, but why not have the HTML page call that program itself? This is perfectly doable, by replacing the static image with a web service that will return a dynamic image.
That looks simple, but there are many different ways of implementing that backend service: a CGI script (the old way), PHP (another old way), then others will come up with a proposition to implement everything from scratch using Django, then cloud developers will propose on using serverless technology such as AWS Lambda. After the programming language is determined, there are several libraries to create images.
While developers argue on what backend service to use, the customer discovers with great surprise that the clock image appears very tiny on his 4K monitor. Who thought somebody would use that page on an high resolution display? Bitmap formats such as JPEG, GIF or PNG with fixed resolution won’t do the trick. The system needs to serve a vector representation of the clock, most likely using SVG. Ah no, this doesn’t work with certain old browsers! Are we sure the customer doesn’t use such as an old browser? We can ask the customer, he will probably say no, not even sure he will understand the question, then figure out he has an old PC stuck with Windows XP and an old Internet Explorer not supporting SVG, but that will appear after the fact, after the SVG backend is in place!
We can discuss forever about the backend service to figure out that first, the customer wants to deploy on something only supporting the old PHP or, even worse, that we forgot something: the clock needs to tick with current time, not just render the current time at the time the page is loaded! While this seems intuitive, developers, overwhelmed by all technical details, may loose track of this obvious fact. A clock ticks, needles move!
Some hacker will come in, proposing to just refresh the whole page every second. This will result in something ugly that flickers. The customer may agree on this, but will of course reject the solution when he sees the flickering result!
At this point, it becomes obvious we are in a dead end. Static page won’t do it, we need something more dynamic.
Dynamic web page
There are many ways to render a clock dynamically on a web page nowadays, using Javascript code. One can write code generating SVG drawing the clock and needles, another will prefer using the HTML5 canvas (and will have to argue with the one that prefers SVG because of his background or former job), another will propose using a library such as Raphael instead of builtin SVG or HTML5, and another one may even propose using Flash even though it is deprecated, because supposedly this is easier and more intuitive, and that works with more browsers.
Then how about having the clock move? Some people will advocate in favor of directly manipulating the document model using Javascript, but that won’t work in a portable way across browsers. Most developers thinking about this solution will immediately shift his mind to JQuery, which is a widespread library allowing to manipulate a web page dynamically in a way portable across browsers. Then a developer with background in data processing will propose using D3 library to make the needles move. JQuery and D3 works in very different ways, they implement different paradigms. Switching from one to the other is not so obvious, but the D3 developer expects the seasoned JQuery programmer to use D3, and the JQuery developer would like to get the D3 guy on JQuery.
Then seasoned frontend developers or people that want to become one will go with the heavy lifting, proposing using frameworks such as Angular JS, React, VueJS, etc. Yeah, just to render a clock! Such frameworks will do the job, at the expense of a lot of complexity, but they can do much much more. A React-based page showing a clock could allow customizing its style on the fly, and the modified style would apply immediately, without flickering. If one decides to add multiple clocks at different time zones on the same page, that will be possible without breaking everything.
However, no developers know all frameworks. A newcomer will likely know something different and then start advocating in favor of switching to what he knows. « I don’t know much about React, but that seems easier to do with VueJS. » The other will prefer Ember, another one AngularJS, etc. All of these frameworks will bring development to a stop if each developer wants to use his own framework.
« Modern » web application
Maybe I should not write « modern », because at the time this article will be read, this may not be modern anymore. In the previous section about dynamic web page, what happened with the backend? Well, it is not needed anymore. Everything can happen on the client side. Now with Javascript and HTML5, web pages can do much more work than before.
One reason a backend may be necessary is persistence. Suppose we want to allow customize the clock and save the settings to a user profile that would be reused on different devices. This simple requirement would make the project a lot more complex, requiring persistent storage (thus a database) and some kind of identification system to know which profile to load. For clock settings, authentication is probably overkill, but it is a necessity for most real-world applications, and that is a huge can of worms.
Things become complex quickly, making it hard for developers to do it right. There are (too) many different solutions. One may prefer to write the backend service using Java, and Java offers tons of frameworks such as Spring MVC, Jersey, etc., for this. Another one will use Python with Django, Tornado, Flask or something else. Yet another one will lean towards NodeJS with Express or some other library. Another one may prefer to keep using PHP or even write something in C++! There are also many possibilities for the database system, such as MySQL, PostgresQL, MongoDB, Redis, etc.
That even doesn’t take into account bold developers that don’t trust anything and absolutely want to implement everything from scratch, using the most basic stuff he can find. Let’s make this backend in C, or even in assembler, and have control over everything!
No matter what is chosen, it will work somehow and will fail somehow. Developers will be stuck with the issues and if they cannot find a solution, others will advise using some other technology.
While dealing with all these options and technical details, many will forget something essential: write automated tests! While that seems unnecessary at first, this becomes fundamental as the complexity of the project increases. Automated tests help going forward with a reasonable confidence that what previously worked still works.
The shocking tendency to give up
By the time developers try to implement this stupid clock, discuss and argue about the best frameworks, the customer will get nothing more than explanations, nothing concrete. At some point, he will just go to a store and buy a watch. Here is my clock. It works, it does what is has to do, it ticks with current time. While the idea was great, it was too long and complex to implement, so giving up and going elsewhere is the way to go.
Another example of this could be a customer requesting a web application to collect data and, after some delays or failures, decide to fall back on a simple Excel (or maybe better maybe not LibreOffice Calc) spreadsheet.
Conclusion
While having multiple solutions available is fine, too much can be worse than not enough. When there are too much options to choose among, not enough guidance and error requires costly refactoring, even rewrite everything from scratch, progressing is hard. We end up redoing the same thing, over and over again. The fact that many people implement the same thing over and over again in silos, independently of others, seems not to help, but gathering everyone as a super large team will cause endless argumentation leading all the development to a full stop.
Maybe developers should spend more time using what is there rather than making it work the way they think it should or reinventing the wheel. Maybe customers should spend more time reflecting about their requirements, discuss more with developers to figure out the consequences of these requirements over the project. Maybe providers, especially web hosts, should stop sticking to obsolete technologies such as PHP and offer facilities to host modern web applications in addition to the old stuff that may still be needed for existing projects, or cloud providers should offer options allowing deployment of web applications at the same cost as simpler web hosting platforms (no virtual machine to run a simple NodeJS server, for example). Or maybe I am wrong, getting out of my mind because all that is going on in the world.