The Problem Of Software
Why “software eating the world” means that eventually we’ll have to make radical changes to the way we do things around here
It’s banal to comment that software is everywhere these days and will be even more everywhere in the years ahead. What’s less banal is to note that hardly anyone really seems to be talking about what this implies.
As is so often the case, economics plays an enormous but under-appreciated role. Large corporations such as the automobile manufacturers, airline manufacturers, and practically every other organization that provides goods and services directly or indirectly to the public, are desperately trying to incorporate software into their business models. So much, so obvious. What these large corporations cannot do, however, is attract the talent necessary to achieve these goals reliably.
This is because large corporations have HR departments that use all manner of delightful tools to determine average salaries and then impose this constraint on hiring managers. In this endeavor they are staunchly supported by CFOs keen to minimize the annoying impact on the bottom line that results from having to actually pay people for the work they perform. Thus, the aforesaid large corporations cannot ever attract talented people who rationally prefer to work for companies that will pay them several times what a moribund large corporation will offer. Corporations often compound this problem by outsourcing software development to bodyshops located where labor is cheap, which invariably results in even poorer outcomes than would be obtained by employing local software developers.
As a result, we have a world in which the government always employs those with the least talent in return for a modest salary, a job for life, and a gold-plated retirement package decades later; large traditional corporations employ the mid-rank with mid-level compensation or outsource with catastrophic consequences; and the best and most competent people work for the tech giants where they receive compensation packages beyond the dreams of 99.5% of all white-collar employees.
This automatic process of talent flocculation inevitably results in nearly all software being sub-standard relative to real-world requirements. As everyone in the chain is under-performing relative to the top talent this means that less capable requirements analysis leads to less adequate software specifications and less competent software development create less robust software that quite possibly only addresses the real operational needs tangentially. Meanwhile, managers eager to secure their performance-related bonuses put pressure on development teams to release code according to an arbitrary timeline regardless of any misgivings anyone may harbor.
In other words, we have created accidental human systems pretty much guaranteed to deliver failures large and small. Sometimes these failures are spectacular, as per the fatal crashes of Boeing 737 aircraft due to software bugs in the flight control systems. More often these failures are merely frustrating, as per apps that fail to store passwords, permit strangers to access personal data, and so forth. But as software plays an ever-greater role in an ever-wider portfolio of products, serious consequences will become weekly occurrences.
Meanwhile the top talent working for the likes of Google, Facebook, and other well-known organizations beaver away to further the noble task of selling more junk to people who are in debt because they’ve already bought too much useless junk.
One is permitted at this juncture to remark upon what a curious world it is that we inhabit.
There is, however, a glimmer of hope. When we look back at the history of industrialization we see an analogous series of events. Early on, corporations were so enamored of the benefits machines could deliver that the concomitant problems were entirely ignored. So what if workers lost limbs or even their lives because of poorly-designed machines that had no protective coverings and were highly failure-prone? Replacement labor was always available at low cost and certainly at a far lower cost than that required to make machines less dangerous and more reliable.
Over time, however, corporations began to notice that there were benefits to retaining skilled workers and viewing them as more than merely anonymous cost items on accounting ledgers. Furthermore, as more and more accidents came to light, public opinion slowly changed. Instead of being delighted at the reduced cost of garments made possible by automation, customers began to expect more from producers. Taking reduced costs for granted, customers began to agitate for greater care and attention to be paid to those tending the machines that manufactured their crinolines dresses and cotton undergarments.
Slowly, and in many cases reluctantly, corporations began to address the most egregious safety issues.
Today an automated factory has safety built into it from the ground up. As a result, industrial accidents arising from machine-human and robot-human interactions are rare indeed. It may have taken nearly two hundred years to get there, but eventually we humans caught up with the implications of our innovations.
While analogies are always charming, a certain caution must also be admitted. Software is not like machinery. With a machine we can fully understand the operation and we can account for all the parts. An internal combustion engine today, for all it has benefited from more than a century of refinements, is essentially unchanged from its precursors that powered the Packards and Austin-Morris jalopies of old. Pistons still slam up and down within cylinders and a fuel-air mixture is still ignited under pressure to produce force.
Software, however, is neither visible nor obvious, especially in the realm of machine learning when very often the internal processes are in no way fully understood by the programmers who develop such systems. Furthermore, unlike the situation with machines, software proliferates exponentially. Today a Chrylser V8 is much the same as a V8 from the 1930s; today’s smartphone app is nothing like a COBOL program running on an early IBM S/360.
We must therefore be circumspect when we look backward to uncover analogous events. As Samuel Clemens, aka Mark Twain, remarked, “History doesn’t repeat itself, but it often rhymes.”
If we are in for a period of rhyming, or at least some degree of assonance, what can we expect from the coming decades as software continues, in the charming phrase, to eat the world?
Obviously we can expect a great many more well-publicized accidents arising from buggy software, and far more unpublicized inconveniences that are insufficiently newsworthy to be reported. It is likely that several decades will pass before techniques and procedures are developed to prevent the most obvious errors.
Simultaneously, clever people will try to solve the problem by using software to fix itself. This trend has been underway for decades and has resulted in a wide variety of tools developers can use to remedy the most obvious faults in their work prior to checking their code into whatever repository their employer favors. Testing tools will continue to play catch-up, being always a step behind whatever new language or technique is invented to make programming faster and easier, but they will nevertheless continue to evolve and improve.
In parallel, consumers will tire of having their toys fail to perform as expected and this will gradually translate into managers being less eager to push new code out into the world before it is truly ready. This will then enable more complete testing, which will catch more bugs and more code that fails to deliver what was really needed. Which in turn will lead to more emphasis on more adequate requirements gathering and definition, which will lead to better software development.
Contemporaneously, the eye-watering compensation provided by the likes of GoogleBook will gradually decline. Many traditional organizations will realize that just as they outsourced component manufacture, they can outsource software development if they do so to select specialists rather than to cut-price bodyshops. Gradually today’s enormous disparities in pay will lessen and best practices will spread, enabling even the mediocre to perform to an adequate standard. The bar, as they say, will slowly be raised and an increasing number of people will be able to pass over it.
And so, gradually and for the most part imperceptibly, continuous improvement and economic adjustments will reduce the many problems that poor-quality software will have imposed on the world.
Assuming, that is, some whizzy new technology doesn’t appear and the entire cycle begins anew under a different guise, rendering software as relevant as a wooden ox-drawn plow.
