We are frequently asked about how we handle the process of fixing bugs in the software we develop. A question of this nature brings up a few important issues. For instance, I recently received the following email from a client that expressed just such concerns particularly well:
“Hi Jon, This issue has come up before but I have not pursued it because of the good service and consideration you have given us... Now others have asked and I feel I have to check into the issue of software bugs. I am usually quoted for a duration of time that will be needed to fix the broken things that we identify. However one could argue that the issue is a problem that lies in the execution of a past feature request. So its resolution is more of a support issue rather than something that should incur new cost. Do you have a certain policy or procedure that I should be aware of in regards to how bugs are resolved? Is it related to how recently the feature that has a bug was developed?” — Troy Web Consulting Client
It’s a reasonable and fair question to ask, in particular as it relates to costs associated with fixing these bugs. On the surface, it seems as though a “bug” exists due to an error on the part of the development team. However, that is merely one of many reasons why a software bug may exist.
What the heck is a “bug” anyway?
Leave it to a consultant to answer a question with a question - but of course, the root of the issue lies in exactly what we’re talking about when we call something a bug. Although the (urban) legend detailing the origin of the term “bug” in the computer industry has roots in a literal bug (a moth) found in the Mark II computer by computer programming pioneer Grace Hopper, the more standard definition is “a failure or a flaw [that] produces an incorrect or undesired result that deviates from the expected result or behavior”.
You will note that based on this common definition of a bug, the word defect is not used. Yet the term bug is used when any unexpected results or behaviors appear in a system. This is because a defect is merely one of many many reasons such unexpected results may surface. Here are some of those reasons:
#1 - Lack of clear specifications, and conflicting specifications causes bugs
In the software development process, it’s not uncommon to “build the plane while you fly it”. That’s a funny way of saying that as a system is being built and tested, the requirements of that system “evolve” - this can be because all the various usage scenarios weren’t fully thought out (or discovered), or perhaps a need to get to market quickly forces the system to get figured out along the way. This approach, often mischaracterized as “agile development”, is a reality in the software industry. The bottom line is there is a constant swirl of unknowns throughout the project that get addressed and resolved along the way; Software is complex and like it or not, the best process to address complexity is to be adaptive. As the “rules” of a software system develop or evolve over the span of a project there are often conflicts between these rules - logical deadends that create “edge cases” in certain scenarios. Bugs surface as a result. Is the code at fault? You betcha, but it hardly could have been known beforehand. The software industry has yet to figure out how to procure crystal balls and until it does, these bugs are not defects. FEATURED RESOURCE Before you start development or hire a software partner, you need to fully understand the wants and needs of your application. Use this template to prioritize the scope of your specifications.
#2 - Updates cause bugs
Software today is not merely a “computer program” - rarely is there a single component involved and more importantly, it’s almost never completely self-contained and built from the ground up. Modern software is a compilation of existing code libraries, running on a core technology or two and it gets delivered to devices (computers, laptops, smartphones, other devices) using internet protocols and other layers of software (browsers, operating systems). To put it in Shrek terms, Software is like onions; it has layers. Software is certainly like Ogres.
Updates happen every day, in all different layers of your software project, sometimes automatically. These updates introduce unexpected behaviors (“bugs”) and there’s no stopping it. The only thing to do is deal with it. There are a variety of ways to do so, all of them grounded in the commitment to constantly keep fixing your software. Bugs 2, Defects 0. Note: It is sometimes tempting to shy away from as many updates as possible in an attempt to minimize this introduction of bugs; This is usually a terrible strategy - ask anyone who decided to back off their security patches on their WordPress website because the updates “broke” their plug-ins; More-often-than-not, such a decision results in a hacked website (because it wasn’t patched) and a gigantic remediation expense.
#3 - Data causes bugs
Most software applications are just conduits to data; They collect data, store data, and retrieve data. Developing code to process data often makes certain reasonable assumptions about that data. Business Rules, as they are called, enforce a certain amount of data continuity and integrity upon which the software relies. What happens when something else (a data migration, another system, a user) corrupts that data? It’s extremely common for the system to begin to yield unexpected results; Translation: “Bugs”. Cause? Not the code. Developer “defect”? Nope.
Even a change to business rules, without the correct analysis of the existing data and other systems that may interact with that data, can render problems in software operation. This is a very common issue when a rule changes to make a data element required. What about all the data in the system that has a null value for that field? Changes happen and every implication cannot always be forecasted - to do so would be grossly inefficient - rather, the best path forward is to make the change and stomp out the bugs. None of which has anything to do with defects.
#4 - APIs / Other Systems cause bugs
Over the past 15 years, we have seen a drastic shift in software architecture - a movement away from the “monolithic” application, transitioning to a service-oriented system (and to an extreme, a microservices ecosystem). This shift addressed a critical problem in the software of the early 2000s - massive code bases that created whack-a-mole scenarios, i.e., make a change to one piece of code, and the ripple effect on the system causes other problems.
The service-oriented approach breaks up a system into pieces, each subsystem addressing its own concerns (e.g., account management, order fulfillment, inventory management). This is great for simplifying a codebase but the orchestration of the various components becomes complexity adding. There is still a dependency between the subsystems, but the dependency is at least well defined so, from a technical standpoint, it’s better. But it’s not all roses…
The problem that surface is not a technology problem but rather a people problem. Subsystems are often managed by different (specialized) teams and one change to a subsystem or component can change the way other components need to interact with it - and it’s not always well thought out enough to be communicated across teams. The bottom line? In the modern stack of integration and interoperability, you cannot control all the processing that needs to take place within the system boundaries of your software. Many of these subsystems are SaaS products over which you have no control.
A recent example of this issue surfaced when we were adding sales tax processing to a large eCommerce system by leveraging the Avalara Tax Compliance software service. During our development and subsequent software testing, we discovered issues in the Avalara API that yielded unexpected results. The bug was not in our system, per se, but rather, in the external system upon which we were relying. We spent a significant amount of time debugging issues that turned out to not even be a part of our system. More bugs, still no defects…
#5 - Lack of “maintenance” causes bugs
In general, software is a dynamic, complex thing - and your software should not be treated as a static asset. Over time, so many things can change (new iPhone comes out, changes to Apple’s App Store terms of service, Browser updates, etc...) that will have a ripple effect on your system. Your investment in a software system requires maintenance - think of it as an insurance plan (assurance plan?) to make sure you are not caught off guard.
One thing we see happen on systems that aren’t well maintained is that while they may work well enough without ongoing attention, over time, the issues are piling up and boxing you out of options until it finally gets to a critical point; When you finally decide to move forward with updates and fixes, the complexity and dependencies of so many updates at once become unbearably expensive and prohibitive. You waited too long to have viable workarounds and now you’re staring at a rewrite.
Other more subtle aspects of scheduled maintenance can keep your expectations met with your software. For example, as a software system runs, the data behind the system grows - whether its transaction data or other logging of system data. This data needs to be actively reviewed - should some data be archived? Deleted? Without keeping the database “cleaned-up”, over time you can have a drastic drop in system performance - to the point of being unusable. Slow response times are certainly not expected behavior (and therefore, “bugs”) but they can arise, as in this case, simply from having the system run too long without monitoring and maintenance.
How is your software development project holding up? Do your know the status of each feature, bug fix, etc. This project status template will help you stay on top of the many moving parts that come with every software project.
#6 - Poor project management causes bugs
You may be beginning to see a theme here. Bugs can run rampant in code through no fault of the “coders”. If that’s the case, is there any hope? Turns out, preventing bugs is more about your process for developing software than it is about the coding itself. One of the biggest challenges with software projects is how unique every project is. The devil is always in the details. The only way to prevent bugs is to manage the process using appropriate methods for that project itself (Not to mention having a good project manager).
We use a hybrid approach to project management by evaluating the project along various criteria to determine the sweet spot between a “predictive” model (waterfall software development model) and an “adaptive” model (agile software development model). Part of the criteria is the criticality of the software and the tolerance for bugs. Yes, believe it or not, having bugs in software is perfectly acceptable in some cases.
Assessing your risk tolerance with bugs is a critical step in planning your project. A lower risk tolerance will require a more predictive model of development, which has its own pros and cons; It is important to know what you will be up against with regards to timeline and budget. It will be important to know how your ticket flow works, and how much budget will be spent on activities like code review, bug tracking, and software quality assurance testing. If you choose the wrong project management model, you can quickly wind up in the doghouse.
#7 - Human Error causes bugs
Defects are a reality - for now at least, humans are designing and creating the code behind software. There will be errors. The issue is far more about what’s done about the errors than the errors themselves.
There are many humans involved in a software project - including the client subject matter experts (SME’s) and the stakeholders who perform acceptance testing. In the case of an outright defect that makes it through to production, you can put effort into figuring out who was to blame but it’s usually more efficient to come together as a team, fix the issue, and focus the remaining energy on improving the process. This will not only reduce the risk of future issues but it will also help you conserve your energy and live to fight another day. Software development is a marathon, not a sprint, despite the agile terminology.
Naturally, we do not charge for bug fixes that are defects in our workmanship; More often than not, it’s not that simple. A solid Software Development Life Cycle Process (SDLC) is the key to effectively managing bugs. Having a tailored, project-appropriate method for managing bugs of all sorts, including defects, will provide a much more predictable set of behaviors from your software.
No surprise that planning is the most important activity in your software project -
checkout our Project Assessment Guide to review your project and see what steps you need to take to prevent bugs in your next project.