Bug Reporting: Fixing the “It’s Broke” Diagnosis We All Hate
During my career, I’ve found myself banging my head on my keyboard; I still buy industrial grade mechanical keyboards just because of this. It can happen for a variety of reasons, but it is often when I encounter a new bug in an email or defect tracking system that says something like:
(Product Name Redacted) is broken. Can you please fix it?
Ok, maybe I’m exaggerating just a little bit. There was usually some small indication of what was broken (and the product name was almost never redacted). The point is that it’s much easier to fix something when the problem is well understood by everyone.
This is my attempt to make the world a little bit better by teaching an engineer to fish…↑ well, at least teach an engineer how to write a useful bug report.
No matter how many tests you write or how much exploratory testing you do, bugs will always happen. It is important that we all understand how to communicate the information related to a bug so that it can be easily triaged, scheduled, and fixed. If your bug report is vague, someone is just going to track you down and mercilessly question you for details until they can reproduce and then fix the problem. Like everything else, it’s easier to do it right the first time.
The Who, What, When, Where, and Why
You’re using some product: maybe it’s an application on your desktop, maybe it’s a web app, or maybe it’s an API that you are coding against. Whatever the environment, something is not behaving in the way you expect that it should and you decide to log a bug (or more politely, an “issue”) for it.
This is not intended to be a checklist for items to include in a bug report, but instead a list of things to review that should help you create a bug report that has the right amount of information to allow an investigator to easily assess it.
Before You Log
Before you do anything else, search the existing issue database to verify that an issue for the bug does not already exist. Duplicating a previously logged bug is a waste of everyone’s time. If you do find that the bug has already been reported, then be sure to update it with any additional information that you have which is not already included.
You may also consider reviewing the severity and scope (described below) to verify they are still appropriate. If the failure indicates that the bug is happening more frequently or is affecting more users, then the bug’s priority should be increased.
When you do create a new bug report, it is important to consider the words you include in it. Be sure to include any key words verbatim. Consider using messages from an error, log, or any package or component names so that others who encounter the same failure can easily find it in your defect tracking system.
It may be obvious, but it’s important to describe only a single bug in each report. Never include multiple unrelated bugs in the same issue.
A good defect report starts with a title that provides a clear description of a single problem. A concise title allows others to quickly grasp the issue and also helps the team to triage and prioritize without having to re-read the details of the issue. If the issue is completely obvious, the title may be all that’s needed (e.g. “Application crashes at startup on Windows 7”) but those cases are extremely rare.
A description should provide an explanation of both the expected correct behavior and the actual observed wrong behavior.
The product documentation should provide a description of the expected behavior. If proper documentation does not exist, then you’ll need to provide some logical reasoning for why you expect the product to behave differently from how it is behaving. Sometimes bug reports evolve into feature requests.
Just include the facts here. You may be tempted to try to guess at the underlying cause of the problem here. Just don’t. Guessing may cause others to waste time investigating something that is completely unrelated. Stick with the facts and let the investigator create a hypothesis based solely on their observations. The person investigating it will likely have greater knowledge of the internal implementation than the reporter and will be better equipped to investigate the cause.
It is difficult to overstate how important this is. If the person investigating the bug can’t reproduce it, they will not be able to fix it. At best, they’ll waste time figuring out how to reproduce it, and at worst, they’ll eventually give up after failing to reproduce the issue themselves.
Many factors can influence whether a bug is reproducible by someone else. A subtle difference in a dependency or environment may prevent the issue from reproducing on the investigator’s system, so include any information that is relevant to the problem when describing how to reproduce it.
- Steps – Include a list of steps that consistently reproduce the issue. Provide enough detail so that someone who is unfamiliar with the problem is able to reproduce it. Include any URLs, menus, or button clicks required to reproduce it. If the issue is exhibited within a graphical interface, consider using a tool to collect a screen recording or GIF of your interactions with the product. If the issue is only intermittently reproducible, indicate how frequently it occurs so the investigator will have an understanding of how rarely they may be able to reproduce it.
- Error Message / Logs – If the bug is causing an error to be reported, then include the error message. If there are any log entries that provide information about the failure then including those will be helpful as well. If the application is running in a browser, then consider including a screenshot of the console log within the browser’s Developer Tools.
- Product Version – Include the full semver of the product where you encountered the problem. If you can provide information about the last version where the problem did not occur, include it because it will help to pinpoint when the issue was introduced.
- OS / Version – Just because the problem occurs on Windows doesn’t mean it will occur on the platform and OS that the investigator is using. A bug may appear due to a subtle dependency change in an OS version, patch, or hotfix, so any information that you can provide about your environment is useful. Include the platform and the version (e.g. Windows 10 Version 1903 Build 18362.116, Ubuntu 18.04.2 LTS, macOS Mojave Version 10.14, etc.).
- Browser – If the problem exists in a web app, does it only happen in Internet Explorer or does Chrome or Firefox exhibit the problem as well? Like the OS, the browser is just as important in diagnosing the problem, so include the browser information along with its version if the application is running there.
- Code – If you’re writing an application that exposes a bug with a third-party library or API, then providing a small app or script that illustrates the problem is very helpful. A small code sample can provide most of the information required for reproduction. Be sure to provide the version information about the compiler or runtime under which your application is executing as well so others can reproduce your environment.
- Platform – This is less common, but the architecture may play a part in the issue. The bug may occur on a system with a 64-bit AMD processor but not on a system with a 64-bit Intel processor. Graphics cards also may cause differing behavior.
- Anything Else – If you see something that definitely provides a clue to the cause of the bug then you should include it. Even if you just notice something that might provide a clue, go ahead and include it.
Scope and Severity
Software engineering, like medicine, is an area where problems are more plentiful than the resources required to fix them. Consequently, bugs are triaged in order to address the most critical ones ahead of the less critical ones.
Scope and severity are respective measurements of who is affected and how bad the issue is. The reporter as well as the investigator and others may assign these metrics to an issue and may adjust them as more information is obtained. These really exist just to help to determine how important it is to fix the bug.
- Scope – This is an indication of how widespread the problem is. Does it affect everyone that uses the product, or does it affect only a small group in a rarely used part of the product? Cosmetic issues or bugs that affect a small percentage of users may never be addressed unless there are other circumstances which increase the importance of the bug.
- Severity – How bad are the effects of the bug? If the bug damages the system or causes other systems to crash, then fixing it will be of much higher priority than a bug which can be avoided with a small work around or a bug which only affects the application cosmetically.
The graph below shows how the levels of scope and severity for a bug can help to prioritize its importance. This becomes even more useful when you quantify scope and severity and consider their product as an indicator of a bug’s level of importance.
Get to Work
Now that you have been introduced to the important parts of a good bug report, you might be tempted to try to pare it down to just the few parts that you feel may be actually required. If you think the bug you are logging only needs a couple of details, you’re wrong.
When you provide less information in your bug report, the investigator has to spend more effort reviewing the things you left in order to reproduce it. A bug report with only a title and a brief description is never complete. You’re already in the environment that reproduces the bug, so help out and share the details you already have.
Once you have created a new bug report it’s time for someone to get to work. Hopefully this article has helped you to create a report with enough information to help them address the bug quickly and with a minimum amount of effort.
The investigator will appreciate that you took the time to create a concise bug report with accurate reproduction steps and environment information.
Prior to joining Accusoft as a senior software engineer in 2007, Steve Brooks consulted to organizations including NASA and Microsoft developing applications and SDKs. A graduate of North Carolina's Appalachian State University, Steve has a Bachelor's of Science in Applied Physics and a background focused primarily in scientific instrumentation development. Since joining Accusoft, he has worked on several products including PICTools, ImagXpress, and PrizmDoc Suite. Steve lives in the mountains of North Carolina where he enjoys spending his free time with his wife and daughters.