How do you fix a bug you can't replicate?

Language

Different programming languages will have their own flavour of bugs.

C

Adding debug statements can make the problem impossible to duplicate because the debug statement itself shifts pointers far enough to avoid a SEGFAULT---also known as Heisenbugs. Pointer issues are arduous to track and replicate, but debuggers can help (such as GDB and DDD).

Java

An application that has multiple threads might only show its bugs with a very specific timing or sequence of events. Improper concurrency implementations can cause deadlocks in situations that are difficult to replicate.

JavaScript

Some web browsers are notorious for memory leaks. JavaScript code that runs fine in one browser might cause incorrect behaviour in another browser. Using third-party libraries that have been rigorously tested by thousands of users can be advantageous to avoid certain obscure bugs.

Environment

Depending on the complexity of the environment in which the application (that has the bug) is running, the only recourse might be to simplify the environment. Does the application run:

• on a server?
• on a desktop?
• in a web browser?

In what environment does the application produce the problem?

• development?
• test?
• production?

Exit extraneous applications, kill background tasks, stop all scheduled events (cron jobs), eliminate plug-ins, and uninstall browser add-ons.

Networking

As networking is essential to so many applications:

• Ensure stable network connections, including wireless signals.
• Does the software reconnect after network failures robustly?
• Do all connections get closed properly so as to release file descriptors?
• Are people using the machine who shouldn't be?
• Are rogue devices interacting with the machine's network?
• Are there factories or radio towers nearby that can cause interference?
• Do packet sizes and frequency fall within nominal ranges?
• Are packets being monitored for loss?
• Are all network devices adequate for heavy bandwidth usage?

Consistency

Eliminate as many unknowns as possible:

• Isolate architectural components.
• Remove non-essential, or possibly problematic (conflicting), elements.
• Deactivate different application modules.

Remove all differences between production, test, and development. Use the same hardware. Follow the exact same steps, perfectly, to setup the computers. Consistency is key.

Logging

Use liberal amounts of logging to correlate the time events happened. Examine logs for any obvious errors, timing issues, etc.

Hardware

If the software seems okay, consider hardware faults:

• Are the physical network connections solid?
• Are there any loose cables?
• Are chips seated properly?
• Do all cables have clean connections?
• Is the working environment clean and free of dust?
• Have any hidden devices or cables been damaged by rodents or insects?
• Are there bad blocks on drives?
• Are the CPU fans working?
• Can the motherboard power all components? (CPU, network card, video card, drives, etc.)
• Could electromagnetic interference be the culprit?

And mostly for embedded:

• Insufficient supply bypassing?
• Board contamination?
• Bad solder joints / bad reflow?
• CPU not reset when supply voltages are out of tolerance?
• Bad resets because supply rails are back-powered from I/O ports and don't fully discharge?
• Latch-up?
• Floating input pins?
• Insufficient (sometimes negative) noise margins on logic levels?
• Insufficient (sometimes negative) timing margins?
• Tin whiskers?
• ESD damage?
• ESD upsets?
• Chip errata?
• Interface misuse (e.g. I2C off-board or in the presence of high-power signals)?
• Race conditions?
• Counterfeit components?

Network vs. Local

What happens when you run the application locally (i.e., not across the network)? Are other servers experiencing the same issues? Is the database remote? Can you use a local database?

Firmware

In between hardware and software is firmware.

• Is the computer BIOS up-to-date?
• Is the BIOS battery working?
• Are the BIOS clock and system clock synchronized?

Time and Statistics

Timing issues are difficult to track:

• When does the problem happen?
• How frequently?
• What other systems are running at that time?
• Is the application time-sensitive (e.g., will leap days or leap seconds cause issues)?

Gather hard numerical data on the problem. A problem that might, at first, appear random, might actually have a pattern.

Change Management

Sometimes problems appear after a system upgrade.

• When did the problem first start?
• What changed in the environment (hardware and software)?
• What happens after rolling back to a previous version?
• What differences exist between the problematic version and good version?

Library Management

Different operating systems have different ways of distributing conflicting libraries:

• Windows has DLL Hell.
• Unix can have numerous broken symbolic links.
• Java library files can be equally nightmarish to resolve.

Perform a fresh install of the operating system, and include only the supporting software required for your application.

Java

Make sure every library is used only once. Sometimes application containers have a different version of a library than the application itself. This might not be possible to replicate in the development environment.

Use a library management tool such as Maven or Ivy.

Debugging

Code a detection method that triggers a notification (e.g., log, e-mail, pop-up, pager beep) when the bug happens. Use automated testing to submit data into the application. Use random data. Use data that covers known and possible edge cases. Eventually the bug should reappear.

Sleep

It is worth reiterating what others have mentioned: sleep on it. Spend time away from the problem, finish other tasks (like documentation). Be physically distant from computers and get some exercise.

Code Review

Walk through the code, line-by-line, and describe what every line does to yourself, a co-worker, or a rubber duck. This may lead to insights on how to reproduce the bug.

Cosmic Radiation

Cosmic Rays can flip bits. This is not as big as a problem in the past due to modern error checking of memory. Software for hardware that leaves Earth's protection is subject to issues that simply cannot be replicated due to the randomness of cosmic radiation.

Tools

Sometimes, albeit infrequently, the compiler will introduce a bug, especially for niche tools (e.g. a C micro-controller compiler suffering from a symbol table overflow). Is it possible to use a different compiler? Could any other tool in the tool-chain be introducing issues?

If it's a GUI app, it's invaluable to watch the customer generate the error (or try to). They'll no doubt being doing something you'd never have guessed they were doing (not wrongly, just differently).

Otherwise, concentrate your logging in that area. Log most everything (you can pull it out later) and get your app to dump its environment as well. e.g. machine type, VM type, encoding used.

Does your app report a version number, a build number, etc.? You need this to determine precisely which version you're debugging (or not!).

If you can instrument your app (e.g. by using JMX if you're in the Java world) then instrument the area in question. Store stats e.g. requests+parameters, time made, etc. Make use of buffers to store the last 'n' requests/responses/object versions/whatever, and dump them out when the user reports an issue.

If you can't replicate it, you may fix it, but can't know that you've fixed it.

I've made my best explanation about how the bug was triggered (even if I didn't know how that situation could come about), fixed that, and made sure that if the bug surfaced again, our notification mechanisms would let a future developer know the things that I wish I had known. In practice, this meant adding log events when the paths which could trigger the bug were crossed, and metrics for related resources were recorded. And, of course, making sure that the tests exercised the code well in general.

Deciding what notifications to add is a feasability and triage question. So is deciding on how much developer time to spend on the bug in the first place. It can't be answered without knowing how important the bug is.

I've had good outcomes (didn't show up again, and the code was better for it), and bad (spent too much time not fixing the problem, whether the bug ended up fixed or not). That's what estimates and issue priorities are for.