If you’ve been working in software development then you’re probably very familiar with the term “production issue”. This is a problem that is discovered in the production environment that causes the application to become unavailable or to exhibit incorrect behavior. For example, a production issue in an online banking system might be that customers’ account balances are doubled every time that they log in.
Because production issues are visible to the end-user, there is generally a rush to resolve the issue as quickly as possible. This often results in a lot of stress, which usually ends up making the problem harder to diagnose and fix. The following are some tips for dealing with production issues in a rational and effective manner:
- Find out exactly what the problem is. Many times a developer will be brought onto the call to answer a question or to help with the debugging process without fully understanding the problem. This will limit the amount of assistance that person can provide.
- Determine if the issue is reproducible in a development or test environment. If the same code is deployed to each environment then the issue is most likely environmental.
- Find out when the issue was introduced to production. If there is a test environment running the code that was last deployed to production, re-create the scenario there and determine whether the unexpected behavior occurs. If such an environment doesn’t exist, have a developer run that version of the application locally. Defects in production may go undetected for weeks or months.
- Avoid the dreaded conference call if at all possible. Management often sets up a call so that everyone who is investigating the production issue can communicate effectively. However, because these calls often last several hours, and because people are often asked to stay on the call “just in case” they are needed, the call loses its benefit as people become less and less focused over time. It’s also extremely difficult to debug the issue when people are asking you questions and talking in the background.
- Take breaks when you need them. Your productivity will seriously decline after several hours of thinking hard and staring at a computer screen. You should insist on taking a break, since giving yourself a mental break may actually allow your subconscious to solve the problem.
- Avoid blaming others. This is unproductive and will probably make the other person much less cooperative. Focus on solutions to the problem.
Try to keep a few of these tips in mind during the next production issue and you will experience less stress and resolve the issue more quickly.
