Case Study: Software Failure Remediation

Organizations experiencing a critical software failure need software failure remediation from highly trained veteran software experts.  Prolifogy Software & Consulting can assist its clients with software failure remediation and perform a detailed analysis of the client’s software.  This is a case study highlighting a past software failure remediation project Prolifogy has been involved with.

 

The Problem

A small-cap, publicly traded B2B company approached Prolifogy to seek assistance with a daily recurring issue involving its server software. The client complained that the server’s responsiveness would decline over the course of each day. It would either slow to an unacceptable pace or stop functioning altogether by the end of each day.

The client took two courses of action. First, it directed its own internal development team to diagnose and troubleshoot the problem, but they were ultimately unable to do so. Second, the client rebooted the affected servers every night in the early morning, when service volume was low. Doing so “reset” the server back to its regular level of performance, but the process involved disruption to their own customers. The client’s own uptime guarantees and SLAs were not being met, prompting their customers to threaten moving to a different vendor.  Software failure remediation was clearly in order.

The Software Failure Remediation Process

The client brought Prolifogy Software & Consulting in to provide software failure remediation services. The client was under the impression that a single line or area of code was most likely to blame, but they could not locate the code in question. They had already attempted disabling or altering various blocks of suspect code on their own.

For various technical reasons, Prolifogy was not able to execute the client’s code on its own local computers in a development mode for further in-house study. Prolifogy was also not able or permitted to profile the running code on the client’s production server. Prolifogy therefore had to operate “in the blind,” under the constraint of static code analysis only, and forego any dynamic analysis it had intended to perform.

Methodology

Prolifogy first performed an initial static source code quality analysis.  There was no more specific direction to pursue at the time.  We readily identified poor multi-threading and exception handling practices as being particularly problematic. After an initial review, we eventually deemed the client’s gradual performance degradation issue to be most likely a conglomeration of different issues converging to create slow server performance. This is in contrast to a single “smoking gun” line or section of code.

Prolifogy then focused on the multi-threading and exception handling issues.  We identified every location in the program we could find exhibiting poor coding practices. Per mutual agreement, we sent regular reports containing code-fix suggestions to the client. The client then fixed the code and tested the fixes before pushing them into production. The arrangement whereby the client itself made the fixes was notable. First, it allowed the client to maintain control over its software while also allowing its development team to learn firsthand about the problematic coding patterns that were causing slowdowns. Secondly, the client could avoid such problems from being introduced again going forward.

Prolifogy directed that additional logging and instrumentation be inserted into the production code, in lieu of a dynamic analysis. Doing so would facilitate troubleshooting by localizing the sources of performance degradation. Upon receiving the runtime logs back after a couple of days of production execution, we were able to continue the software failure remediation by focusing on and further pinpointing the areas where the most delays were observed.

Observations

The client noticed clear improvement after only a couple of reports. Indeed, the server was not slowing down as much as it had previously. The failed software turnaround process was starting to work. The client readily observed that the fixes were having a positive effect on its server software. This report-and-fix cycle repeated a few more times: Prolifogy suggested modifications, and the client made the changes, tested them, and deployed the changes to production each time. After a short while, the client’s server performance issues were completely remedied.

Prolifogy conducted training with the client’s development team after the suggested fixes were fully implemented. This allowed the team to solidify the knowledge they had gained throughout the troubleshooting process. It also allowed them to more comprehensively understand the source of the problem and ask any follow-up questions.  We advised the client that nothing nefarious had taken place and that nobody should be fired. Rather, we regarded it as a learning process and team building exercise for the client.  Finally, the client decided to hire new software developers to augment its existing team. Prolifogy participated in the job interview process to vet candidates and provide opinions on the candidate pool. After careful review, the client made its choices. The client reported being very happy with the talent selections they had made under our advisement when we followed up a year later.

Conclusion

No linter or AI agent was capable of performing the level of troubleshooting necessary to overcome the reported problems at hand.  We also could not have expected conventional troubleshooting tools or an AI agent to have concluded that the root cause was numerous systemic failures rather than a single “smoking gun” section of code.  Moreover, Prolifogy had to overcome significant obstacles, including its inability to conduct a dynamic software analysis, in arriving at the final failed software turnaround solution. We could not have identified and ultimately resolved these issues without decades of hands-on industry experience and vast amounts of formal education.

There is no such thing as a boiler plate remediation process. Every project is different and every client has different needs, comfort levels, security requirements, and expectations. All of these details are discussed in advance of each engagement. Although our availability is subject to change, we can often be available to begin work in a day or less, depending on the urgency and nature of the engagement. Contact us for more information.