NYSE Proves it’s Not Too Big to Fail
Big news for investors last week was the 4-hour shutdown of the New York Stock Exchange (NYSE) on July 8th. Early speculation by news outlets blamed the outage on a coordinated cyber-attack, most likely by China. Fuel was added to the speculative fire when United Airlines grounded all flights for nearly two hours due to a computer glitch and The Wall Street Journal’s homepage went down that same morning.
Most fears were allayed when the FBI, the Department of Homeland Security and NYSE officials all stated that there was no evidence of any cyber-attacks. As the day wore on, it became clear that the real culprit was a software upgrade glitch.
According to the NYSE, on Tuesday evening it deployed a software release on one trading unit in preparation for a July 11 industry test of the upcoming Security Information Processor (SIP) timestamp requirement. Wednesday morning pre-trading revealed communication issues between customer gateways and the trading unit with the new release. It was determined that the NYSE customer gateways did not have the proper configuration compatible with the new release. Identifying the problem early, allowed the NYSE to make sure the issue was quickly fixed before the official trading day began—or so it thought.
Thinking all was solved, the NYSE opened for business per usual at 9:30am. It soon became apparent that the “fix” resulted in additional, unforeseen communication issues. Frustrated, officials shut down the market for 4 hours to get the mess straightened out.
News reports suggest traders barely batted an eye, they simply shifted their trading to the roughly dozen or so other outlets available to them (i.e. NASDAQ). Ultimately, a poorly executed software upgrade caused a 4 hour shut-down, a reputation ding, and potentially a long-term loss of business—as traders may decide to stick with other brands. While there is no lasting harm to the data or the system, perhaps not so the hit to NYSE’s reputation.
There are real risks in software development and multi-environment deployments. To avoid these situations, ECHO Technology Solutions has developed its own set of checks and balances to its deployment and testing practices. While many companies have one favored method of testing, ECHO employs several methods to get the highest level of coverage over the software. For each environment we combine testing checklists, deployment checklists, task management and client review processes prior to reaching production. Contact us to learn how we can help you properly position your testing and deployment practices to mitigate your software risks.