As of 1st August 2023, Knight Capital Group commenced operations with a staff of more than 1400 people, holding a 17.3% share of NYSE and 16.9% of NASDAQ. Working as a broker at Knight Capital is an opportunity of a lifetime for those interested in the US equity trading industry.
Today presented an unprecedented situation. At the opening of the New York Stock Exchange at 9:00 AM, the first retail investor commenced trading. Within 45 minutes, software used by Knight Capital had processed over 4 million transactions, resulting in a financial loss of $460 million and pushing the company to the brink of insolvency.
It appears that the recent disaster was caused by a combination of unfortunate circumstances related to computer code. The development team had previously deployed an upgrade to the live server, which appeared to be innocuous; however, the bug contained within was a ticking time bomb.
Occasionally, there can be issues with software that have a far-reaching effect, such as the Log4j scandal, which caused disruption in the computing sphere for a number of weeks, affecting millions of people around the world.
Recently, a security vulnerability was identified in Log4j, one of the most popular Java-based logging applications. If exploited, this vulnerability could enable malicious actors to remotely execute code on a target machine and access sensitive data or install malware.
How significant was the issue? When you take into account that Apple, Amazon and Twitter all rely on Log4j, it becomes clear why Akamai Technologies recorded over 10 million attempted exploitations of the issue every hour in the US alone.
A Bug? What’s a Bug?
When computer code produces an inaccurate or unexpected result or behaves in an unintended way, it is referred to as a bug. While poor coding practices may be a factor in the occurrence of bugs, this is not always the case.
The NASA Climate Orbiter, worth $125 million, sadly ended up in flames on Mars. The magnitude of force, in pounds, that the thrusters should apply was determined by a software programme, yet another reader of the data interpreted it as metric. What can be learnt from this incident?
Upon individual evaluation, each code segment operated successfully. The engineers from Lockheed Martin Astronautics in Colorado ran the calculations, yet there was an oversight and the conversion to the metric system was overlooked. Nevertheless, NASA utilised the traditional unit of newtons per square metre for their calculations.
An example of this is the incident at Knight Capital, which was caused by a section of outdated code that had not been removed from their systems. When a flag was introduced in the update, the software misinterpreted it as being in a testing environment and consequently tried to execute a large number of transactions.
The inquiry concluded that Knight Capital did not have either formal code reviews or a quality assurance division, as evidenced by the study of the incidents. This means that there was no one responsible for detecting errors. Consequently, there were insufficient safety mechanisms in place.
The Quality Assurance (QA), DevOps Engineering, Software Testing and Code Review Teams are all invaluable resources however, they are not sufficient to guarantee that defects will not be introduced. In certain cases, the only way to address an issue is to identify it when it arises in production and rectify it in a timely manner.
The View from Without
Due to the restrictions imposed by the software development process, such as meeting customer needs, meeting deadlines, collaborating with other developers and responding to sudden changes, it is easy for issues to go unnoticed.
It is often said that hindsight is a wonderful thing, and it is often much easier to assess code after it has been written, rather than whilst under pressure. Every programmer has looked back and wished they had done something differently, but when time is not of the essence, it is easy to make such assessments.
Conversely, our customers utilise our products in a completely different environment. They have unrestricted access to it on their preferred platform. Under these circumstances, it is to be expected that customers may encounter issues.
Bugs can cause a range of issues, from being intriguing to frustrating for users. However, could they be harnessed to the benefit of developers, the public, and ourselves?
Step in Bug Bounty Programs
Many companies, websites and software developers offer ‘bug bounty programmes’, rewarding users who report any bugs, particularly those that could be used as security exploits, with a financial or public reward.
Twitter and Google, two of the most prominent names in technology, both make use of Bug Bounty programmes. The US Government, along with many other countries, offer rewards to those who responsibly disclose any security vulnerabilities found on official websites.
There have been communities developed around a number of websites which collate information related to Bug Bounty programmes. This has been met with widespread acceptance. It is important to understand why this has been the case.
It has become apparent that software development can benefit greatly from the collective knowledge of a large group of people. While individual developers can only test to a certain extent, by engaging with the wider community, users and developers are able to collaborate in order to identify and address any issues concerning bugs and security vulnerabilities.
By incentivising the discovery of security vulnerabilities, we are able to collaborate with knowledgeable professionals to increase the reliability and security of our product. This is an open-source initiative to further enhance the safety of our software.
Bug bounty schemes have proven to be beneficial for young programmers, as they have been able to demonstrate their ability and understanding of software development practices and security protocols through the development of code.
Establishing a bug bounty programme is a straightforward process for businesses. There are numerous websites that offer these programmes, so all that is required is to promote your own initiative and set a reward that is appropriate to the size of your business.
In conclusion, bug bounty programmes should not be seen as an alternative to quality assurance or other security practices such as DevSecOps; rather, they should be seen as a supplementary approach. To fully realise the potential of software, it is essential that the highest quality is achieved.