What’s BI?
Business intelligence (BI) has a history that dates back more than 150 years. The development of BI was initially separate from computers, however the widespread availability of computers and databases created an opportunity for BI to gain traction and flourish. The growth of BI has been closely connected to the evolution of computer technology and databases.
BI Pen-and-paper
The widely recognised phrase “business intelligence” was first used by Mr. Richard Miller Devens in his 1865 book Cyclopaedia of Commercial and Business Anecdotes. Devens used this phrase to illustrate the success of Sir Henry Furnese, a prominent banker, in leveraging knowledge to gain an advantage over his rivals. His pioneering concept pointed to the fact that it was preferable to devise a corporate strategy based on data and facts rather than relying on intuition. Subsequently, other individuals who recognised the value of information built on this idea.
In the late 1800s, Frederick Taylor developed the first structured approach to business analytics in the United States. His scientific management method began with the analysis of time and motion, which was used to evaluate manufacturing processes and the physical movements of workers in order to discover opportunities to increase industrial productivity. Taylor’s pioneering efforts in the field of business analytics laid the groundwork for the subsequent innovations that continue to be made in the field.
Taylor ultimately gained the role of consultant to Henry Ford, who was the pioneer of tracking the duration of the manufacturing process of each component of his Ford Model T on the assembly line in the early 20th century. His efforts and accomplishment revolutionised the industrial sector worldwide. Even so, he still resorted to traditional methods such as pen and paper.
Business Intelligence is facilitated by technology
In the 1930s, the development of electronic computing technology was still in its early stages; however, the advancement of this technology was hastened during World War II as a result of the Allied forces’ efforts to decrypt German codes.
Prior to the 1950s, computers primarily relied on punchcards or punched tapes as the primary method of storing data. These punchcards and tapes were cumbersome and cumbersomely large, containing microscopic holes to hold the relevant data for the computer to process. However, in 1956, IBM revolutionised the world of computing by creating the very first hard disk drive. This innovation allowed for the storage of large quantities of data, with easier access compared to the punchcards and tapes of the past.
In 1958, Hans Peter Luhn, an IBM researcher, released a highly influential document entitled ‘A Business Intelligence System’. This document outlined the concept of ‘selective distribution’ of materials to ‘action points’ according to ‘interest profiles’. Luhn’s work continues to have a major impact on the field of business intelligence, as he accurately predicted major trends that have since become commonplace, such as the use of information systems to track and anticipate user preferences through the use of machine learning. As a result, Luhn is often referred to as the ‘Father of Business Intelligence’.
Despite Luhn’s design having generated a great deal of interest, it was ultimately found to be too costly for practical use, necessitating further technical advancement before it could be considered a viable economic option.
Over the course of the following decade, the usage of computers saw an unprecedented surge in popularity, despite the fact that each individual computer was an immense machine that required an entire floor of a building to house it, and the expertise of several highly qualified engineers to keep it running. This prompted experts to re-examine the potential of computers to draw conclusions from data. However, the main issue at the time was that there was no centralised system in place to store all the data in one location. Without any form of organisation, data is unable to generate any meaningful information. Consequently, the initial versions of database management systems were created to solve this problem, which would later be referred to simply as databases. These systems enabled the earliest forms of database searches, employing the use of a binary tree method. Despite the fact that this method solved numerous problems at the time, it is now considered to be an overly complex and inefficient process. Nevertheless, for those organisations which could afford it, this new technology provided immense value, allowing them to use the existing data to draw meaningful insights.
Big Players Enter The Market
In 1970, Edgar Codd of IBM published ‘A Relational Model of Data for Large Shared Data Banks’, which served as a catalyst for the development of next-generation relational databases, capable of significantly higher data storage and manipulation capacities. Surprisingly, IBM chose to ignore Codd’s design, instead opting to protect income for their current database systems. It was only after competitors started using the design that IBM followed suit.
By the late 1980s, there was sufficient market demand for the emergence of the inaugural providers of business intelligence solutions. Notable among them were SAP, Siebel, and JD Edwards, who were then referred to as decision support systems (DSS).
At this stage, the primary concern was that these databases had issues with compartmentalization. Because they were so limited in scope, their versatility in terms of applications was significantly reduced. Even small discrepancies, such as one database labelling cities as “OH, NJ, and NY” while another labelled them as “Ohio, New Jersey, and New York,” complicated the ability to cross-reference information.
Despite initial skepticism, more and more examples of effective commercial data utilisation have become evident. One of the most renowned at the time was the Nielsen rating. This marketing technique was used to gauge the audience size of a particular television program at any given moment, utilising the Audimeter. This device was attached to a television and monitored which channel was being viewed.
The Nielsen ratings were highly respected within the TV industry, as they were the most closely monitored national rating survey. However, four times a year, the Nielsen ratings would become unavailable due to what were known as “black weeks”. Without a reliable way of measuring ratings during these periods, television networks would fill their programming schedules with repeats.
In September 1973, Nielsen revolutionised the way television ratings were collected and reported when they introduced the Storage Instantaneous Audimeter (SIA). This cutting-edge technology connected 1,200 residences to the company’s business intelligence computer in Florida, allowing them to provide national ratings in a mere 36 hours, a massive improvement from the one to two weeks it took previously. With the SIA, national ratings became accessible every day of the week, year round, eliminating the need for “dark weeks” and making data far more readily available.
At the end of the 1970s, Larry Ellison and two colleagues created the first commercial incarnation of the Oracle database. This was the first of its kind in the market, bringing a revolutionary new approach to database management – the relational database system. This advancement superseded the existing hierarchical and network databases, providing more powerful search functions and greater flexibility. This groundbreaking technology will have a lasting impact on the development of business intelligence in the coming decades.
BI Needs More Space
The introduction of more cost-effective storage spaces and the development of better databases enabled the emergence of the next generation of business intelligence products. Ralph Kimball and Bill Inmon provided two distinct but complementary approaches to the challenge of consolidating all of an organisation’s data into a single location, in order to facilitate analysis: data warehouses (DW). Bill Inmon is widely regarded as the founder of the data warehouse concept.
Data warehouses are databases that are designed to facilitate the aggregation of large volumes of data from multiple sources (typically other databases). This approach enables a much more comprehensive level of analysis, allowing for the examination of data from disparate sources in an interrelated manner. In spite of this, the complexity and cost of implementation and maintenance of data warehouses was still a significant challenge. The cost associated with the IT personnel necessary to manage and maintain the data warehouse was particularly high.
At the time, top management relied upon the insights generated by Business Intelligence (BI) tools, such as Crystal Reports and Microstrategy, to inform decisions. Additionally, Microsoft Excel (which was released in 1985) was also a popular tool for business intelligence purposes. Consequently, it is clear that business intelligence was a fundamental part of the decision-making toolset.
In 1989, Howard Dresdner of the Gartner Group coined the term “business intelligence” to refer to a set of practices and systems designed to improve corporate decision-making through the use of factual evidence. He saw this concept as an umbrella term that would encompass a range of related topics.
Business Intelligence 1.0
In the 1990s, the cost of data warehouses decreased significantly as the number of competitors in the industry grew, and IT professionals became increasingly knowledgeable about the technology. This period of time has come to be known as the “Business Intelligence 1.0” era.
Data became widely accessible to all members of the company, not just those in executive positions. However, at the time, formulating additional questions was still deemed to be too expensive. Once a question had been established, the answer could be quickly obtained; however, it was limited to that particular request.
In order to reduce the amount of time and effort required, a number of new tools and components were developed to speed up the processing of various queries.
- ETL (Extract, Transform, and Load) was a set of tools that provided an easy way to design the flow of data within a data warehouse. These tools were similar to the functions of a programming language, making the process of constructing a data warehouse more straightforward.
- Online Analytical Processing (OLAP) has enabled analysts to develop a variety of visualisation options for the data they are examining, enabling them to make more precise conclusions based on the available information.
ETL (Extract, Transform, and Load) and OLAP (Online Analytical Processing) technologies remain vital parts of modern business intelligence systems. Both of these technologies are necessary for efficient data analysis, enabling organisations to access and analyse data from multiple sources, as well as to make informed decisions. In addition, ETL and OLAP can help to uncover patterns and trends that would otherwise be difficult to detect. As a result, these technologies have become essential components of any business intelligence system.
This was the period of time in which enterprise resource planning (ERP) technologies began to rapidly expand. These software systems, which are designed to manage and automate company processes, are extremely comprehensive, connecting multiple applications to allow companies to effectively manage their operations. Additionally, ERP systems provide the necessary structured data for data warehouses, which have become a vital component for almost every major organisation around the world in the years since.
In 1995, Microsoft released Windows 95, the first operating system designed for ease of use by the everyday user. This ushered in a new era of personal computing, transforming computers from a niche item to a commonplace household good. The implications of this shift were profound and far-reaching, having a significant impact on the way people accessed, created, and consumed data over the following decades.
BI has been disrupted by the data explosion in the new millennium
By the turn of the century, business intelligence solutions had become an essential component for any medium to large-scale enterprise, as they were seen as a critical factor for sustaining a competitive edge in the market. As such, it had become a “must-have” for any ambitious organisation aiming to stay ahead of the competition.
From the perspective of providers of solutions, the vast array of solutions started to become consolidated into the control of a few major competitors, such as IBM, Microsoft, SAP, and Oracle.
In recent years, the need to keep data warehouses up to date presented a challenge to many businesses. This challenge inspired them to move towards making their data warehouses their single source of truth, meaning that other applications would rely on the data in the warehouse rather than their own stored data, thereby reducing the risk of data incompatibility. The implementation of this strategy was not an easy task, as it involved overcoming several technological obstacles. Nevertheless, the potential benefits of this approach were deemed to be worth the effort, and over the following years, the solutions available on the market adapted to accommodate this new method.
As the volume of data increased and the advantages of Business Intelligence (BI) tools became evident, efforts were focused on improving the speed of access to information and making it easier to obtain. Furthermore, BI tools were designed to be more user-friendly, allowing non-technical users to acquire data and insights without the need for technical assistance.
The advent of social media in the early 2000s gave the general public a platform for their thoughts and opinions to be aired publicly on the internet, thus giving interested organisations the opportunity to collect and assess this data. By 2005, the business world had become increasingly interconnected, leading to the requirement for organisations to have access to real-time data, enabling them to integrate it into their data warehouses as it happened.
In 2005, Google Analytics was launched, providing customers with a free means to analyse website data. This was also the same year when the term “big data” was first used. Roger Magoulas of O’Reilly Media famously defined it as “a huge amount of data that is practically impossible to manage and interpret using conventional business intelligence software.
As the volume of data continued to surge, companies began searching for more efficient ways to cope with the additional storage and processing power required. It was unfeasible to construct bigger and faster computers, so the most practical solution was to utilise multiple machines simultaneously. This gave rise to cloud computing, which is now a widely used technology.
Modern Applications of BI
In recent years, terms such as big data, cloud computing, and data science have become increasingly commonplace. It is difficult to determine which new innovations have had the most profound impact. Despite this, there are some remarkable cases which serve to illustrate the growing potential of modern analytic techniques.
Cookies, advertising, and ad technology
In 2012, The New York Times revealed a remarkable discovery made by Target: the retail giant had been able to detect a high school student’s pregnancy before her parents even knew. By analysing the items the woman had purchased, Target was able to identify a pattern of 25 goods that strongly suggested pregnancy. This information could be used to send targeted coupons to pregnant women, as their purchasing preferences are likely to shift during this period of their lives.
A parent visited a Target in Minneapolis and demanded to speak with the manager after discovering that their daughter had been given discounts on baby items despite being a high school student. The manager, on behalf of the company, offered profuse apologies and contrition. A few days later, the parent contacted Target again to offer an explanation: it turns out that the daughter was expecting a baby in August and the parent had been unaware of her daughter’s condition. The parent expressed regret for their outburst and apologised for any inconvenience caused.
Data analytics is alive and well today, as demonstrated by this story.
Politics
The Obama reelection campaign was widely considered to be a resounding success, largely due to the innovative use of data analytics. Jim Messina, the campaign manager, devised a strategy that focused on the collection of data about voters in order to more effectively encourage registration, ensure that they voted for Obama, and ultimately, motivate them to cast their ballots on election day. This approach was supported by a team of approximately 100 data analysts, who used HP Vertica to create an environment in which their work could be programmed in R and Stata.
Airwolf was a software program that was developed to synergize the efforts of the field and digital teams in order to maximise the outcome of door-to-door campaigns. It was designed to ensure that once a voter was contacted, their interests would be recorded and they would receive personalised emails from local organisers which were based on their most pertinent campaign issues.
Analysts are now equipped with the necessary tools and data to quickly and easily respond to almost any query, regardless of the source of the data. The successful implementation of big data analytics by the Obama campaign has made it an essential component of every political campaign since.
Science
Despite the completion of the Human Genome Project in 2003, numerous issues remained unsolved. Even though the sequence of nucleotide base pairs that make up human DNA had been identified, a more in-depth comprehension of human genetics still required further investigation – and it was an ideal situation in which to use big data. The average human genome consists of roughly 20,000 genes, all composed of millions of base pairs. Merely mapping out a single genome requires a hundred terabytes of data; when sequencing multiple genomes and examining gene interactions, that figure can be multiplied many times, reaching the hundreds of petabytes range in specific scenarios.
Utilising analytics in their 2016 research, scientists at the University of Haifa aimed to investigate the “social character” of genes. This has been a long-standing endeavour for scientists, as they have sought to comprehend the intricate genetic components that are responsible for the emergence of complex medical conditions. This objective has proven to be difficult to accomplish, as it is often the case that genetic manifestations of certain illnesses are caused by the collaborative efforts of multiple genetic markers. Therefore, researchers have had to methodically analyse a full genetic sequence, as well as track the interactions between numerous genes.
Despite the fact that there is still a significant amount of data that needs to be analysed, considerable progress has been made in our understanding and potential treatment of a wide range of genetic conditions, both major and minor.
The Way Forward
We have now reached an era in which technology has become incredibly advanced; Facebook can recognise faces in photographs, Google can anticipate the type of advertisements that would be most suitable for individual profiles, and Netflix can suggest programs based on preferences. This remarkable progress has been made possible through the ability to collect, analyse, and interpret massive amounts of data.
The prevalence of big data continues to increase exponentially. It is estimated that 90% of the data currently available was created in the last two years. This demonstrates the incredible rate of growth in the production of data. Eric Schmidt, the former CEO of Google, famously remarked at the Techonomy conference in 2010 that the entire world produced five exabytes of data from the dawn of civilization to 2003. Remarkably, this same amount is now created every two days.
The handling of large amounts of data still presents numerous challenges, such as data quality, which has been a persistent issue in business intelligence. To make sense of the data collected, there is a need for personnel skilled in analytics, the practice of interpreting data. There are a variety of analytics available today, comprising descriptive analytics, predictive analytics, prescriptive analytics, streaming analytics, and automated analytics. To gain insight from data, analytics employs modern technologies such as artificial intelligence, machine learning, and a variety of statistical models. Mathematics is now a popular field of study due to the rise of analytics.
Business intelligence (BI) solutions are becoming increasingly tailored to meet the specific needs of a range of sectors, including healthcare, law enforcement, and more. These solutions are designed to operate across a variety of platforms, and they utilise a selection of visualisation techniques to provide users with interactive visual interfaces that allow them to reason through data. Moreover, mobile BI is now becoming a reality, allowing users to take advantage of these capabilities regardless of their location.
The convergence of big data, machine learning, and analytics may drastically alter the way we live our lives in the future. For instance, your refrigerator may be able to detect your eating patterns and order groceries accordingly, meaning you won’t need to make a trip to the store. Additionally, your doctor may even be able to predict when you are likely to become ill and contact you before any symptoms arise. Undoubtedly, these technologies will revolutionise how we live and interact with the world.
As we live in the information era, business intelligence is an essential element of our modern society, enabling us to make sense of the vast amounts of data available to us. In recognition of this, many universities and colleges now offer courses in business analytics, allowing students to gain a deep understanding of the subject. The history of business intelligence is still fairly recent, yet it is quickly advancing and becoming ever more complex. We are only just beginning to scratch the surface of the potential of business intelligence and the best is yet to come.
