The Socialization of Business Intelligence

by Eugene A. Asahara
Created: December, 31, 2008; Last Update: December 31, 2008

Overview

This article describes how we can more effectively make the leap from using computers as mere single-use tools to using networked machines as full partners in everyday decision making via the application of Business Intelligence (BI) techniques. With the maturity of the Internet, it is time to consider the ineluctable ramifications of our pervasive dependence on networked computers. Fully integrated and seamlessly melded, these systems are poised for a full scale in human-machine collaboration. Boundless opportunities lie in our ability to monitor, analyze, plan, and take action on these systems.

This article will proceed as follows:

1. Definition of Business Intelligence

2. Evolution of Business Intelligence in the Workplace

3. Expansion of Business Intelligence Beyond the Workplace

a. Phenomenon of Niche Incubation

b. Risk Assessment as Resulting Technology

Definition of Business Intelligence

Business Intelligence is the science and art of developing computer systems that integrate data across an array of functional (single-topic) data silos in a business enterprise, resulting in reporting and analytic capabilities derived from an all-encompassing view of that business. Such a comprehensive view enables users to render more informed, coordinated, and sensible business decisions. BI systems are a bit different from most functional computer systems, which for the most part, simply automate or add value to a specific business process or task as would any other non-computer machine such as a stapler, drill press, or automobile. These are discreet, single-use tools.

The difference between BI systems and functional systems is that BI systems attempt to automate decision making rather than task performing. That is: BI systems assist information workers (IW) in making better decisions. The difficulty in automating these tasks is very evident. Decision making does not deal in blacks and whites, but in weighing infinite variables, and drawing inferences in an effort to extract meaningful conclusions based on dynamic, continually changing data. Such decisions are beyond documented procedures and "best practices".

Emergence and Evolution of BI

Figure 1 – The Intelligence of a Business. Part human, part machine.

Figure 1 depicts the "intelligence of a business" and the impact improving information technology has on the magnitude of business. That magnitude is represented by the increasing breadth of the series of blue and green bars encoded as follows:

· The blue color depicts the proportion of the "intelligence of the business" that is "human intelligence". The blue color is on top because human intelligence is the primary intelligence.

· The green color depicts the proportion of the "intelligence of the business that is "machine intelligence". Further, the two shades of green depict the part of the machine intelligence that is data (dark green) and the part that is rules (light green).

· The width of the bar illustrates the magnitude of a business. For example, the leftmost bar shows that when a business is composed of mostly human intelligence, the magnitude of business is rather small.

What you see is that the proportion of the intelligence of a business that is human shrinks as information technology improves. The increasing proportion of machine intelligence to human intelligence in the "intelligence of a business" affords humans the ability to make more complex decisions than they otherwise could.

The first bar, at the far left, shows the intelligence of a business only decades ago (or today in under-developed countries) functioning primarily on human intelligence. The business rules were confined to human thought and action, supplemented by paper and perhaps primitive adding machines. Paper stored primarily data (logs, transactions), but could also record knowledge about a business.

The second bar depicts the introduction of early computers from the 1960s through the mid-1990s. With easy storage and retrieval of relatively large amounts of data, the intelligence of a business was significantly enhanced. Business rules could also be encoded into these machines as computer programs, but the limitations of computer technology meant that these programs were rather simplistic compared to software of today such as Customer Relationship Management (CRM) or Forecasting/Planning software.

The third bar brings us to where we are today, thanks in a large part to BI. BI applications pioneered the integration of data across an enterprise, the storage and retrieval of massive amounts of historic data, and the art of intuitive visualization of this ocean of data. We now see the beginning of the encoding of business rules.

The first perceptible appearance of light green (representing RULES) in this bar is a benchmark event. It shows that encoded business rules now have a noticeable impact on the intelligence of a business. But the significant presence of rules is relatively new to BI. They have emerged through an evolutionary progression fueled by BI-based technologies such as Data Mining (at least as is presented in SQL Server Analysis Services) and Performance Management.

Data Mining is indeed about the discovery of rules. Such rules are discovered through running oceans of data through sophisticated algorithms. Rules include among other things associating seemingly disparate events, clustering objects by their similarity of properties, and reverse-engineering decision paths.

Performance Management (PM) introduces rules as well, but differs from data mining, as it involves the manual encoding of business rules in the form of composing formulas and defining thresholds. Such formulas include key performance indicator (KPI) values, goals, trends, and statuses as well as formulas determining values such as profit or taxes, which can be subject to "IF-THEN" sort of logic.

Both Data Mining and PM are rapidly growing applications of the infrastructure laid out by BI. As they grow, the light green portion of the third bar in Figure 1 should increase, probably relieving pressure and accentuating human intelligence (the blue portion). At some point, that increase of rules will lead to the fourth bar where the full partnership of human and machine intelligence, Decision Advisory, enables the feasible and effective management of widely-scoped for businesses.

Decision Advisory

Line graphs and bar and pie charts (and other BI visualization tools such as Cube Browsers) are the visible tip of a huge iceberg called a BI system. Such visualizations effectively process a mountain of raw data, the hidden mass of the iceberg, into relatively few summarized chunks of information. (It’s easier to see from a line graph that sales are trending upwards versus looking at the massive list of sales transactions.) However, those visualizations still just present inputs to support our decision-making we humans do, without computer assistance.

BI For Our Daily Lives

The next step for BI would be to begin offering suggestions to the less complex, less risky decisions in our daily lives. Or at least provide a short list of warnings or opportunities that we may not see. So in the case of online weather forecasting sites, instead of merely providing information from which I should infer there is a dangerous condition such as black ice, it would actually tell me such things that a human, intelligent, skilled weatherperson would. (It actually does, but I suspect that at this time it is a human weatherman writing the "Weather Advisory".)

Such features are already becoming common our daily lives, but are rife with false positives (reams of irrelevant information or annoying pop-up warnings) or false negatives (overlooking something we should be told). Addressing the issue of false negatives and false positives is the key to successfully expanding upon these aspects of BI in our daily lives.

I like to call this "Decision Advisory". The difference between "Decision Advisory" and the more familiar "Decision Support" (DSS – sometimes used interchangeably with BI and much more common in the 1990s) is that the former actually makes recommendations as a human assistant or consultant would. The former simply integrates and presents data in a format conducive to making decisions while still leaving the bulk of the actual decision making to us without further computer input.

It’s not that BI/DSS purposefully draws the line at providing information and not advisory. The limitation is due to the fact that the mere integration of data (the foundation of BI) isn’t enough to go that next step towards machine generated inferences. Crossing the line from providing information to advisory depends on the level of sophistication of the system being deployed to summarize the data. The summarization of a large amount of data into a nugget of information is the key to effective analysis. At this time, users of BI/DSS are pretty much limited to simple summarizations of sets of values such as Sum (simply adding the values), max (the largest value of a set), min (the smallest value of a set), average, and standard deviation.

An example of a sophisticated summarization would be evaluating a series of numbers with the ability to draw inferences such as "This represents an upward trend" or "This represents erratic behavior". I discuss this in the article, Data to Information to Data to Information. Another example would be the clustering of many customer attributes resulting in similarities that could be labeled "Soccer Mom", "Nascar Dad", or "Gen-Xer".

Further, there are two major characteristics of sophisticated summarizations most BI systems of today do not directly support. First, summarizations should be hierarchical; meaning summarizations could provide inputs to other summarizations. Second, the rules of summarizations should be (but don’t need to be) completely transparent to other summarizations readable from rule to rule. SCL is such a technology that satisfies both characteristics. These two characteristics are the basis for Rule Integration, which I will address at the end of this article.

Socialization of BI

As BI applications have become mainstream in the enterprise environment, people have become accustomed to using software in a capacity beyond that of a mere simple tool. BI applications are indispensible for complex problem solving within the framework of a company’s business strategy. When a new phenomenon is successfully assimilated into a culture, it is "socialized". The socialization process of BI at our workplace leads to two things that will expand this new ability to our private lives.

Of course, the first is that those of us who are exposed to BI at work are accustomed to the idea of using computers to help us make decisions. Lately in our private lives, we’re beginning to use software applications to actually advise us in our decisions. (For example, weather forecasting websites integrate data from many sources and formats that all-encompassing view into terms I normally use to decide how to plan the day or determine a commute. They also artfully offer the ability for me to further analyze the weather from different perspectives.)

Second, the research that went into the maturing of BI over the past decade provided a sandbox (an experimental environment) for software vendors to fully forge the techniques assisting in decision making that we can eventually apply to the consumer world. For those not exposed to BI at work, the ability to introduce the notion of using a computer to support our decision making is possible in large part to the maturity of BI technologies.

Niche Incubation

The rigidity and discipline of the business world combined with greater competitive pressures due to the information age forced the emergence and incubation of BI technologies. The consequences of making poor business decisions by executives could mean severe crippling or death for the corporation. However, such expensive and relatively complex software is probably overkill for our home lives. Decisions in our home lives are not so complex and the consequences not usually so grave that we would think of inventing something as complex and unwieldy as a BI system. Advice from friends and a few Google searches will do pretty well.

Many such software applications we commonly use in our private lives have been incubated under niche circumstances. Accounting software is a good example. The scale of transactions in a large enterprise made the expensive and sometimes obtuse hardware, software, and technicians worth the trouble. Smaller companies and households could get by with paper and a good accountant (or just paper for most households). Eventually the improved state of the components of accounting software were such that the trouble was minimized to a point where it added net value for smaller companies and millions of households (the trouble of learning, implementing, and maintaining the software was worth the benefits).

A decade ago, Business Intelligence was something only large companies could afford. However, in 1998, "BI for the masses" was born with the release of the relatively inexpensive and relatively easy-to-use SQL Server 7.0 which included the first version of Analysis Services. Now, "small" (generally fewer than 100 employees) to medium sized businesses could feasibly implement BI systems. The "mom and pop" businesses are still simple enough where the "BI" can be contained in the head of the entrepreneur.

When I refer to a niche, this is different from "early adopter". Early adopters are the first to embrace a new, but maybe not fully mature, kind of clunky technology that is thrown out into the market. The product and concepts are already rather concrete, but may improve over time in terms of quality, usability, and price. The first people to get onto an online Weather service were the early adopters of this specific application of decision support in the consumer world.

MSN Weather again provides a great foundation for an example of an early adopter. It’s interesting to see how my neighbors all now know when the snow has really stopped and it is worthwhile going out to shovel snow. I think many of them probably wait for me and a couple others in our neighborhood. We are the relatively early adopters to actively using the integrated data that is presented in a way conducive to decision-making by AccuWeather or MSN Weather to decide that the snow is indeed over. Eventually, most people will learn to use these services.

Niche incubation is the period that a technology emerges in a specialized environment and is further developed because it is worthwhile to the practitioners of the niche. Such technologies are usually cutting edge, immature and expensive to own and operate. Only those who really need it will put up this that trouble.

Eventually, niche technologies mature enough where they are well-defined, mature, and easy enough to own and operate. At that point, environments where the benefits of the technology are not so integral can adopt the technology perhaps as a competitive differentiator. The first adopters would probably be less intense versions of the specialized niche environments; for example, healthcare management software first utilized by hospitals, then clinics, then sole-practitioner medical practices.

A personal example is an application I wrote in my medical practice management software, PreceDent, that I called "Take Action Email" way back during the dot-com days. Take Action Email was a component that tracked issues in dental offices with the same discipline that software bugs are tracked. Because of the complexity of software and the cutting-edge nature of software development it is painfully buggy. The relatively large number of bugs that pop up during the testing period required a system to help track the progress of those bugs from discovery to resolution.

When bug tracking software and other supporting technologies became easy enough to use, issue-tracking could be applied to businesses where it wasn't core to operations, but would certainly add significant value. A great example is applying this focus to customer support as a major component of Customer Relationship Management (CRM) software.

It’s also possible that once the niche technology leaves the incubation environment, it will be utilized and evolve from there in unexpected ways. This is a kind of exaptation; where an existing technology or feature developed for a specific purpose happens to be useful in an unexpected manner.

That is my expectation for BI once it leaves the niche environment of the business world. I don’t expect people in their private lives to check on their Performance Dashboard every few hours or master relatively complex software BI packages. For most people, life outside of work covers a broader range of activities and one has more freedom to choose. I suspect "BI in the private world" will be manifest as something embedded in a wide array of software applications than as a distinct "BI System" in the business world.

Risk Assessment: The Next BI-Related Technology to Leave Niche Incubation

The killer app for Business Intelligence (or a killer app for software in general for that matter) is: Tell me about something bad before it happens, without throwing too many red herrings or crying wolf too much. And, provide justification for this prediction.

(The next step is, "Then tell me what to do about it." That is the holy grail of Artificial Intelligence which has lead to terrible disappointments in the 1950s and the 1980s.)

With that in mind, a BI-related application that I predict will explode beyond the confines of its niche incubation environment with profound effects is Risk Assessment (RA). RA is integral to industries where operations involve "life and death" issues (life and death for the business anyway) that are based on complicated predictions (the "genes" of RA), margins are small, liabilities are high, and the event we’re trying to predict comes so fast there isn’t time to react. One transaction can wipe out all profits for years to come or even put the company out of business.

Banking and insurance are two "traditional" examples of industries where accurate prediction is vital to the success of the business. (I choose to use the term "traditional" since the banking industry is famously poor in its risk assessment as of late.)

Characteristic of any niche technology, RA is more trouble than it’s worth for entities for which the niche technology isn’t integral to operations. To recap from the previous section, it’s immature and expensive to own and operate. It requires complex, usually custom equipment, highly-skilled, expensive, and relatively rare practitioners, and patterns and practices are at a state not much beyond being "passed on by oral tradition".

All businesses must make predictions such as predictions on inventory levels and shifting customer tastes, but for most businesses, a fair ballpark figure will suffice. Every business represents a set of bets that have been placed on issues ranging from where the customers will be to when a particular practice will be restricted by the government. However, for most businesses, most decisions do have the luxury of some margin of error. RA probably would add value to most businesses, but perhaps not net value, when considering the trouble of implementation.

To illustrate, a retail business for example will operate better if predictions of inventory levels are at least reasonable. However, the efficiency of the retail business will improve along with the improvement of the accuracy of inventory levels predictions. Money free from excess inventory can be used elsewhere, storage costs will be minimized, and the business avoids the risk of being stuck with a warehouse of products going out of style or perishing. A retail business can withstand and stumble along with a pretty significant margin of error. It also can usually employ some tactic to use perpetually inaccurate predictions to its advantage (sales to bring in more customers, a reputation for having a large selection, etc).

But operating with less efficiency due to less accurate inventory level predictions is a different story from the risk of an insurance company facing bankruptcy if the prediction of the number of deaths related to a life insurance product is significantly off. For a retail business, less than perfect inventory level predictions will not result in killer blows, just consequences that can probably be remedied over time.

Another major reason a business would also not need to implement RA is that it’s small enough where the complexities can be comfortably contained in a few or even one human brain. A formalized approach to assessing risk would be overkill in such situations. Such a business is like the left-most bar in Figure 1 where the bulk of the intelligence of the business is human.

Like Performance Management (the rage in mainstream BI at the time of this writing), RA builds on the data integration and analytic infrastructure provided by BI. However, there is a greater utilization of statistics-based data mining algorithms surfacing correlations that sheds light on relationships that would be incorporated into the centerpiece of RA – "cause and effect" models.

Note: I take liberties with the term "cause and effect". The correlations surfaced by data mining algorithms do not necessary discover cause and effect relationships in the strictest sense. For example, a cause and effect relationship is flipping a light switch turns a light off or on. A correlation is obese people are more prone to heart disease. But adding such relationships to a "cause and effect" graph still adds value to the model as long as the user of the model understands the correlations only represent a statistical relationship.

Cause and effect models are the core products of knowledge mapping. They are simply a web of relationships such as "eating too much leads to obesity" and "obesity leads to heart disease". The problem is that they can quickly become unwieldy in their complexity as relationships are discovered and added, rendering them nearly impossible to effectively author and read. The simple example I just presented demonstrates this as eating too much doesn’t always lead to obesity. The person in question may be an athlete or the person may be painfully underweight to begin with.

Currently, the notion of cause and effect models in the Microsoft BI Stack is limited. Cause and effect models are an advanced concept in the current "BI for the masses" philosophy of the Microsoft BI Stack. The presence of these models are limited to the relatively simple Strategy Maps (authored with Visio) of PerformancePoint Server and the dependency networks of Analysis Services Data Mining.

At this time, despite how easy SQL Server 2005’s data mining capability is to employ, data mining is severely under-utilized. It is beginning to be used to generate trends and find associations, but concern over the quality of the prediction (the risk that the prediction is wrong) and what to do about it is beyond where most BI implementations care to go. It’s easier to stick to analyzing the past (current BI) than to venture into predicting the future.

What is exciting about RA is that such a capability is fundamentally the basis for human intelligence. That is, we are good at predicting an outcome before we actually commit to an action. Since we humans are self-aware, each a species of one, we each are aware of our own extinction. Thus, we strive for decision quality that preserves us individually and not just for a level where the species will go on. For other species, losing a few members or even purposefully sacrificing a few members is fine, as long as the species survives. Therefore the proliferation of this RA into the general population holds significant possibilities.

The Limitation of Predictive Analytics

Currently, the term, "Predictive Analytics", is gaining traction in the BI world. Although I’m very happy about this, Predictive Analytics isn’t sufficient for trustworthy decision making by individuals in relation to "Risk Assessment for the Masses". Predictive Analytics is completely based on statistics which are presented as "data mining models" such as decision trees, association rules, and clusters. Statistically-based predictions are made from this model for things such as targeting customers as likely buyers of a product.

Such rules work very well when the underlying currents (foundational environments such as the economic and political conditions) remain pretty constant … and as long as "highly unlikely" and perhaps unknown, unexpected outlier event do not strike. Such highly unlikely events that may never have crossed our minds are usually not reflected in predictive analytics models as they involve outlier data (the two ends of the bell curve) that are usually truncated in order to "normalize" data.

Unfortunately, these highly unlikely events are more likely than we think. And the outcome of such events are usually very profound. With over six billion people in the world, terms such as "one in a million" chance have lost its scope. Additionally, the term, "the strength of the weak link", suggests that profound change usually happens via weak relationships tying two disparate entities resulting in sweeping changes for one or both entities.

Two great books on this subject are:

· The Black Swan: The Impact of the Highly Improbable, by Nassim Nicholas Taleb

· Linked, by Albert-Laszlo Barabasi

The reason I bring up the matter of the limitation of predictive analytics is that we need another technology in order to enable real automated decision making in our everyday lives. That technology is Rule Integration.

Rule Integration

When the utilization of Risk Assessment explodes outside of its niche, a new phenomenon will develop that will seem very familiar to old BI hacks. That is, the integration of rules from "rule silos", analogous to the integration of data from functional data silos. The development of "Rule Integration" is crucial to paving the path for RA in the world at large.

We make decisions from predictions (and other decisions) and we make predictions on rules (and other decisions and predictions). Every decision is the result of the processing of rules. But what are rules? Just about every procedure we can think of. There are very many things that are rules that we normally don’t think of as rules and they currently exist in software applications or other machines that may share data, but don’t really have access to the rules in a machine.

This is similar to how a conversation with someone involves an exchange of words, but neither party can really get into the neurons of the other to determine the rules by which their words were chosen. So in order to smooth out the conversation, we each try to express "where we’re coming from" or "the context behind our actions".

Intelligence is about determining which rules are applicable to a certain situation for which a decision must be made. In order to enable Risk Assessment for the masses, we must build an infrastructure for the integration of rules.

The article, Rule Integration - The Step After Data Integration, is essentially Part 2 of this article.