Soft-Coded Logic Software Development Kit Beta 1.0 (SCL SDK) – Putting the “I” Back into BI

 Originally posted on http://blogs.msdn.com/eugenea on May 29, 2006 12:41pm

After an investment of a year of my personal “fun development time” into SCL, I’m releasing a beta of my SCL Software Development Kit Beta 1.0 for the .NET development world to try.

 

I’ve posted a couple of blogs and articles on SCL on this MSDN blog site and www.softcodedlogic.com. (Please read the articles I have posted there.) However, the most significant text on SCL is in my upcoming article, “SCL Web Services Model, Part II of III”. I had planned to post this article much sooner, but it is turning out to be much more ambitious than I had first thought. I hope to release this article around the middle of July, 2006; about a month from now. At this time, however, I’d like to offer a short preview of it. Using three illustrations, I summarize the where, what and why, and how of my vision.

 

So, what do I mean by “Putting the ‘I’ back into BI”? Explaining my meaning begins with the caution that I don’t believe there is a widespread agreement on what BI does and what it includes. For me, I begin to attempt a definition from this point of view: Businesses are like any creature out there “in the jungle” with a will to survive. Successful creatures execute a way to make a living (gather enough resources to keep them going), outperform their competitors, adapt when the rules that are beyond their control change, and defend themselves against their enemies (whether competitor or predator).

 

From that point of view, business intelligence is the intelligence of businesses just as our human intelligence is the intelligence of humans and the intelligence of cats is the intelligence of cats. Is that definition expecting too much from BI? I don’t think so. Business intelligence has always been there. The issue is what part or how much of the BI “system” is human and what part or how much is machine? At one time and still in many parts of the world, it was or is pretty much 100% human. Now, it’s part human (“CXX” folks and MBA types) and part machine (data warehouses, OLAP and data mining).

 

Also from that point of view, our OLTP (CRM, Inventory, HR, Accounting, etc.) systems are the systems that support us in making our living. The process of producing and delivering our product and maintaining the equipment is in itself predictable. Define your product and develop and optimize the processes and models for producing and shipping it. A creature gathers food, eats it, digests it, and expels the waste. What isn’t predictable is when and how forces that threaten the model will arise and how to address it.

 

We use our BI tools (ETL, data warehouse, OLAP, and data mining) to build models for our business. A model is a web of cause and effects. For example, late deliveries lead to dissatisfied customers, dissatisfied customers leads to fewer referrals, fewer referrals leads lower profits, lower profits leads to cutting corners, cutting corners leads to a bigger risk of late deliveries. Every decision we (we humans, other creatures, and businesses) make is based upon a model  built in our minds of a given situation. If we have never encountered the situation before, we employ what appears to be a reasonable facsimile (metaphor).

 

Using our data mining application, we find causes that correlate to positive effects. This is added to our model. We identify groups that share characteristics with another group that is beneficial towards our purpose. We can then "exploit" that group. We then create KPIs to monitor each of those aspects to the model.

 

Changes to these KPI values detected by our monitor tools are reported and treated no differently than a burglar alarm. A force is sent to attempt to neutralize the threat to the model. There is much invested in the model and it must be protected. Your BI tools will help you to plan the most optimized attack using every leverage point at your disposal. However, if the threat cannot be neutralized, the model must successfully adapt to the new conditions, or the business will die. Still, most businesses must avoid changing the model as it is costly, painful (there’s usually growing pains associated with change), and risky.

 

To summarize, business intelligence monitors threats to the business model, decides on how to protect the model, and formulates any change to the model. What I like about looking at BI from this point of view, looking at the human and machine parts holistically, is that it prevents BI from degenerating into "glorified reporting". This is where retrieving and reviewing important KPIs become rote.

 

Thoughtless dependence on KPIs is the sort of thing your competitor wants. In martial arts (or most sports), predictability is one of the best things your opponent can give you.  If your competitor knows you're obsessed with a set of KPIs, your competitor knows you're blind elsewhere. And, as many information workers have discovered, with enough effort it's possible to make a KPI look good when it really isn't. (The multiple ways there are to make something look like something else is a very interesting BI subject I will elaborate on in a future article.)

 

The machine part of a business’ intelligence has been growing with ETL, data warehouses, and OLAP. However, these machine aspects are primarily “sensors” of the BI system. It does allow a CEO to quickly see things she’s never seen before such as factual, immediate lists of her top 50 customers or profit trends, just as night-vision goggles now allow soldiers to see fairly clearly in the dark.

 

The problem now is that businesses are becoming so big and so complex that there is a need for the machine part of the BI system to assist in areas beyond sensory. SCL offers a starting point. It is a development platform for flexibly and robustly encoding rules into our business models at many entry points: .NET applications, the SQL Server relational engine using CLR integration, Analysis Services, Integration Services using custom transformation sources and script components and tasks, and Web Services.

 

SCL is based on the AI language Prolog (which stands for “Programming in Logic”; inference.). However, I’ve added features relevant to current software architecture such as reaching out to other data sources (including other SCL databases) and integrating with .NET code and SQL Server 2005 (primarily by exposing SCL as an ADO.NET provider). And, I’ve stripped out features that are at this time irrelevant in order to simplify the language; much like how C# stripped out pointers (or at least reduced its presence) that are prominent in C resulting is a much simpler language. The result is a language that effectively augments the Microsoft BI platform with broad inference capability.

 

 

Figure 1 illustrates what I call the “AI Stack,” which shows how SCL enhances the current data processing, mining, and warehousing scheme. The stack divides the flow of data into two distinct phases, delimited by the gray dash line. Below the line is simple gathering and aggregating of data—just the facts.

 

Above the gray dash line statistics come into play. Here data mining algorithms employ clustering or associations to “discover” facts, such as creating labels for objects like “Soccer Mom” or “GenXer.” At the very top of the stack is AI, which included SCL. SCL is a mechanism for inferences to be drawn and deductions to be made from the facts gathered and aggregated below.

 

Figure 1 – What does SCL Offer Beyond Data Mining?

 

Figure 2 illustrates what I hope to accomplish with SCL: Creating computing devices that are robust enough to be treated as if they were human. (We are on our way there: We already interact with traffic lights as if a real policeman were standing there.) AUTOMATED TELLER MACHINES

 

The current state of our relationship to computers is rather one sided. For the most part, humans have had to adapt their behavior to derive the maximum benefit from computers. Using GUIs and Web Services of significant substance still requires a lot of training, accommodation, and compromise on the part of humans. At present, humans are required to adapt to the constraints by which computers process data. I propose a solution where computers execute actions based on robust and flexible sets of facts/conditions—where machines behave in a more human manner. This solution can be achieved by architecting off SCL.

 

Figure 2 – The relationship between humans and computers.

 

 

Figure 3 represents a picture of the how SCL produces robust recognition. It roughly shows how actions are triggered. The important concept at this point is that the picture of how to recognize a dog can be encoded as SCL (Prolog).

 

There are multiple ways to recognize something. In Prolog, each “rule” is a recognition. It is this mechanism that leads to inference:

 

How do we recognize that an object we are seeing named Lassie is a dog? If we can see and hear, one way is to reason that:

 

Lassie has four legs and barks.

Dogs have four legs and bark.

Therefore, Lassie is a dog.

 

The Prolog for this inference looks like this:

 

bark(Lassie)

legs(Lassie,4).

dog(@name) :- legs(@name,4),bark(@name).

 

What if the observer is hearing-impaired? Can we still infer that we are seeing a dog?

 

/* For someone that cannot hear.

bark(@name) :- looksaggresive(@name),openedmouth(@name).

/* For someone that can hear.

bark(@name) :- soundslike(@name,woof),louderthan(60,decibels).

 

 

Figure 3 – Robust recognition of a dog.

 

So, download the SCL SDK Beta 1.0, go through the 55+ page tutorial I provide, and let me hear from you. Your development environment requires Windows XP Pro or higher, Visual Studio 2005, and SQL Server 2005 Developer Edition (see the included document for more installation details).

 

A big caveat, though: I’m planning a tighter refresh for Beta 1.0 that I plan to release within a few weeks. I've been extremely busy with BI engagements and haven't had enough time to really harden everything, including the documentation. I chose to post this now as there are a few people out there that's been waiting for a long time. Hopefully, this is good enough to get started. If not, please try the refresh in a few weeks. Related to that, remember, this is just a beta of the SDK, a tutorial to let interested folks play with it.

 

Update: July 4, 2009

The SCL SDK is currently in the middle of a major refresh. The Beta 1.0 SDK has been removed, but the Beta 1.0 documentation is now available for viewing.