How individual customers feel about the brands they choose to do businesses with can be best described as their brand temperature, or mood. In an online environment where much or all of the relationship is defined digitally, brand attitude towards the online channel is often called digital mood.

Digital mood is important since it affects a customers’ propensity to buy and ultimately their loyalty. It is a fickle thing, changing dynamically based on the latest experience, but influenced by the customers unique past experience with the brand as well. 
It’s important because a customer’s attitude at any point in time needs to be taken into account in the tone of any communication with the customer.
Here’s an example: a prospect is on line applying for a new account. After several steps they abandon the process, an all too common problem impacting as many as 4 in 5 online applications. If you’ve designed your new account opening process well, then you’ve captured both the email details early on and key value indicators which allow you to classify the potential value of the client.

Customer experience is what you’d say about a company or brand anecdotally at a dinner party. It’s tied in heavily with your emotional state towards a brand at a given point in time.

It is fundamentally hard to measure, because it has been traditionally difficult to capture good quality and reliable data. Customer satisfaction surveys, if done periodically only touch a moment in time, and get low response rates. More recently web based event driven customer satisfaction surveys have been proliferating, and these produce more interesting data, but the sheer quantity threatens to overwhelm customers.

Of course the most useful feedback comes from verbatim comments, not multiple choice selections and these are difficult to analyze, though significant progress is being made in this area by some vendors.

This difficult of getting good quality experience data is why we rely on handling complaints from customers reactively, even though it is well documented that only a tiny proportion of customers complain. The latest figures I’ve seen suggest that as few as 1 in 100 dissatisfied customers actually complain.

For example, I stayed last year at the New Yorker hotel in Midtown Manhattan. I had an appalling experience - I woke with more than 50 bed bug bites. I didn’t complain, but I did write a slamming review on Expedia.com (as have quite a few others).

It felt great – even like revenge! The fact that I put the name of the hotel in this blog means that it will be picked up by search engines and prospective visitors will be able to view what I thought about my experience.

This is one example how the online channel fundamentally changes the way that we can now measure (and therefore potentially act upon) individual customer experiences, since customers now have a voice in the blogosphere, and this data can be captured and analysed.
If your business is to provide a service over the internet, however, then you have a richness of experience data not found in other channels. Businesses such as Expedia, online banking operations, online retailers or Software as a Service (SaaS) providers can precisely track the experience of customers, and measure how effective their service is.

Measuring it enables you to prioritize resources, fix problems and to identify which customers were affected.

Measuring it is one thing, acting upon it is another, since you effectively have to track experience at an individual level, and be able to act in real time – after all there’s no point in trying to take remedial action days or weeks after your customer experiences a negative or positive event.
When thinking about acting on negative customer experiences, there are two distinct phases: acting while they are still on the site itself (e.g. reshaping network traffic, or changing a business process), and acting once they have departed (e.g. sending an email or opening a case in a CRM system). Both of these types of actions must be automated in some fashion in order that the action can become a repeatable and robust process.
In many cases customer value is a critical component here. The kind of action that you take is very much affected by both the value of the customer, and well as the actual process involved when a negative experience was encountered. For example, a high value customer experiences a ‘page not found’ error in the middle of applying for an online savings account. Clearly the action you’re going to take is probably phone based, whereas your reaction to an unusually slow response from an obscure and rarely visited part of the site is going to be different.

It’s been a while since I’ve been able to blog. A continuous travel schedule as I spend more time in the US than in Europe has kept me more than a little busy!

However, the last two weeks have seen huge changes in the BI industry. Now that both Business Objects and Cognos are being consumed (by SAP and IBM respectively), attention is inevitably turning to the next wave of independent BI vendors. Some are speculating that BI is not a category, but in fact an extension of the database or application (see Ephraim Schwartz at Infoworld for example).

While it certainly can be considered that BI can be used as part of the application, there are several really solid reasons why BI will remain independent as well as part of the application / database stack. Since I was fielding calls from journalists asking for my take on this left and right, I thought that I’d try and summarize as follows. Please chip in and comment if I’ve missed anything.  Five key reasons for independence in BI 1.      Heterogeneous data access
We live in a heterogeneous world, and will continue to do so for years to come. The average corporation has more than 20 different types of databases, so any BI tool needs to support a huge range of different ways of accessing data. Most independent BI vendors spend up to 50% of their R&D dollars supporting hundreds of different types of connectivity. This is significant expenditure, and it will inevitably be cut once the independents are incorporated into monolithic vendors. Engineering focus will inevitably shift to closer integration with other products within the business in order to drive product synergies and enable customers to be cross sold more products.  

2.      Switzerland; Thou shalt not become beholden to one mega vendor
When you need support for a heterogeneous join across DB2, Oracle and SAP data sources who you gonna call? Will Oracle support you accessing a DB2 database as well as an independent? Will SAP do what ever it takes to fix your problem when the data is in DB2 or Oracle? This is the powerful emotional argument that has sold thousands of CIO’s on a future with independent BI. You have a heterogeneous environment spanning SAP, Oracle and others and a myriad of different database types – when push comes to shove, is you BI vendor going to support you to the ends required given their competitive tensions with the other parties involved?  3.      Standardization on one environment
The analysts, led by Gartner, have been telling business for years to reduce the numbers of BI tools that they use. It’s obvious why: fewer tools = more consistent basis for decision making; and of course lower cost. But the absence of a leading independent vendor makes this hard. Assuming the Cognos acquisition goes through, which is likely, SAS is to some extent the ‘last man standing.’ However, standardizing on SAS for BI would be a bizarre move – they have a proprietary feel and lack the breadth and usability of either Cognos or Business Objects. Better to consider Qliktech or one of the other young upstarts.

4.      Product roadmap
In a single stroke, the product roadmap that your independent BI vendor committed to has gone. Hopefully you weren’t depending on any of these new features, but if past acquisitions are anything to go by, one of the first things to get de-committed is the product roadmap. It’s obvious why – everything changes with an acquisition; management, strategy, priorities, customer relationships, and of course the roadmap. 

5.      Innovation
A heavily concentrated software industry is absolutely not in the interests of innovation. Development priorities get focused on integration with the myriad of different products and technologies of the parent, (often through acquisition as well) not on innovation of the products themselves. As a result, don’t look to your newly acquired BI vendor to be innovating any time soon. Most if not all of the senior management will almost certainly leave once their earn out conditions have been met, leading the BI business lacking both strategic direction as well as tactical steering. To understand where BI is headed, look to the young pretenders – this is where innovation is to be found, and ultimately where the next generation is being born.
                                                                                                                                        

Sandy Kemsley’s blog on business process management on eBizQ reports on a Presentation given by Gartner Analyst Bill Gassman hilighting the difference between BI and BAM. She comments that

“So what’s the difference between BI and BAM? According to Gassman, BI is used for insight and planning, and is based on historical — rather than real-time — data. BAM is event driven, and issues alerts when events occur. Personally, I think that there’s a spectrum between his definitions of BI and BAM, and it’s not clear to me that it’s a useful distinction; in many cases, data is trickle-fed from operational systems to BI systems so that the data is near-real-time, allowing dashboards to be driven directly from the BI system. True, traditional BI tools will typically see update intervals more like 15 minutes than the near-real-time event alerts that you’ll find in BAM, but that’s not a problem in some cases.” 

The problem with definitions such as “BI” and “BAM” is that real life is inevitably more grey and less black and white. Bill does a good job at classifying vendors to provide separation and distinction between vendors, but perhaps I can add a twist to the definitions which may help.

In the context of BPM, it’s useful to think about an additional dimension which has not been discussed here: when do you need to do the analysis: design time vs run time?

Rather than trying to look at what is real time and what is ‘near real time’ which is think is less useful, I always suggest that a good starting point in optimizing processes is to consider when the optimization needs to be done. Is the requirement to do it while the process is running on a continuous basis, or are you seeking to understand how well your processes executed so that future instances can be tuned?

Querying process logs to produce reports and dashboards can help business analysts to tune processes, but is an “after the fact” (design time) activity. This is traditional BI applied in a BPM context and there are several notable examples of BI vendors collaborating with BPM vendors to facilitate this.

If you ask users what they think BAM is, then they will describe a process oriented dashboard that can give visibility into processes executing in real time. There is real value for process analysts to get visibility of bottlenecks of processes modelled in BPM tools.

When you want to optimize processes on a continuous basis, based upon the process instance in flight, then this needs to be done automatically, and in real time. For example, you might want to reconcile both supply and demand in real time in order to adjust service levels or adjust prices dynamically.

A real time dashboard alone won’t achieve this goal, and part (or all) of the processes and the data needed to do this are often not modelled in a BPM tool. So these types of systems tend to be data centric (not necessarily BPM centric), and event driven.

The characteristics also are of more sophisticated analytics which are automating an analysis process to detect exceptions, calculate risks and forecast, say, delivery times. In order to do this often a significant quantity of both real time and historical data is required, but processing this in real time is beyond the capability of a BI tool or even database. The results of these analytics are used to then initiate processes, or provide information to modelled processes as an analytic process step. Oh, and of course this all has to happen in real time.

Building ‘business intelligence’ (in its broadest definition) into processes is clearly an obvious way of making your processes smarter. Is it 5th generation BAM?  Well maybe, but the use of deep analytcs and historical data has characteristics reminiscent of BI, but applied in an event driven, closed loop way. A long way from traditional BI.

JBoss World Berlin provided me some fresh insight into the challenges associated with providing Business Intelligence in a Service Oriented Architecture. The show itself was vibrant, with the JBoss announcement of their new Enterprise Service Bus (ESB) taking centre stage. The conference provided yet more evidence that getting business insight from your SOA is an idea which is becoming broadly accepted.

History shows us that wherever you build applications, or deploy packaged apps, there is always a need for BI. It’s no different in SOA. Traditional query based BI doesn’t fit well in SOA – but that doesn’t mean that you’re not going to need it. If fact with every SOA project it’s a matter of time before this requirement hits every project.

Here’s a specific scenario for one customer we discussed this with: The IT team from a leading company built and deployed a mission critical application using SOA. Within a matter of a few days after the go-live, business operations were asking for information about the transactions processed. As often happens, Business Intelligence hadn’t been mandated as a requirement for the project, and as a result it wasn’t built in. Unfortunately since the application had been distributed across multiple servers, multiple log files have to be extracted, transformed and loaded into a database. Because of the data volumes involved, the queries perform appallingly, making the system practically unusable for the business.

Of course a much simpler and more elegant solution would have been to build visibility into the project from day one. Given the nature of the project, and a Service Oriented Architecture, it would have been much simpler to use event driven BI to analyze the message traffic in flight. In fact that’s what this company looks set to do.

It was only when I was in discussion with a customer this week that I realized that most people think that primary benefit of real time Business Intelligence is its ability to analyze data in real time. And of course, they’d be right, but only partially. Another massive advantage is the way that data is analyzed, which is fundamentally different from traditional techniques since data is analyzed as a stream. As a consequence stream analytics enable new classes of analysis applications that were not previously possible. It is particularly relevant where data volumes are high.

Traditional data analysis relies on a person analyzing data in batches. First the data warehouse is updated, then queries can be run, and an analyst (or in some cases a business user) can then begin the search for insight. The search starts at an aggregate level, and usually enables the analyst to drill down when something he notices requires further investigation. This leads to an element of chance that any problem will be spotted. As data volumes increase beyond what can be stored in a spreadsheet, it’s almost guaranteed that significant items will be missed: it’s simply not possible to analyze everything.

Event stream processing does things rather differently.

Firstly each event, or transaction, is analysed at an individual level. This is a systematic approach where every event is evaluated individually, not at an aggregate level. This is particularly relevant to policing of business processes, compliance, data cleansing and security applications where checking every item is important.

Secondly, data is analyzed in a stream, not as a batch. This means that each event is analysed sequentially, one at a time. So as every event is checked, it is compared with previous and historical patterns of events. Detecting significant sequences of events becomes a breeze. This is particularly relevant to CRM analytics scenarios, where you might want to detect churn or cross sell signals based upon how the customer is interacting with you. We’ve also seen that when you can respond to customer events in real time the response rate to a promotion is up to 50% higher than an offer made some time after. Sequences of events are also highly relevant to fraud and data cleansing applications, to name but two more.

Of course in addition to this, the analysis is done automatically, and in real time. No analyst has to notice, technology is doing the heavy lifting for you.

I just found EDS’s Next Big Thing blog, and specifically a piece by Charlie Bess titled “The New Role For Business Intelligence”

Charlie is spot on with his comments about ‘context’ being really critical in order to deploy BI more broadly. Let me extend the thought a bit further.

Today we rely on the reader of the report / viewer of the dashboard to interpret the data correctly in the context of historical performance, and their knowledge and experience of the business. By sticking to a paradigm which essentially reports on historical performance, we are relying on the human analysis of data.

This is a theme that I’ve developed in by free eBook “In Search of Insight” which can be downloaded at http://www.seewhy.com/ebook

BI companies have struggled to deploy beyond the 5% barrier of potential users, and this reliance on the presentation of historical data is significantly to blame. As soon as you want to deploy more broadly, in particular into operations, then the requirements change:

(1) Latency becomes very important. I’ve lost count the number times that I’ve heard something along the lines of “the reports arrives just too late to be really useful.” In fact analysts Ventana just produced a survey which found that “100 percent of respondents who said their alerts did not provide guided analysis also said their alerts were always out of date.”

(2) Operations teams require context. In fact ‘information in the context of the business process’ is critical in an intraday environment. There just isn’t enough time to start rooting around to try and interpret the data manually. So operational Business Intelligence systems need to be able to provide all the information in one place, and in a way that paints a crystal clear illustration of the problem or opportunity at hand. This is the so called ‘actionable insight’ that has been so over hyped and rarely delivered.

Of course in a real time world, then the context can be used by computers to automatically interpret the data for you and provide you with problems or opportunities, rather than presenting data. This is a very exciting area because it enables computers to perform tasks that human analysts are spectacularly bad at: automatically checking and validating every transaction, for example, so that exceptions or errors can be spotted.

This capability opens up completely new avenues for Business Intelligence as a real time process step, and enables closed loop automated actions to be driven off the analysis.

It’s not often that an industry changes irreversibly, but the advent of Open Source Software / Free software (OSS/FS) is having exactly that effect. While many Open Source advocates will not be surprised by the assertion, it has particular relevance to the Business Intelligence industry which has remained largely untouched by this new model.

I was prompted to this conclusion by a discussion with a customer that had downloaded the SeeWhy Community Edition and was using it to build a pilot implementation in conjunction with Open Source Business Intelligence software from Pentaho. [Incidentally I’m really excited about this project since it elegantly demonstrates the fit between BI 2.0 and traditional query based BI.]

Business Intelligence in particular needs incremental and iterative development. Business users are generally very bad at describing what they need, and requirements in BI projects invariably change through the life of the project. Consequently starting small with a series of small incremental steps, responding to evolving requirements, generally works best for BI.

Traditional software licensing models don’t support this mode of development. The vendor wants you to pay upfront for their software, which drives you and them to make the project bigger and therefore riskier.

Here’s why: in order to justify the return on investment needed to make the business case for the software purchase, the project grows in size. The vendor will gladly help you to increase the scope of the project, seeing more commitment and services. You will be looking for a big ROI number to help smooth the purchase through the purchasing process.

OSS/FS changes this fundamentally. You do not need to produce a theoretical ROI. You do not need to build a business case justifying a purchase.

Instead you just do it. Download it. Deploy it small. Evolve it. Get value.

Only when the project is delivering value, might you then consider support or maintenance, or upgrading to a professional version as the project grows in scope. But by this point the project has been proven to succeed, with (hopefully) a measurable ROI.

This path is also fundamentally different because it engages customer and vendor into a symbiotic relationship. The risks are aligned: the vendor wants to get your application into production (as you do) not simply to do a ‘drive by shooting’ where the vendor disappears as soon as the software license is sold. Under the new model, the vendor’s subscription pricing reinforces this: you pay only when the software delivers value, and only as long as it continues to do so into the future.

Andre Yee’s latest blog on Privacy, Information Theft and Security is prompted by yet more information disclosures, and makes the point that its sloppy process that is allowing this to happen. When will we get more serious about security? He’s absolutely right.

In many cases an insider is involved, colluding with accomplices on the outside. This recent case involved a member of staff in HSBC’s back office operations in India, leading allegedly to losses of $425,000. HSBC say that existing operational procedures (unspecified) identified the fraud. While we have to congratulate HSBC for ‘fessing up and for prosecuting the alleged offender, I’m alarmed that their existing procedures don’t kick in until $425k has been lost.

What is astounding is that while all financial institutions have a comprehensive audit trail, very few of them actually do anything proactive with it.

Bizarre as it may sound, the audit trail is rarely looked at – in fact it’s often only examined after a breach has come to light, and then only to size the problem. Note here that I didn’t say that it is used for gathering evidence, simply because these incidents are often dealt with more discretely. Despite terminating thousands of staff for dishonesty each year, as an industry, financial institutions (HSBC aside) are among the most secretive when it comes to prosecuting insider theft. After all, who wants to trust your hard earned savings to banks with a dishonesty problem?

Of course this issue is all the more topical due to the current fad of outsourcing customer service functions to cheaper locations globally. Who’s policing these remote sites? Who’s checking the audit log for inappropriate access to confidential data?

Government might ultimately weigh in here. While driven by mass data thefts, new federal legislation, in the form of the US Data Accountability and Trust Act (DATA), is undoubtedly coming that will require disclosure of any compromise of personal confidential data. That’s the theory at least, though DATA looks likely to be so watered down it will be ineffective when it eventually arrives.

So I suspect that while legislation will force disclosure under some circumstances, it’s a sorry state of affairs that we need legislation in the first place. What’s really needed is better security – yes we need to look at those audit logs every day, preferably automatically to validate the transactions that are being done on our systems. Validating the transactions will enable financial institutions to find data breaches when they are still minor, and before they grow into a public spectacle.

It’s a simple choice – fix the problem at source, or wait for a big data theft to hit you and then you too can have your fifteen minutes of fame.

I just got off the phone with Rick Whiting’s Information Week after a great discussion around real time BI. The call was prompted in part by his article “Businesses Mine Data To Predict What Happens Next” which touces on real time BI, but mainly covers the rise of predictive analytics. The article starts:

“Real-time information, once a competitive differentiator that produced more timely and relevant business decisions, is now a commodity. Even midsize companies process transactions as fast as the New York Stock Exchange, while decision makers communicate and collaborate over broadband networks as if they were in the same office. Sheer speed isn’t the advantage it once was.”

Of course, predictive analytics and real time BI are converging, but while many organizations are considering real time BI, real time predictive analytics are some way further out into the future. The analysis of real time information is actually still beyond most organizations – real time data is not. Processing transactions ‘as fast as the stock exchange’ is one thing, making sense of them to drive real time intelligent decisions is another matter. The difference is what it takes to turn huge quantities of real time data into useful, actionable information. There’s a big difference.

Going back to BI basics, from data we get information, which once analyzed (hopefully) gives us some insight. The process of getting from data to information requires some preparation, aggregation, and context. Getting to insight requires interpretation of the data (either manually or using embedded analytics) to then identify opportunities, costs or risks that the business can then act upon. This interpretation of data can involve using predictive analytics to predict a score to enable a problem to be identified before it occurs. Equally it could be a simple calculation to predict whether the shipment will occur on time or not.

What Rick misses in his article is that Predictive analytics today is generally an offline process, analyzing batches of data after the fact to identify patterns. This has lots of issues. It’s very manual, and it’s not fast. If fact it’s a very definitely not real time – it’s typically performed by highly skilled analysts. Once the analysis is complete, it’s almost always out of date. And the results may not actually be all that useful.

Somebody once described data mining to me as the ‘art of telling you the bleeding obvious’ because it might just prove that there is a statistically significant relationship between ice cream sales and the month of the year. Clearly it is obvious that ice cream sales will go up in the summer!

So let’s assume that you’ve mined your historical data, and found something useful. The challenge now is how to take the knowledge and make an impact on day to day operations. Traditionally there is a big disconnect here – the analyst writes a report, presents a PowerPoint, and maybe one or two key take out points get implemented. But the results are often not deployed live to make smarter operational decisions. There are always exceptions, in particular in the financial services industries, but generally there is a lack of automation of analysis processes. This of course is set to change.

Don’t get me wrong, predictive analytics definitively brings value. The challenge is how to take the insight gained and make it actionable. This brings you back full circle to real time data and business processes.

In an ideal world your predictive model can be deployed directly into the transaction stream within minutes, provide real time scores on massive quantities of data, which can change the outcome in real time of customer interactions. Oh, and the model will then self update itself as the data changes so that the model doesn’t go out of tune.
Real time data meets predictive analytics, enabling smarter in process decisions.

I need convincing that this is a commodity.