Real-time information no longer fast enough?
I just got off the phone with Rick Whiting’s Information Week after a great discussion around real time BI. The call was prompted in part by his article “Businesses Mine Data To Predict What Happens Next” which touces on real time BI, but mainly covers the rise of predictive analytics. The article starts:
“Real-time information, once a competitive differentiator that produced more timely and relevant business decisions, is now a commodity. Even midsize companies process transactions as fast as the New York Stock Exchange, while decision makers communicate and collaborate over broadband networks as if they were in the same office. Sheer speed isn’t the advantage it once was.”
Of course, predictive analytics and real time BI are converging, but while many organizations are considering real time BI, real time predictive analytics are some way further out into the future. The analysis of real time information is actually still beyond most organizations – real time data is not. Processing transactions ‘as fast as the stock exchange’ is one thing, making sense of them to drive real time intelligent decisions is another matter. The difference is what it takes to turn huge quantities of real time data into useful, actionable information. There’s a big difference.
Going back to BI basics, from data we get information, which once analyzed (hopefully) gives us some insight. The process of getting from data to information requires some preparation, aggregation, and context. Getting to insight requires interpretation of the data (either manually or using embedded analytics) to then identify opportunities, costs or risks that the business can then act upon. This interpretation of data can involve using predictive analytics to predict a score to enable a problem to be identified before it occurs. Equally it could be a simple calculation to predict whether the shipment will occur on time or not.
What Rick misses in his article is that Predictive analytics today is generally an offline process, analyzing batches of data after the fact to identify patterns. This has lots of issues. It’s very manual, and it’s not fast. If fact it’s a very definitely not real time – it’s typically performed by highly skilled analysts. Once the analysis is complete, it’s almost always out of date. And the results may not actually be all that useful.
Somebody once described data mining to me as the ‘art of telling you the bleeding obvious’ because it might just prove that there is a statistically significant relationship between ice cream sales and the month of the year. Clearly it is obvious that ice cream sales will go up in the summer!
So let’s assume that you’ve mined your historical data, and found something useful. The challenge now is how to take the knowledge and make an impact on day to day operations. Traditionally there is a big disconnect here – the analyst writes a report, presents a PowerPoint, and maybe one or two key take out points get implemented. But the results are often not deployed live to make smarter operational decisions. There are always exceptions, in particular in the financial services industries, but generally there is a lack of automation of analysis processes. This of course is set to change.
Don’t get me wrong, predictive analytics definitively brings value. The challenge is how to take the insight gained and make it actionable. This brings you back full circle to real time data and business processes.
In an ideal world your predictive model can be deployed directly into the transaction stream within minutes, provide real time scores on massive quantities of data, which can change the outcome in real time of customer interactions. Oh, and the model will then self update itself as the data changes so that the model doesn’t go out of tune.
Real time data meets predictive analytics, enabling smarter in process decisions.
I need convincing that this is a commodity.
