Stream analytics tame the data explosion
It was only when I was in discussion with a customer this week that I realized that most people think that primary benefit of real time Business Intelligence is its ability to analyze data in real time. And of course, they’d be right, but only partially. Another massive advantage is the way that data is analyzed, which is fundamentally different from traditional techniques since data is analyzed as a stream. As a consequence stream analytics enable new classes of analysis applications that were not previously possible. It is particularly relevant where data volumes are high.
Traditional data analysis relies on a person analyzing data in batches. First the data warehouse is updated, then queries can be run, and an analyst (or in some cases a business user) can then begin the search for insight. The search starts at an aggregate level, and usually enables the analyst to drill down when something he notices requires further investigation. This leads to an element of chance that any problem will be spotted. As data volumes increase beyond what can be stored in a spreadsheet, it’s almost guaranteed that significant items will be missed: it’s simply not possible to analyze everything.
Event stream processing does things rather differently.
Firstly each event, or transaction, is analysed at an individual level. This is a systematic approach where every event is evaluated individually, not at an aggregate level. This is particularly relevant to policing of business processes, compliance, data cleansing and security applications where checking every item is important.
Secondly, data is analyzed in a stream, not as a batch. This means that each event is analysed sequentially, one at a time. So as every event is checked, it is compared with previous and historical patterns of events. Detecting significant sequences of events becomes a breeze. This is particularly relevant to CRM analytics scenarios, where you might want to detect churn or cross sell signals based upon how the customer is interacting with you. We’ve also seen that when you can respond to customer events in real time the response rate to a promotion is up to 50% higher than an offer made some time after. Sequences of events are also highly relevant to fraud and data cleansing applications, to name but two more.
Of course in addition to this, the analysis is done automatically, and in real time. No analyst has to notice, technology is doing the heavy lifting for you.
