Real-time R&D enabled by the power of complex event processing

A week or so back, the WSJ Journal featured an in-depth article (http://tiny.cc/v9cAH) on the new face of innovation. Central to the article was the role that technology can now play in speeding and transforming the pace of innovation. While much has been written about how the web can be used to quickly determine how customers respond to a new idea, such rapid turn-around can now be accomplished on the retail floor, or wherever your touch-points are with your customer. How is this possible?

The authors cite the use of “digital systems” which collate and filter large amounts of data in real-time. So, when Wal-Mart rolls out a new product display, real-time statistics can be gleaned that determine how successful it is. While not mentioning the underlying technology, clearly what empowers such capabilities is complex event process (CEP), which is sometimes synonymous referred to as event stream processing (ESP). Using CEP, thousands, of even millions of events, can be analyzed through patterns or filters. Thus, using CEP, small experiments can be frequently run to determine their efficacy, without having to resort to the latent “after-the-fact” reporting that data warehouses use to provide. Greg Linden is quoted in the article as pointing out that “Constant, continuous, ubiquitous experimentation is the most important thing”. I call the use of CEP coupled with such rapid experimentation as ‘real-time R&D”.

While the companies that are cited in the article tend to be Fortune 500 enterprise customers, CEP doesn’t have to be limited to only those with deep pockets and a large IT organization. In my book, I cover an open source CEP implementation called Esper (http://esper.codehaus.org/). It’s surprisingly simple to use-and-learn, but the payback can be immense. Recently, I’ve also looked at the CEP capabilities now offered through JBoss Drools called Fusion. While I haven’t had ample time to fully kick it’s tires, it appears to offer similar capabilities to Esper (though I prefer Esper’s EPL language constructs, which more resemble SQL), and has the added benefit of being married to a rules engine — after all, you want to make decisions based upon the outcome of your pattern filters.

For anyone not yet versed on what CEP can provide, I’d recommend giving Esper or Fusion a try, and of course, my book provides a lot more coverage on the concepts and features typically found in a CEP engine.