Before we jump into Complex Event Processing (CEP), let us first talk about Simple Event Processing (SEP). Imagine a system where we have an application that processes data (events) generated from another system, and then creates some output as a response. A very basic example could be as follows. You take out a calculator and tap in 2+3. For the calculator, the input data is the addition of two numbers i.e. 2, 3. The calculator processes it and gives an answer of 5, which is shown as the response. So, in a nutshell, Simple Event Processing analyses and triggers actions based on a single event that occurs in an upstream system. Naturally, SEP can be much more complex in real-world applications.
Now let’s go over Complex Event Processing (CEP). CEP allows for pattern recognition and triggering actions based on a combination (pattern) of multiple events of different types and coming from different sources. CEP as a concept was coined in the first half of the 1990s when big enterprises identified the need for processing multiple events from a variety of sources simultaneously and creating a valuable insight out of the same. CEP is a classic example of Event-Based Architecture. The idea is to process a stream of events coming from different source systems in real-time and to come up with meaningful and actionable insight as soon as possible. Like in any other form of data processing, CEP works on a collection of information from various sources and then processes them in order to give insight. This allows one to take appropriate actions. The heart of the entire process lies in processing the inbound events with preset business matrices and rules.
The illustration below gives an idea of how a CEP system might look.
CEP processes incoming events based on an existing pattern, in a real-time fashion. In comparison to Simple Event Processing, CEP systems execute data manipulation on via an algorithm that is pre-stored. The process achieves speed by discarding any irrelevant data in the beginning. As soon as the incoming events are compared to all the stored patterns, the result/response is sent out straight away, giving the process real-time capabilities. CEP is used for highly demanding, continuous-intelligence applications that enhance situational awareness and support real-time decisions. In addition to this speed, CEP systems are also highly scalable and performance-oriented. This allows them to create an insightful response in real-time.
CEP has a wide variety of applications in various industries. For example, in the Telecom industry, CEP can help operators understand the dynamic needs of their customers on a real-time basis. Let’s explore this example more to understand CEP. If a Telecom operator knows details about its customers (such as their spending capacity, brand preferences, location, usage patterns, available parking slots at a nearby parking lot, etc.) then they can send offers pertaining to a given brand when the customer is near that brand’s store. This type of marketing would make more sense when the customer is near the store, rather than after 1 hour (when the customer may be miles away from the store). So, the key elements of CEP are the variety and velocity of processing the data, allowing for real-time actions based on the output.
There are various open-source and propriety tools that can help in processing complex events like Apache Spark, Flink, Samza, etc from the open-source world and powerful data processing tools like Ab Initio, Microsoft Azure Stream Analytics, Stream Insight from the propriety software space. However, key factors like performance, scalability, built-in integration capabilities, shorter time-to-market, support of enterprise requirements and vendor support with defined SLAs and massively parallel processing capabilities from propriety tools can easily outcast open-source tools.
Ab Initio data platform provides features like a custom build CEP applications and out-of-the-box Customer Interaction Platform (CIP) that can help organizations on their complex event processing requirements with all the above-mentioned advantages of propriety tools. With a combination of other tools from Ab Initio suite like Metadata Hub, Acquire> It, Business Rule Engine, enterprises can have enriching features like end-to-end lineage, organization-level business glossaries, flexible control on transformation rules and the ease of plug and play should there be a requirement to integrate new source systems.
Data Insights as a technology partner with Ab Initio and the only partner in Europe to support CIP can help in creating high performant, realtime event processing platforms for big enterprises. Get in touch with us to understand how our expertise can help your organization to solve your business challenges by leveraging the value of your data to the fullest.
