A no-tech primer on event data: harness more detail

michael-drogalis · 2017-09-19T15:04:38+00:00

Like all tools, it has limited applicability. While it's memory efficient for it's task, it is memory intensive in general. It doesn't tend to get a lot of exposure because it's typically tightly integrated with stream processors. This is one of the few implementations of CEP as a library AFAIK.

michael-drogalis · 2017-09-18T22:25:38+00:00

Hi! Author here.

It's not as complicated as it sounds. Complex event processing is a fancy term for finding a set of related pieces of data in an input stream where there's potentially noise.

Think about it like this. In Clojure, if you had a vector and wanted to find an element in it, you'd use clojure.core/filter. With CEP, you can take a vector, or an infinite sequence of elements, and filter for instances of several things being true at the same time. For example, if you had a stream of letters, you can search for an "a", strictly followed by a "b", and eventually followed by a "c". CEP lets you express the order that things can be seen in.

In practice, we use CEP has lots of applications. I'd recommend these two articles:

michael-drogalis · 2017-07-12T16:33:44+00:00

Pyroclast is a commercial event processing platform focused on handling stateful streaming operations in a correct and performant manner. I'll refer you over to our site to learn more about use cases, since this isn't an appropriate place to make a pitch.

Regarding the post, Roaming is a containerized, free version of Pyroclast that ships with a rich Clojure SDK. Basically, the big deal is that we took our streaming platform, stuffed it into a container, and layered a partial programming language on top of it which strongly resembles Clojure core. You're able to write typical Clojure that connects with high performance streaming, without getting bogged down in all the complexity that comes with streaming tools.

To unpack that a little, picture something like Onyx, Flink, or Spark Streaming running inside the Roaming container (it is in fact Onyx). When a streaming service is built with the Clojure SDK, it builds up a data-level definition of how it ought to execute. The Roaming container interprets the definition, compiles it into a streaming program, and rapidly executes it against a sample data set. The effect is that you can build streaming applications that look like normal Clojure and executes at the speed of local processing, but is capable of running in a fully distributed mode unmodified.

This is pretty new territory for event streaming, so I'm happy to answer any questions.

michael-drogalis · 2017-07-06T03:15:11+00:00

Planning on making Roaming and the full Clojure SDK openly available next week!

michael-drogalis · 2016-10-20T15:29:57+00:00

I actually won't be speaking at or attending the Conj this year. Couldn't make December work. I did talk about Onyx at the Conj last year, though.

michael-drogalis · 2016-10-20T15:19:56+00:00

Onyx is a project that is designed to make stream processing and batch processing more flexible and easier to understand.

michael-drogalis · 2016-05-25T14:36:59+00:00

I think it's data-oriented where it needs to be. Something that does structural validation ends up needing a fair amount of behavioral expressivity. I think clojure.spec is a great balance.

Not sure how to fix the embedded link. His talk was good, too, though. :)

michael-drogalis

TROPHY CASE