goldrush/README.org

# Goldrush #

Goldrush is a small Erlang app that provides fast event stream processing

# Features #
* Event processing compiled to a query module
 - per module private event processing statistics
 - query module logic can be combined for any/all filters
 - query module logic can be reduced to efficiently match event processing

* Complex event processing logic
 - match input events with greater than (gt) logic
 - match input events with less than (lt) logic
 - match input events with equal to (eq) logic
 - match input events with wildcard (wc) logic
 - match input events with notfound (nf) logic
 - match no input events (null blackhole) logic
 - match all input events (null passthrough) logic

* Handle output events
 - Once a query has been composed the output action can be overriden
   with one or more erlang functions. The functions will be applied to each
   output event from the query.

* Handle low latency retrieval of compile-time stored values.
- Values stored are also provided to functions called on event output.
- Handle job execution and timing which can also get values stored
- create input events that include runtime on successful function executions.

* Handle fastest lookups of stored values.
- provide state storage option to compile, caching the values in query module.

* Usage 
  To use goldrush in your application, you need to define it as a rebar dep or
  include it in erlang's path.


Before composing modules, you'll need to define a query. The query syntax 
matches any number of `{erlang, terms}' and is composed as follows:

* Simple Logic 
 - Simple logic is defined as any logic matching a single event filter

Select all events where 'a' exists and is greater than 0.
#+BEGIN_EXAMPLE
    glc:gt(a, 0).
#+END_EXAMPLE

Select all events where 'a' exists and is greater than or equal to 0.
#+BEGIN_EXAMPLE
    glc:gte(a, 0).
#+END_EXAMPLE

Select all events where 'a' exists and is equal to 0.
#+BEGIN_EXAMPLE
    glc:eq(a, 0).
#+END_EXAMPLE

Select all events where 'a' exists and is not equal to 0.
#+BEGIN_EXAMPLE
    glc:neq(a, 0).
#+END_EXAMPLE

Select all events where 'a' exists and is less than 0.
#+BEGIN_EXAMPLE
    glc:lt(a, 0).
#+END_EXAMPLE

Select all events where 'a' exists and is less than or equal to 0.
#+BEGIN_EXAMPLE
    glc:lte(a, 0).
#+END_EXAMPLE

Select all events where 'a' exists.
#+BEGIN_EXAMPLE
    glc:wc(a).
#+END_EXAMPLE

Select all events where 'a' does not exist.
#+BEGIN_EXAMPLE
    glc:nf(a).
#+END_EXAMPLE

Select no input events. User as a black hole query.
#+BEGIN_EXAMPLE
    glc:null(false).
#+END_EXAMPLE

Select all input events. Used as a passthrough query.
#+BEGIN_EXAMPLE
    glc:null(true).
#+END_EXAMPLE


* Combined Logic
 - Combined logic is defined as logic matching multiple event filters

Select all events where both 'a' AND 'b' exists and are greater than 0.
#+BEGIN_EXAMPLE
    glc:all([glc:gt(a, 0), glc:gt(b, 0)]).
#+END_EXAMPLE

Select all events where 'a' OR 'b' exists and are greater than 0.
#+BEGIN_EXAMPLE
    glc:any([glc:gt(a, 0), glc:gt(b, 0)]).
#+END_EXAMPLE

Select all events where 'a' AND 'b' exists where 'a' is greater than 1 and 'b' is less than 2.
#+BEGIN_EXAMPLE
    glc:all([glc:gt(a, 1), glc:lt(b, 2)]).
#+END_EXAMPLE

Select all events where 'a' OR 'b' exists where 'a' is greater than 1 and 'b' is less than 2.
#+BEGIN_EXAMPLE
    glc:any([glc:gt(a, 1), glc:lt(b, 2)]).
#+END_EXAMPLE


* Reduced Logic
 - Reduced logic is defined as logic which can be simplified to improve efficiency.

Select all events where 'a' is equal to 1, 'b' is equal to 2 and 'c' is equal to 3 and collapse any duplicate logic.
#+BEGIN_EXAMPLE
        glc_lib:reduce(
            glc:all([
                glc:any([glc:eq(a, 1), glc:eq(b, 2)]),
                glc:any([glc:eq(a, 1), glc:eq(c, 3)])])).
#+END_EXAMPLE

The previous example will produce and is equivalent to:
#+BEGIN_EXAMPLE
    glc:all([glc:eq(a, 1), glc:eq(b, 2), glc:eq(c, 3)]).
#+END_EXAMPLE



* Composing Modules 
  - All query modules must be compiled before use

To compose a module you will take your Query defined above and compile it. 
#+BEGIN_EXAMPLE
    glc:compile(Module, Query).
    glc:compile(Module, Query, State).
    glc:compile(Module, Query, State, ResetStatistics).
#+END_EXAMPLE


- At this point you will be able to handle an event using a compiled query. 

Begin by constructing an event list.
#+BEGIN_EXAMPLE
    Event = gre:make([{'a', 2}], [list]).
#+END_EXAMPLE

Now pass it to your query module to be handled.
#+BEGIN_EXAMPLE
    glc:handle(Module, Event).
#+END_EXAMPLE

* Handling output events
  - You can override the output action with an erlang function.

Write all input events as info reports to the error logger.
#+BEGIN_EXAMPLE
    glc:with(glc:null(true), fun(E) ->
         error_logger:info_report(gre:pairs(E)) end).
#+END_EXAMPLE

Write all input events where `error_level' exists and is less than 5 as info reports to the error logger.
#+BEGIN_EXAMPLE
    glc:with(glc:lt(error_level, 5), fun(E) ->
         error_logger:info_report(gre:pairs(E)) end).
#+END_EXAMPLE

Write all input events where `error_level' exists and is 3 or 5 as info reports to the error logger.
#+BEGIN_EXAMPLE
   glc:any([
     glc:with(glc:lt(error_level, 3), fun(E) ->
         error_logger:info_report(gre:pairs(E)) end),
     glc:with(glc:lt(error_level, 5), fun(E) ->
         error_logger:info_report(gre:pairs(E)) end)]).

#+END_EXAMPLE


# Composing Modules with stored state #

To compose a module with state data you will add a third argument (orddict).
#+BEGIN_EXAMPLE
    glc:compile(Module, Query, [{stored, value}]).
#+END_EXAMPLE

# Accessing stored state data #

Return the stored value in this query module.
#+BEGIN_EXAMPLE
{ok, value} = glc:get(stored).
#+END_EXAMPLE

Return all stored values in this query module.
#+BEGIN_EXAMPLE
[...] = Module:get().
#+END_EXAMPLE


* Composing Modules with stored data
  - You can create query modules with local state to compare to event data in `with' and `run'

To compose a module with state data you will add a third argument (orddict).
#+BEGIN_EXAMPLE
    glc:compile(Module, Query, [{stored, value}]).
#+END_EXAMPLE

* Accessing stored data in constant time 
 - You can use query modules in a way similar to mochiglobal 

Return the stored value in this query module.
#+BEGIN_EXAMPLE
{ok, value} = glc:get(stored).
#+END_EXAMPLE


* Job processing with composed modules 
  - You can use query modules to execute jobs, if the job errors or not, process an event.
  - `with' is similar to `run', the main difference is additional statistics and execution order
  - when a job completes in error, the event data will contain an additional {error, _} item

To execute a job through the query module, inputting an event on success.
#+BEGIN_EXAMPLE
    Event = gre:make([{'a', 2}], [list]).
    {ExecutionTime, Result}= glc:run(Module, fun(Event, State) ->
        %% do not end with {error, _} or throw an exception 
    end, Event).
#+END_EXAMPLE

* Event Processing Statistics 

Return the number of input events for this query module.
#+BEGIN_EXAMPLE
glc:input(Module).
#+END_EXAMPLE

Return the number of output events for this query module.
#+BEGIN_EXAMPLE
glc:output(Module).
#+END_EXAMPLE

Return the number of filtered events for this query module.
#+BEGIN_EXAMPLE
glc:filter(Module).
#+END_EXAMPLE


* Job Processing Statistics 

Return the number of job runs for this query module.
#+BEGIN_EXAMPLE
glc:job_run(Module).
#+END_EXAMPLE

Return the number of job errors for this query module.
#+BEGIN_EXAMPLE
glc:job_error(Module).
#+END_EXAMPLE

Return the number of job inputs for this query module.
#+BEGIN_EXAMPLE
glc:job_input(Module).
#+END_EXAMPLE

Return the amount of time jobs took for this query module.
#+BEGIN_EXAMPLE
glc:job_time(Module).
#+END_EXAMPLE


* Some Tips & Tricks 
  - This is really just a drop in the bucket.

Return the average time jobs took for this query module.
#+BEGIN_EXAMPLE
glc:job_time(Module) / glc:job_input(Module) / 1000000.
#+END_EXAMPLE


Return the query combining the conditional logic of multiple modules 
#+BEGIN_EXAMPLE
glc_lib:reduce(glc:all([Module1:info('query'), Module2:info('query')]).
#+END_EXAMPLE

Return all statistics from this query module.
#+BEGIN_EXAMPLE
glc:info(Module).
#+END_EXAMPLE

* Build

#+BEGIN_EXAMPLE
 $ ./rebar compile
#+END_EXAMPLE

or

#+BEGIN_EXAMPLE
    $ make
#+END_EXAMPLE

* CHANGELOG 
0.1.9
- Add support for running jobs

0.1.8
- Add support for not equal

0.1.7
- Support multiple functions specified using `with/2` 
- Add support for greater than or less than operators
- Add state storage option for output events or lookup

0.1.7
- Add job execution and timings
- Add state storage option 

0.1.7
- Add job execution and timings
- Add state storage option 

0.1.6 
- Add notfound event matching

0.1.5 
- Rewrite to make highly crash resilient
  - per module supervision
  - statistics data recovery 
- Add wildcard event matching
- Add reset counters