Logging to MongoDB, Part II

Integrations
#1

In part 1, you read about the power of a Wire Data analytics approach to I.T. Ops and the business it serves. So we’ll dig right in with a use case.

Our demo app.

I’ve created an example application for this series of posts.

Imagine remote devices with a JSON-RPC api that sends back critical device metrics. This data may be relevant for regulatory reasons, for operational reasons, etc. Think smart-grid in the power industry, a turnstile at a concert venue, or remote fulfillment centers: the ‘internet of things’.

I cooked this demo up in my head - any similarity to a real environment is coincidental. That said, its pretty common stuff that should have real-world applications. Imagine remote industrial plants with valves pushing important stuff through them. Velocity, temperature, and pressure are the key metrics. Each plant has multiple valves, with unique numbers to identify them.

To illustrate, have a look at the screen shots below:

###URI detail view:

So we can see that it’s a REST-like thing, and that the URI is broken out into cities, with the metric specified, e.g.

/metrics/atlanta/pressure

The payload of the responses looks like this:

image description

Some interesting data in there! You can see the following:

  1. Each city has multiple ‘valves’, each reporting their metrics.
  2. There are three metrics we can focus on.
  3. The response payload is JSON formatted data (it could be XML, etc. but I like JSON so it did that).

Let’s start with a trivial trigger that logs response payload to MongoDB. This tiny trigger allows us to do some powerful things:

/*
Step 1: Grab response payload, then log in MongoDB.
*/

// Set up a list of paths we want to ignore for now.
var ignore_paths = ['/metrics/put','/metrics/delete','/metrics/all'];

if(event == 'HTTP_RESPONSE'){
    
    // bail if not 200 OK. We'll come back to this later. 
    if(HTTP.statusCode !== 200) return;
    
    // bail if it's a path we ignore.  
    if(ignore_paths.indexOf(HTTP.path) > -1){
        return;
    }else{  // store in MongoDB if we get this far.
       
        // Insert payload into MongoDB. It's this easy:
        Remote.MongoDB.insert('valves.metrics', JSON.parse(HTTP.payload));
    }
}

I’ll walk through the trigger, then we’ll query our data in Part III.

  1. If it’s not a 200, bail from the trigger:

    // bail if not 200 OK. We’ll come back to this later.
    if(HTTP.statusCode !== 200) return;

  2. If it is a 200, make sure it’s a path we’re interested in, then send to MongoDB:

    // bail if it's a path we ignore.  
if(ignore_paths.indexOf(HTTP.path) > -1){
   return;
 }else{  
     // Insert payload into MongoDB. It's this easy:
     Remote.MongoDB.insert('valves.metrics', JSON.parse(HTTP.payload));
  }

This line is where the magic happens:

Remote.MongoDB.insert('valves.metrics', JSON.parse(HTTP.payload));

We’re telling our trigger to send JSON-parsed payload into the ‘valves’ database, into the ‘metrics’ collection. Think of a MongoDB collection roughly as a MySQL table…

###What? ExtraHop is able to log Payload to MongoDB?

Yes, friends, it is. Drink in the power of Wire Data greatness :slight_smile:

Now that it’s there, we can query our metrics and discover some powerful business / ops insights, right from the wire…we’ll do that in part III.

#2

the images are missing for this post. is there a new version available ?