Feature Requests for Splunk Integration

Some of what we would like to see/accomplish:

  • detections (not alerts) in Splunk ES
  • identity and asset information which is appended to ES and ITSI assets and identities
  • ability to write triggers and if needed send that data to Splunk as an event
  • metric data

Some additional points …

Splunk is moving forward with metric data collection - perfmon for example. This integration is being rolled into Splunk App for Infrastructure which in-turn will be the primary integration piece for all ES and ITSI metric-based information. ExtraHop needs to be a part of this, and become better integrated with Splunk.

As I have been installing/diagnosing this, I have the following observances:

Point 1 - extrahopoid
I know it was in previous versions of the Splunk app, but it was deprecated at some point, and yet it still is asking for it even with a completely fresh install of BOTH the Splunk ExtraHop Add-On and the Splunk ExtraHop App on Splunk 8.X.

This is the current scheduled search upon which much of the app depends on:
/opt/splunk/etc/apps/extrahop_app/default/savedsearches.conf:search = sourcetype=“extrahop”| stats count by object_type oid host | fields - count | eval from=“data” | append [|inputlookup extrahop_deviceoid_lookup |rename hostname as host, otype as object_type | table oid,host,object_type | eval from=“lookup”] | stats values(from) as from by oid,host,object_type | where mvcount(from) = 1 AND from=“data” | join oid host object_type [search sourcetype=“extrahop” | stats count by object_type oid host source | top 1 source by oid host object_type] | fields - count percent from | extrahopoid | outputlookup extrahop_deviceoid_lookup append=true

This search, if it cannot run, does not populate a lookup file which is required by the app. Therefore errors abound and the app is useless.

Point 2 - Python 3
The version 2.0.0 of the Splunk for ExtraHop requires Splunk version 8.x and Python 3 … however Splunk Enterprise still defaults to python 2.7 for 8.x for now to support other vendors who have not made the transition. The ExtraHop Add-On app does not specify in the python to use the the Splunk python3 path, therefore it fails unless you force python3 at the Splunk core - which breaks other apps who have not transitioned to python3; or you manually change all the python files and then start troubleshooting additional python library reference issues.

Point 3 - Installation documentation
The ExtraHop documentation does not do a good job of explaining how everything is supposed to connect and work between Splunk and ExtraHop. I’ve seen documentation for HEC, Syslog, Direct-to-Splunk, API, ODS, record store, etc.
Everything becomes a confusing conglomerate and your never quite sure what your getting until you do it - if it even works, or stays working. The directions need to be broken out better and documented more concisely.

Additionally, In general the ExtraHop documentation site is a wall of text and everything starts blending together visually - headers, sub-headers, type fonts and colors. See this page for example: https://docs.extrahop.com/current/integrate-extrahop-with-splunk/

Point 4 - App and Add-On Dependency
The dependency between the Add-On and App in Splunk is strong. MOST customers who are going to be integrating ExtraHop are probably going to have a distributed Splunk environment with many Search Heads and dedicated input Heavy Forwarders. Due to how the app works and its dependencies, you cannot install and configure your add-on inputs on the HF and then run the app/add-on separately on search heads. Doing so causes many errors and the app fails to work; unless you devote significant time in re-configuring how the inputs, props, transforms, custom commands, etc. work and where they look for information.
Documentation on how all of this dependency works and what is going on is virtually non-existent.

Our customers constantly ask for easier ways of sending detections to a 3rd party system.

Right now, the process is brutal:

  • write a trigger to harvest all detections (hook DETECTION_UPDATE event)
  • cherry pick the detection you want (that’s hard because we don’t list all detection titles)
  • process the detection fields (have you tried to process offender/victim, it’s brutally hard)
  • craft the message payload
  • properly format the message
  • send message

That is way, way too much work for “I’d like to know the next time this particular detection fires”.

By contrast, customer see the Alert dialog’s email field and SNMP toggle and ask “can I have this for Detections?”

This isn’t a new ask. This request – make it easier to send specific events off box – has been around as far back as firmware version 3.5.

@gumby, what does this scenario look like using the REST API? This is the preferred approach for integrations rather than triggers.

I don’t know.

Do I need to do this research before the feature request can move forward?

I’m confused.