Best way to fully capture and store a large HTTP payload

triggers

#1

Hi guys, I am trying to capture some very large Solr search queries over HTTP that are approximately 128 KB in size. I understand the Detail Key has a 4KB hard limit, so the first thing that comes to mind is output the payload results to MongoDB or Splunk. Before I take the time to set them up, I want to make sure there are no limitations that would result in partial data like the limit on the Detail Key. What do you recommend as a reliable method and data store to offload the 128 KB payload so the whole query remains intact? Thanks!

P.S. I’m also aware that kind of payload capture will very likely create some performance issues so it will be used sparingly.


#2

Both RemoteSyslog and MongoDB have maximum message sizes. These maximums default to 1K and 4K respectively but are configurable (see the docs). One of these would be the best way to get 128K of data off the system, but you should do this as surgically as possible to reduce potential performance impacts.


#3

Thanks, PacketMan. Surgical is the idea but a partial query isn’t useful so we’ll have to figure something out on our end. I appreciate the reply.


#4

Charlie, just to be clear, it is possible to extract the full query by changing some configuration parameters on the ExtraHop (and making sure wherever you are sending the query can handle receiving the full 128K). For example, if you were to use the remote syslog capabilities to send the query to Spunk, you would have to increase the message_length_max config parameter (see the RemoteSyslog section of the trigger API docs). As the docs say, there is a little overhead for the remote syslog message header so this parameter would need to be a little more than 128K. Then in the trigger editor, you would have to set HTTP payload bytes to buffer to 131072). Again, the performance caveat applies!


#5

One other thing - for this case, I would recommend setting the remote syslog send_interval parameter to 0 so the ExtraHop will send the message immediately which will reduce the memory impact of this. This parameter needs to be set in the running config as well. So, to add to the example from the trigger API docs:
“capture”: {
“rsyslog”: {
“host”: “splunkium”,
“port”: 54322,
“ipproto”: “tcp”,
“message_length_max”: 140000,
“send_interval”: 0
}
}