Dataset vs Sampleset vs Calculator.exe

triggers

#1

I’d like to determine the average payload size over time by URI for responses that contain a particular string. The statistics for both total count by URI and size by URI are available natively, but I can’t plot them against each other natively using the interface, and it is impossible to export the data sets to CSV where I could line them up in Excel and turn them into a pivot table.

So I created a Dataset and a Sampleset in the trigger to try to capture the response length average per URI. However, the numbers don’t add up. Here is my trigger:

if (event == “HTTP_RESPONSE”) {

var app = "Update Panel";

if (HTTP.contentType == "text/plain") {
    if (HTTP.payload && HTTP.payload.indexOf("updatePanel") != -1) {
        debug("found updatePanel");
        var len = HTTP.rspSize;
        var uri = HTTP.uri;
        Application(app).commit();
        Application(app).metricAddSampleset("http_rsp_len", len);
        Application(app).metricAddDetailSampleset("http_rsp_len_detail", uri, len);
        Application(app).metricAddDataset("http_rsp_len_ds", len);
        Application(app).metricAddDetailDataset("http_rsp_len_detail_ds", uri, len);
    }
}

}

What I end up with looks the part, but the math doesn’t add up. For example, for a given interval over a six-hour period, I see it was fetched 538 times in the “Web” panel when I hit “Transaction Metrics.” Over the same time period, http_uri_detail:rsp_l2_bytes shows 142,799,163 bytes for the same URI.

This yields an average of 265,425 bytes. But when I look at the “http_rsp_len_detail_ds” Dataset I created above, I see the average as 167,436 bytes – this is substantially different, while my Sampleset shows only 82 samples and an average of 123,588. This is all within the same time period in the same application container.

It would be great if I could do simple arithmetic expressions on the custom app page to divide the bytes by the count without having to do a trigger, not to mention do a simple CSV export, but if nothing else I would expect Dataset to yield similar results to a division of aforementioned sums.

What method do I trust here?


#2

This may not reconcile everything, but let me toss out a few things that stand out to me.

  1. Your trigger operates off of HTTP.rspBytes, which according to the trigger API guide is L4 response bytes. If you’re elsewhere looking at L2 bytes, you may actually intend to use HTTP.rspL2Bytes in your trigger. (Late correction to this post: Trigger actually uses rspSize. See related comment below regarding rspSize vs rspBytes.)

  2. If you’re comparing the black lines in the orange-black-orange presentation across a sampleset and a data set, know that in a sampleset, the black line’s the mean. In a data set, it’s the median.

  3. If I’m reading you correctly, the 82 samples in your trigger vs 538 samples you see elsewhere could be related to how much payload you’re inspecting when you specify a # of payload bytes. The ‘updatePanel’ string you’re looking for may be falling outside of the payload sometimes.

However, I wonder if you don’t mean to look for ‘updatePanel’ in HTTP.uri instead of HTTP.payload in your trigger. If you elsewhere seeing ‘updatePanel’ in the ‘Transaction Metrics’ detail popup, that suggests to me it’s a component of the URI, so that’s where it would be best searched for in a trigger.

Just a few things that stand out when I read your post. Let me know if I’ve misread you anywhere or if the numbers begin to make more sense after taking these points into consideration.


#3

Thanks for the thoughtful answer. Let me address your points with a corresponding enumeration.

  1. I’m actually looking for the HTTP response length. I’m not interested in including the L2-L4 header data in the calculation, so rspBytes should give me what I need. It’s true that I’m calculating rsp_l2_bytes in the detail metric, but that’s because it’s the only detail metric available. I can’t imagine that the Ethernet, IP, and TCP headers would account for the amount of data swell I’m seeing, but there could definitely be something to this.

  2. I’m aware of the fact that the black line corresponds to the median; that’s fine. But the max shows 177kb versus my calculated average (with L2 bytes, granted) of 265k. But assuming 180 bytes per frame overhead for the TCP and IP headers, I would need my average TCP length to be 362 bytes just to make up the difference between the dataset max and the calculated average. Something still doesn’t add up here.

  3. All of the metrics are in the context of the same application, and are triggered on the same response, so that doesn’t explain the discrepancy.

  4. I’m very much looking for updatePanel to occur in the HTTP response body, not in the URI or anywhere in the HTTP request or response headers. If I was looking for a URI component, I’d be looking in our Splunk origin server logs. The use case here is to identify components on a complex application set that are using Microsoft’s updatePanel Javascript library, which doesn’t have any definitive evidence of use outside of the HTTP response.

Thanks again for considering my problem.


#4

Thanks for the clarifications. One correction to my point #1. I erroneously said you were using HTTP.rspBytes (L4 bytes). In fact your trigger uses HTTP.rspSize, which is the size of the HTTP response body, no HTTP headers. So that’s probably the largest possible delta… the difference in response L2bytes and the size of just the response body. Considering what you know about HTTP header sizes here, could that explain things?

If we’re still not quite there, any screenshots you felt comfortable posting might crystallize the situation.