Store raw responseBody and decompress when needed #6389
Conversation
| if (responseData == null) { | ||
| return EMPTY_BA; | ||
| } | ||
| if (contentEncoding != null && responseData.length > 0) { | ||
| try { | ||
| switch (contentEncoding.toLowerCase(Locale.ROOT)) { | ||
| case "gzip": | ||
| return decompressGzip(responseData); | ||
| case "x-gzip": | ||
| return decompressGzip(responseData); | ||
| case "deflate": | ||
| return decompressDeflate(responseData); | ||
| case "br": | ||
| return decompressBrotli(responseData); | ||
| default: | ||
| return responseData; | ||
| } | ||
| } catch (IOException e) { | ||
| log.warn("Failed to decompress response data", e); | ||
| } | ||
| } |
There was a problem hiding this comment.
Have you considered something like interface ResponseDecoder { String decode(byte[] contents); } or interface Response { String toString(); byte[] toByteArray(); }?
Then there could be something like SampleResult#setResponse(Response) or SampleResult#setResponseDecoder(ResponseDecoder)?
It just does not sound quite right to tie SampleResult with specific implementations of the decoding
|
So I've done some benchmarking with this code, comparing it with default 5.6.3. 
Results are surprisingly good! Somehow, the benefit is relatively more on cpu rather than memory. I've ran a benchmark, running identical JMX script against the same environment at the same time. Both doing 100 threads, 300 requests per second. It was a recording of 5 transactions of a generic website, with a running time of about 80s per threadgroup. Ran it on 2 separate aws ec2 instances, 2cpu/2gb mem, max heap at 1500m The one without updated code, did about 15% cpu average during the run, while the updated code where we only decompress when the body is actually used (assertion/postprocessor), did about 10% on average during the run. Also the total memory usage of the VM was about 10% less, meaning the JVM didn't spiked as high. I think it has less impact on memory, because the responseData and previousResult gets overwritten every next sampler, so it only holds the latest responseData. But not having to decompressing all responses (for nothing) and not needing to do much heavier garbage collections, it saves more on cpu usage. |
|
Any status update on this item? The improvement is impressive on both memory and cpu consumption. I've been using a forked version for a long time with this improvement, but it would be great if this can be merged to JMeter code. |
vlsi
force-pushed
the
feature/lazy-response-decompression
branch
from
91a2eaf to
0b3ef6b
Compare
This commit addresses vlsi's architectural feedback on PR apache#6389 by implementing a ResponseDecoder interface pattern to decouple decompression logic from SampleResult. Key Changes: - Created ResponseDecoder interface in core module for pluggable decoders - Implemented PlainResponseDecoder for uncompressed responses - Created ResponseDecoderFactory in HTTP module with support for: * gzip/x-gzip compression (with relax mode support) * deflate compression (with relax mode support) * Brotli compression - Modified SampleResult to: * Store raw (possibly compressed) response data * Track content encoding via new responseContentEncoding field * Lazily decompress on getResponseData() using registered decoders * Cache decompressed data to avoid repeated decompression * Use a registry pattern to avoid circular dependencies - Updated HTTPHC4Impl to: * Disable automatic decompression (removed RESPONSE_CONTENT_ENCODING interceptor) * Extract and store Content-Encoding header value * Store raw compressed response data - Updated HTTPJavaImpl to: * Remove inline gzip decompression * Extract and store Content-Encoding header value * Store raw compressed response data Benefits: - Decoupled architecture as suggested by vlsi - Memory efficiency: only decompress when data is actually accessed - CPU efficiency: avoid unnecessary decompression for responses that aren't read - Maintains backward compatibility - Supports all existing compression formats (gzip, deflate, brotli) The implementation uses a registry pattern where the HTTP module registers its decoders with SampleResult, avoiding circular dependencies between core and protocol modules.
2 participants
Description
Based on idea and discussion here, this PR make it work
Motivation and Context
For every single requests which is compressed, JMeter always decompresses the data and stores the uncompressed data in responseData. This causes high memory usage per thread and adds cpu time for decompression when the responseData is never used in an assertion, post processor or listener.
How Has This Been Tested?
First of all, the tests from the build succeeded (after they failed for deflate which needed some extra care). Then I've ran some of my scripts against various websites and all worked.
With debugging I've checked that the responseData is actually only decompressed when accessed and it worked. I'd like to do a benchmark later under load and compare impact on memory and cpu
Screenshots (if appropriate):
Types of changes
Removed decompression from HC4 and Java HTTP implementation and doing the decompression in getResponseData(). This way, it is only decompressed when the data is accessed.
Checklist: