Robin D. Wilson
Email: <<MailTo(rwilson2+jmeter@gmail.com)>>
NOTES on "strange pause" during testing run
When testing a fairly simple HTTP(S) 'registration' process, enabling the "Retrieve All Embedded Resources" flag for the HTTP Samplers caused an unexpected 'pause' (~35 seconds in duration) to be regularly added to the test cycle. The addition of the "Retrieve All Embedded Resources" flag caused the test to increase the amount of data being transmitted from the test servers to the JMeter 2.4 client to be multiplied by about 10X. The pauses occurred at regular intervals during the test run - where the test would run fine for about 1 min, and 25 secs, and then 'pause' for about 35 secs, and then continue for another 1 min, 25 secs, then pause for another 35 secs, with this pattern repeating until the end of the test. One of the key symptoms of the problem was that the servers involved and the JMeter client all showed 'idle' (CPU, Network I/O, and Disk I/O) during the 'pause' period - indicating that they were not 'catching up' on handling requests, but that they had nothing to work on during the pause. The problem was eventually traced to have 'iptables' enabled on the Apache (HTTPS) web server that was being hit directly by JMeter. The iptables service is the Linux 'firewall' application. And while it was not filtering any of the test packets out, it was parsing each packet - which was substantially slowing its ability to service incoming network traffic. Eventually, the server would back up on servicing incoming packets to the point that it would fill its buffers, and stop accepting requests for a specified 'backoff' period (~35 seconds). Disabling 'iptables' on the Linux hosts resolved the problem completely.
Test conditions:
Client Configuration:
- JMeter 2.4 being run on a Windows XP box
- Testing 4 HTML (over HTTPS) pages
- Key Issue - when "Retrieve All Embedded Resources" is enabled, test runs for ~500-600 iterations (2400 of the 4 pages + 10X that in embedded resources), then pauses for about 35 seconds, then starts running at full speed again. (THE PAUSE IS NOT INTENDED, AND IS THE PRECISE PROBLEM EXPERIENCED.)
Server Configuration:
- Apache Web (HTTPS) Server - configured for AJP Proxy to Tomcat Java App Server
- Main page hitting AJP to Tomcat
- Embedded Resources (images, .js, .css, flash movies, etc.) served directly from Apache docroot
- Tomcat Server (receives AJP requests only from Apache Server)
- MySQL DB Server (receives JDBC connections from Tomcat Server only)
Root Cause:
- The 'iptables' service (Linux Firewall) on the Apache Server had been enabled
- Even though no packets were filtered, parsing all incoming packets slowed the network interface down to the point of filling the buffers - causing a backoff period
- Disabling the 'iptables' service solved the problem. Command 'service iptables stop' was issued, as well as 'chkconfig iptables off' (to permanently disable service on reboot).