Troubleshooting and Diagnostics techniques.

Techniques & Reference


JMX Clients

JDK tools (Experimental)

Profilers & Heap Analyzers

Notes on using JMX clients

When running a JMX client (JConsole, VisualVM) on the same machine as the target JVM process it is possible to connect without pre-configuring a JMX port, using the local connector stub. This method relies on being able to create a protected temporary file, accessible only to a user with administrator privileges. Java processes which are accessible via the local connector will automatically appear in the client.

NB(1) On Windows, this means that the temporary directory must be located on an NTFS formatted disk. See the following link for more details.

NB(2) On Windows, if Tomcat is started using a service wrapper, this will prevent JConsole & VisualVM from using the local JMX connector stub.

JConsole and Remote Management FAQ

From Java 6 a process needn't have the management agent enabled when it starts, as the Attach API permits the management agent to be activated on demand.

Common Troubleshooting Scenario

If you have already looked into Tomcat logs, there are no error messages, and you just want to find out what is going on, you may try the following

  1. Look into Tomcat access log (the log file generated by AccessLogValve).

    • If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall).
    • You will see what IP address your client is using, and whether it is using an IPv4 ( or IPv6 address (0:0:0:0:0:0:0:1). Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4.

  2. Take a thread dump. This way you will find out what Tomcat is actually doing.

    • If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress.
  3. Try debugging.

    • A good place for a breakpoint is org.apache.catalina.connector.CoyoteAdapter.service() method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts.

Troubleshooting Response State Problems

If you encounter problems that manifest themselves as accessing a request / response that is an inconsistent state.

E.g. 61289, 58457

The main suspect is your own web application keeping a reference to Request / Response objects outside of their life cycle.

  1. Set the following system property in Tomcat configuration:

    • org.apache.catalina.connector.RECYCLE_FACADES=true

    This flag instructs Tomcat to recycle facades to its internal objects, so that it is easier to spot illegal access when it happens, instead of waiting until internal state of referenced object becomes corrupted from concurrent access.
  2. Read about Java ImageIO issue.


FAQ/Troubleshooting_and_Diagnostics (last edited 2017-10-02 15:02:33 by KonstantinKolinko)