Comparing Java Runtime Analysis (or Profiling) Tools

Runtime analysis is a practice aimed at understanding software component behavior by using data collected during the execution of the component. The analysis provides an understanding of the following aspects of the application execution environment:

• Execution paths and code coverage.
• Memory utilization and memory leaks.
• Execution performance and performance bottlenecks.
• Thread analysis and related concurrency issues.

Enterprise Java applications that are designed to run on modern multi-core processors typically benefit from the use of a Java runtime analysis tool, as it provides information on memory leaks, performance bottlenecks and even concurrency issues such as deadlocks.

Recently at work, I got the opportunity to evaluate the leading Java profiling tools. An initial review of the leading tools in this arena resulted in the following list - JProfiler, YourKit Java Profiler, Java Visual VM, DevPartner Java Edition, Rational Purify, JProbe and OptimizeIt. A preliminary investigation shortlisted the candidates to JProfiler, YourKit Java Profiler and Java Visual VM. Both JProfiler and YourKit Profiler are leading award-winning tools and I was basically looking to compare them with the Java Visual VM, a free tool available with JDK 6.0 Update 7 (Windows). The other tools were rejected for various reasons - the DevPartner for Java product kept crashing, Rational didn't have a standalone Java edition, OptimizeIt only worked well with JBuilder, and finally, based on reviews posted on the web, JProbe was considered somewhat inferior to JProfiler and YourKit Profiler.

The three selected candidates (JProfiler 5.2.2, YourKit Java Profiler 8.0.13 and Java Visual VM 1.6.0_14 were compared against the following evaluation criteria:

• License Cost: While Visual VM is free, both JProfiler and YourKit Profiler are commercial tools that provide an option to purchase both standalone (node-locked) licenses and floating licenses. The license cost for both these products is more-or-less in the same range.
• Ease-of-use: This was an important consideration as the profiling tool should be intuitive to use, with the results being presented in an easily understandable format. All the tools faired equally in this category.
• Performance (CPU) Profiling: CPU profiling helps to identify hotspots (or method) that result in higher CPU usage. All the three tools provide comprehensive analysis, however, YourKit and JProfiler have better presentation options that display the data using call graphs and trees.
• Memory Utilization: This form of analysis presents information regarding the memory usage. The three tools under consideration provide a view which lists all the objects and their associated memory consumption. Again, the presentation of JProfiler and YourKit is slightly better than Visual VM.
• Thread Analysis: Provides a view of the threads running in the VM. All the tools under consideration provide very good thread analysis capabilties and also detect concurrency issues such as deadlocks.
• Code Coverage: This criteria was also under consideration; however, it was deemed less important in comparison with the other criteria.None of the tools being evaluated provided code coverage analysis.
• Remote Profiling: This is the ability to perform the runtime analysis from a remote machine. JProfiler and YourKit provide all the features that are available for local analysis; however, Visual VM only provides a limited set of features for remote analysis.
• IDE Integration: All three tools integrate well with Eclipse and other major Java IDEs.
• Supported Platforms: The three tools selected support all the major version of the common operating systems - Windows, Linux, Mac OS X and Solaris.

Based on the comparative analysis, it was clear that both JProfiler and YourKit were slightly superior products with some great profiling features. While Visual VM may not have all the features provided by the other two products, it is extremely intuitive and provides all the basic features that are desired from a runtime analysis tool. It is also important to note that since Visual VM is extensible via a plug-in architecture (just like Eclipse), it is poised for growth and contributions from the open-source community will eventually make it a compelling product, possibly at par (or even better) that JProfiler and YourKit.

Therefore, we decided to use Java Visual VM, as it satisfies our current needs and there is already some contribution from the development community in the form of some really useful plug-ins.

Comparison of Java source code analysis tools

Recently at work, I got the opportunity to evaluate some leading static source code analysis tools for Java. The intent was that a good source code analysis tool would help to improve the quality of the product by detecting problems in the code, well before the QA folks get it.

At expected, there are a plethora of tools in the market - some that are commercially available, whereas others that are provided free with the open-source initiative. I selected Coverity Prevent for Java, one of the leading commercially available tools, in part due to the existing relationship that my present employer has with Coverity. I also decided to pick two leading tools, FindBugs and PMD, from the open-source arena, as they seem to be the most popular tools in use. The selected tools were used to analyze the same Java codebase on a Windows 2008 Server machine with 1.5 GB allocated to the analysis.

Comparing source code analysis tools from different vendors is not a typical apple-to-apple comparison. Each tool has its own inherent strengths and does better than the other in certain areas. Anyhow, following an evaluation methodology that has always worked very well for me, I evaluated these three candidates again the following criteria:

• License Cost: An important consideration in any evaluation. Being a commercial product, Coverity Prevent has as associated license cost, whereas the other two are basically free.
• Quality of the Analysis: Obviously concurrency and resource ultilization issues are deemed more important that ununsed method / variables. This was a difficult comparison to make, as all the three tools reported a wide spectrum of problems. In general, however, I found the Coverity Prevent and FindBugs analysis to be better than PMD.
• Speed of the Analysis: Since the objective is to integrate the analysis with the nightly build, a short analysis time is preferred. While Coverity Prevent took hours to analyze the codebase, both FindBugs and PMD were done in minutes.
• Eclipse Integration: Having an Eclipse plugin is essential to report defects during day-to-day development. Fortunately, the three tools selected provide one.
• Rule Customization / Extension: The ability to customize the existing rulesets and add new ones was considered a desirable feature. While all three provided the option to add / drop certain rulesets from the anaysis, only FindBugs and PMD allowed the user to create new customized rulesets.
• Defect Reporting: This considered the ability of the tools to report the defects in the most intuitive and convenient manner. Coverity Prevent has a great web-based defect manager that allows the user to remotely look at the defects, review the associated source code and act on them. FindBugs has a GUI option that displays the defects and provides associated source markups, however, PMD doesn't provide the source markups and it is only available as a command-line program. All three tools provided an option to export the defects in different output formats (html, xml).

Based on the evaluation criteria given above, we selected FindBugs, which is in line with the wisdom of the web - "If you have never used a source code analysis tool before, try a free one first and see if that works for you". Typically, projects use multiple open-source code analysis tools and I found a lot of references to projects that have used both FindBugs and PMD.

We are now looking for a Java runtime analysis tool and I am doing an evaluation for that. I hope to post the results of that as well.