How to test image performance of Java libraries with JMH and IDEA IDE

What is JMH?

Java Microbench mark Harness is a very nice, free tool which allows you to create Java performance tests. This set of blog posts shows how we have used it to set up a set of performance tests. In our case, we wanted to run some tests of other Java Image libraries against our own JDeli image library. The tests need to be:

• Fair
• Replicable
• Consistent

To achieve this we have created a set of tests using JMH (Java Microbench mark Harness), some code examples we have put on GitHub and some sample images which are available on the web. The tests can be run from Java, directly. In our case, we have also chosen to run them from the IDEA IDE (which has some very nice JMH integration).

Prerequisites

In order to run the tests, you will need:

• Java8 or above
• IDEA (free or commercial version)
• The Image libraries we are testing
• A copy of our code from GitHub
• The sample images to test

How to test PNG read performance in Java

The easiest way to do this is to show, so let us kick off with a Test to investigate Java read performance using Java MicroBenchmarking using IDEA for PNG images.

Download our sample project from Github. Installing the IDEA JMH plugin is also recommended. While we can run JMH from the command line, IDEA makes it much easier.

If you download the code from GitHub and open it in IDEA, you should see a view similar to this (possibly with additional image tests added).

We have added a generic class ReadData to read images (optimised to only read once). Inside the class PNG (or its parent class ReadTest) you will find a whole series of simple tests. Here is the one for Apache Imaging
@Benchmark
public void Apache(BenchmarkState images) {

for (String pngFile : images.filesToRead) {
try {
BufferedImage img = Imaging.getBufferedImage(new File(pngFile));

} catch (Exception ex) {
System.out.println(ex);
}
}
}

We also have tests for our JDeli library (which will need a free trial download jar). Place the trial jar in the project’s libraries folder. If you do not wish to install Apache, JDeli, etc you can just delete the test.

You will also need to download some sample PNG files. There is a nice collection at PNGsuite. For our test, we are placing them in a directory called testImages/images-for-read-tests/png (there is a readme.txt with the link as well). Not all images will open in all libraries so you may find this can impact the results.

IDEA makes it very easy to run the test and you can see the output in the bottom window.

JMH will run a series of warmup iterations for the JVM (to eliminate the impact of a cold start on results) and provide some averaged stats on all the tests. The higher score the better.

Here is a sample output on one file

Here is the output on the entire PNGsuite.

Feel free to experiment, and let us know what you find!

We will be using JMH to compile some statistics for JDeli performance which is displayed on our website and used to optimise our code.