XML Parsing Performance Benchmark of VTD-XML 1.5

Objective

Since its initial release, VTD-XML has undergone several rounds of improvement. This report show how well VTD-XML fares against some of the well-known XML parsers. The old version of the benchmark can be found here.

Testing Methodology

Hardware

• Processor:  Pentium M 1.7GHz (2MB L2 integrated cache). Large cache size is typical among multi-core server processors.
• Memory: 512 MB 

Software

• Windows XP
• JDK version 1.506; default set to server JVM.
• XML parsers: Xerces DOM (with and without node expansion) and SAX version 2.7.1, VTD-XML 1.5 (with and without buffer reuse), XPP3 1.1.3.4.O, Piccolo 1.04

Benchmark Apps

    For DOM and VTD-XML, the benchmark programs generate hierarchical structures.

    For SAX and PULL parsers, the benchmark programs scan over the entire documents without any processing logic.

Notes on Performance Tuning and Performance Measurement

    For performance numbers, all benchmark programs first loop thru the parsing code a number of iterations so the server JVM compile them into native code to obtain optimal performance, before the real measurement of parsing throughput and latency starts.

    It should be noted that comparing VTD-XML with SAX or PULL is not really fair comparisons: VTD-XML allows random access; SAX and Pull are forward only.

   A wide selection of XML files, ranging from very small (1k) to big (15MB)  are chosen and grouped into small (<10k), medium sized (<1M), and big.

   Benchmark programs for measuring parsing performance can be downloaded below:

• benchmark.java (VTD-XML)
• benchmark_BR.java (VTD-XML with buffer reuse)
• benchmarkSAX.java  ( JDK's built-in SAX parser)
• benchmarkDOM.java (JDK's built-in DOM parser)
• benchmarkDOM_d.java ( Deferred node expansion)
• benchmarkPiccolo.java ( Piccolo SAX parser, widely considered to be the performance leader)  
• benchmarkPull.java  (XPP3)

  Benchmark programs for measuring memory usage can be downloaded below

• benchmark_mem.java (VTD-XML)
• benchmarkDOM_mem.java (JDK's built-in DOM parser)
• benchmarkDOM_mem_d.java (deferred node expansion)

 Benchmark programs for doing node iteration can be downloaded below

• benchmarkDOM_nav1.java (use DOM's node iterator)
• benchmark_nav1.java (use VTD-XML's autoPilot's iterator)

  XML files used in the benchmark can be downloaded here.

Parsing Performance

Throughput Comparison

• Small files

• Medium Sized files

• Big XML files

Latency Comparison

• Small Files

• Medium XML files

• Big XML files

Memory Usage

Because SAX and Pull do not build data structures in memory, so the meaningful comparison is between DOM and VTD-XML. To that end, we benchmark  the multiplying factor which is the ratio between the memory usage and the document size.

Navigation Performance

The goal for this part is to benchmark the performance of the XML parsers visiting every single node after finishing building the hierarchical structure.

Small Files

Mid-Sized Files

Large Files

Conclusion

As the next generation XML parser, VTD-XML delivers compelling improvement in both memory usage and parsing performance. Moreover, its performance and memory usage benefits apply to all file sizes and purposes. So it should satisfy even the most demanding XML processing needs and enable new and exciting XML applications.