Joshua Martell Joshua Martell - 1 year ago 127
Java Question

Fast compression in Java?

Is there a very fast compression library for Java? The standard gzip library is slower than I would like. I'm looking for something similar to that's native Java code that provides fast compression and decompression. Thanks!

A few other fast compression libraries for future reference:

QuickLZ -
C/C#/Java - GPL or commercial

libLZF -
C - BSD style license

FastLZ -
C - MIT style license

C - GPL or commercial

zlib -
C / Java (GZIP and deflate) - Commercial friendly license

Hadoop-LZO integration (JNI):

Snappy-Java (JNI):

Benchmarks from the QuickLZ folks:

Answer Source

You could use the DeflatorOutputStream and InflatorInputStream. These both use LZW compression. You could just use the library they provide.

EDIT: Real time performance is usually measured in terms of latency, however you quote numbers in terms of throughtput. Could you clarify what you mean by real-time.

For latency, using the BEST_SPEED, each call took 220 ns + 13 ns/byte on average.

Note: in low latency situations you often get many times the latency you might expect when the CPU is running "hot". You have perform the timing in a realistic situation.

EDIT: This is the compression rates I got with Java 6 update 21;

Raw OutputStream.write() - 2485 MB/sec

Deflator.NO_COMPRESSION - 99 MB/s

Deflator.BEST_SPEED - 85 MB/s.

Deflator.FILTERED - 77 MB/s

Deflator.HUFFMAN_ONLY - 79 MB/s



Note: I am not sure why the default setting is faster than the "best speed" setting. I can only assume the former has been optimised.

The output buffer size was 4KB, you might find a different size is best for you.

EDIT: The following code prints for a large CSV file. The latency is for a 5KB block.

Average latency 48532 ns. Bandwidth 91.0 MB/s.
Average latency 52560 ns. Bandwidth 83.0 MB/s.
Average latency 47602 ns. Bandwidth 93.0 MB/s.
Average latency 51099 ns. Bandwidth 86.0 MB/s.
Average latency 47695 ns. Bandwidth 93.0 MB/s.


public class Main {
    public static void main(String... args) throws IOException {
        final String filename = args[0];
        final File file = new File(filename);
        DataInputStream dis = new DataInputStream(new FileInputStream(file));
        byte[] bytes = new byte[(int) file.length()];
        test(bytes, false);
        for (int i = 0; i < 5; i++)
            test(bytes, true);

    private static void test(byte[] bytes, boolean print) throws IOException {
        OutputStream out = new ByteOutputStream(bytes.length);
        Deflater def = new Deflater(Deflator.BEST_SPEED);
        DeflaterOutputStream dos = new DeflaterOutputStream(out, def, 4 * 1024);
        long start = System.nanoTime();
        int count = 0;
        int size = 5 * 1024;
        for (int i = 0; i < bytes.length - size; i += size, count++) {
            dos.write(bytes, i, size);
        long time = System.nanoTime() - start;
        long latency = time / count;
        // 1 byte per ns = 1000 MB/s.
        long bandwidth = (count * size * 1000L) / time;
        if (print)
            System.out.println("Average latency " + latency + " ns. Bandwidth " + bandwidth + " MB/s.");    
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download