Karussell Karussell - 7 months ago 27
Java Question

Efficient merging of multiple, large xml files into one

I searched the web and I searched stackoverflow up and down. No solution. Although I found solutions how to do this within pure xslt here.

But the problem is that the resulting xml will be several hundred MB large. So I must do this with SAX in Java. (please no xslt solution, although I tagged it with xslt ;-))

Let me explain with more detail. I have several multiple xml files (preferable InputSteam's) which should be parsed. The files or InputStream's looks like

inputstream1

<root>
<doc>
<tag>test1</tag>
</doc>
<doc>
<tag>test2</tag>
</doc>
...
</root>


inputstream2

<root>
<doc>
<tag>test3</tag>
</doc>
<doc>
<tag>test4</tag>
</doc>
...
</root>


inputstream1+inputstream2+...+inputstreamN = resulting xml. It will look like

<root>
<doc>
<tag>test1</tag>
</doc>
<doc>
<tag>test2</tag>
</doc>
...
<doc>
<tag>test3</tag>
</doc>
<doc>
<tag>test4</tag>
</doc>
...
</root>


Do someone has a solution or a link for this? Is this possible via implementing a custom InputSource or should I use a custom ContentHandler? Or is this possible with joost/stx?

The nice thing if I could use a ContentHandler would be that I could apply some minor transformations (I already implemented this). But then the problem is that I don't know a way to pass multiple files or InputStream's as InputSource:

XMLReader xmlReader = XMLReaderFactory.createXMLReader();
xmlReader.setContentHandler(customHandler);
xmlReader.parse(getInputSource()); // only one InputStream possible


or should I parse the InputStreams directly within my ContentHandler?

Answer

I finally managed this via the following snippet:

  finalHandler = new StreamResult(new OutputStreamWriter(System.out));
  // customHandler extends DefaultHandler
  CustomTransformerHandler customHandler = new CustomTransformerHandler(
         finalHandler);
  customHandler.startDocumentExplicitly();
  InputStream is = null;
  while ((is = customHandler.createNextInputStream()) != null) {
    // multiple inputStream parsing
    XMLReader myReader = XMLReaderFactory.createXMLReader();
    myReader.setContentHandler(customHandler);
    myReader.parse(new InputSource(is));
  }
  customHandler.endDocumentExplicitly();

The important part was to leave the startDocument and endDocument methods empty. All other methods (characters, startElement, endElement) will be redirected to the finalHandler. The customHandler.createNextInputStream method returns null if all inputstreams are read.