FrustratedWithFormsDesigner FrustratedWithFormsDesigner - 1 month ago 6x
Java Question

Effecient way to convert a stream of Strings into grouped list of strings

I have a function which will receive a

. This stream represents the lines in a file (as called by
). The file itself is actually the concatenation of many files into a single file, something like this:

__HEADER__ # for file 1
more data
__HEADER__ # file 2 starts here
some more data...

I need to convert the stream into multiple physical files on the filesystem.

I've tried the simple approach, something along the lines of:

String allLinesJoined = lineStream.collect(Collectors.joining());
// This solution seems to get stuck on the line above ^
String files[] = allLinesJoined.split("__HEADER__");
for (fileStr : files)
// This function will write each fileStr to a separate file
// (filename is determined by contents of fileStr)

But the input file is about ~300 MB (and could get larger) and this solution seems to get stuck on the first line. Maybe it would complete if I had more memory...?

Is there a better way to do this, if my starting point is a
, or should I start making other changes so that this bit of code can just read through the file line by line, without using the streaming API?

(the order of the lines does matter, in the context of these files)


I need to turn one big file represented as
in to many little files. Each little file begins with
and all lines after, until the next
. The current library uses streams to provide the file, but is it even worth trying to do this with streams, or will my life be easier if I change the library to offer non-stream functionality?


That kills the whole idea of streams.

Try forEach():

    Stream<String> lineStream = Files.lines(Paths.get("your_file"));

    lineStream.forEachOrdered((s) -> {
        if ("HEADER".equals(s)) {
            // create new file
        else {
            // append to this file