Nick Connicdev Nick Connicdev - 11 months ago 45
Bash Question

How to grep and remove from a file all lines between a separator

I have a file that looks like this:


How can I do a while loop and go through the file, dump anything between two
occurrences into another file for further processing?
I want to add only line2, line3 to the second file on the first iteration. I will parse the file; and on the next iteration I want line5 line6 in second file to do the same parsing again but on different data.

Answer Source

It sounds like you want to save each block of lines to a separate file.

The following solutions create output files f1, f2, containing the (non-empty) blocks of lines betwen the ===SEPARATOR=== lines.

With GNU Awk or Mawk:

awk -v fnamePrefix='f' -v RS='(^|\n)===SEPARATOR===(\n|$)' \
  'NF { fname = fnamePrefix (++n); print > fname; close(fname) }' file

Pure bash - which will be slow:

#!/usr/bin/env bash

fnamePrefix='f'; i=0
while IFS= read -r line; do
  [[ $line == '===SEPARATOR===' ]] && { (( ++i )); > "${fnamePrefix}${i}"; continue; }
  printf '%s\n' "$line" >> "${fnamePrefix}${i}"
done < file