twicejr twicejr - 17 days ago 7
YAML Question

Merging yaml config files recursively with bash

Is it possible using some smart piping and coding, to merge yaml files recursively?
In PHP, I make an array of them (each module can add or update config nodes of/in the system).

The goal is an export shellscript that will merge all separate module folders' config files into big merged files. It's faster, efficient, and the customer does not need the modularity at the time we deploy new versions via FTP, for example.

It should behave like the PHP function:

array_merge_recursive


The filesystem structure is like this:

mod/a/config/sys.yml
mod/a/config/another.yml
mod/b/config/sys.yml
mod/b/config/another.yml
mod/c/config/totally-new.yml
sys/config/sys.yml


Config looks like:

date:
format:
date_regular: %d-%m-%Y


And a module may, say, do this:

date:
format:
date_regular: regular dates are boring
date_special: !!!%d-%m-%Y!!!


So far, I have:

#!/bin/bash
#........
cp -R $dir_project/ $dir_to/
for i in $dir_project/mod/*/
do
cp -R "${i}/." $dir_to/sys/
done


This of course destroys all existing config files in the loop.. (rest of the system files are uniquely named)

Basically, I need a yaml parser for the command line, and an array_merge_recursive like alternative. Then a yaml writer to ouput it merged. I fear I have to start to learn Python because bash won't cut it on this one.

Answer

You can use for example perl. The next oneliner:

perl -MYAML::Merge::Simple=merge_files -MYAML -E 'say Dump merge_files(@ARGV)' f1.yaml f2.yaml

for the next input files: f1.yaml

date:
  epoch: 2342342343
  format:
    date_regular: "%d-%m-%Y"

f2.yaml

date:
  format:
    date_regular: regular dates are boring
    date_special: "!!!%d-%m-%Y!!!"

prints the merged result...

---
date:
  epoch: 2342342343
  format:
    date_regular: regular dates are boring
    date_special: '!!!%d-%m-%Y!!!'

Welcome to perl ;).

Comments