mlegge mlegge - 4 months ago 27
Linux Question

Pipe envsubst output to hive

Working with Hive 0.13.0, I would like to evaluate variables against a template and then immediately execute the resulting Hive code (avoiding a temporary intermediate file is preferable).

Here is a (non-working) example of what I'd like to do:

template.hql

SELECT COUNT(*) FROM ${TABLE};


In the shell:

export TABLE=DEFAULT.FOOTABLE
envsubst < template.hql | hive


Is there a particular reason this does not work, and is there a proper way to achieve it?

Answer

The substitution works as expected:

$ cat template.hql
SELECT COUNT(*) FROM ${TABLE};
$ export TABLE=DEFAULT.FOOTABLE
$ envsubst < template.hql
SELECT COUNT(*) FROM DEFAULT.FOOTABLE;

So I suspect hive does not read queries from the standard in. I see from an online manual that it supports the -f parameter, so you can create the file manually:

TMPFILE=$(mktemp)
envsubst < template.hql > $TMPFILE
hive -f $TMPFILE
rm $TMPFILE

If you're on a newish version of bash, you can avoid an intermediate file:

hive -f <( envsubst < template.hql )

I'm not sure, but also check if hive -f - might read from stdin.

Comments