user2144835 user2144835 - 1 year ago 60
Bash Question

awk in non-interactive is leaving some data missing

I'm running an awk command in a shell script, and it's breaking the datafile. All of the data doesn't load, and there's a newline character at the end of each line.

awk -v name="$filename" -v batch="$BATCHNAME" -F'|' 'BEGIN{OFS="|"}{$2=batch;print > (name".locked.concur")}' $filename.locked.concur

If I run this as a command, it works perfectly.

I've just found that i can use fflush() and it seems to have solved the issue.

So now i have:

awk -v name="$filename" -v batch="$BATCHNAME" -F'|' 'BEGIN{OFS="|"}{$2=batch;print > (name".locked.concur") fflush()}' $filename.locked.concur

Am I setting myself up for disaster?

Also, the file is renamed to $filename.locked.concur0 -- I'm guessing this zero is the exit code that's returned from fflush? How do I get rid of this 0?

full code:

for filename in `cat ${INFILEDIR}/file_list_concur`


rm -f $OHAP_DAT/bad/$filename.bad

##Create Batchname based on filename

BATCH_GROUP=`ls $filename.locked.concur |cut -d '_' -f 1`
BATCH_DATE=`ls $filename.locked.concur |cut -d '_' -f 2`

echo 'Batch Date = '$BATCH_DATE
echo 'Batch Group = '$BATCH_GROUP
echo 'Batch Name = '$BATCHNAME
echo 'File Name = ' $filename

gawk -v name="$filename" -v batch="$BATCHNAME" -F'|' 'BEGIN{OFS="|"}{$2=batch;print > (name".locked.concur");}' $filename.locked.concur

##End Batchname

data=$OHAP_DAT/in/$filename.locked.concur \
log=$OHAP_DAT/log/$filename.log \
bad=$OHAP_DAT/bad/$filename.bad \
bindsize=512000 << end_of_sqlldr

if [ -f $OHAP_DAT/bad/$filename.bad ]
echo \!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
echo \!! 'date'
echo \!! SQL*Loader bad file was found!!
echo \!! Script $filename Failed - BAD FILE!
echo \!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Answer Source

the problem is indeed that you are reading from and writing to the same file. this should work better:

gawk ... print > (name".locked.concur.tmp")}' $filename.locked.concur
mv -f $filename.locked.concur.tmp $filename.locked.concur

better still:

gawk ... print}' $filename.locked.concur > $filename.locked.concur.tmp
mv -f $filename.locked.concur.tmp $filename.locked.concur